Sequence-to-Set Generative Models
Authors: Longtao Tang
Ying Zhou
Yu Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: generative models
set data
sequences
e-commerce orders
representation learning

Abstract: In this paper, we propose a sequence-to-set method that can transform any sequence generative model based on maximum likelihood to a set generative model where we can evaluate the utility/probability of any set. An efficient importance sampling algorithm is devised to tackle the computational challenge of learning our sequence-to-set model. We present GRU2Set, which is an instance of our sequence-to-set method and employs the famous GRU model as the sequence generative model.
To further obtain permutation invariant representation of sets, we devise the SetNN model which is also an instance of the sequence-to-set model. A direct application of our models is to learn an order/set distribution from a collection of e-commerce orders, which is an essential step in many important operational decisions such as inventory arrangement for fast delivery. Based on the intuition that small-sized sets are usually easier to learn than large sets, we propose a size-bias trick that can help learn better set distributions with respect to the $\ell_1$-distance evaluation metric. Two e-commerce order datasets, TMALL and HKTVMALL, are used to conduct extensive experiments to show the effectiveness of our models. The experimental results demonstrate that our models can learn better set/order distributions from order data than the baselines. Moreover, no matter what model we use, applying the size-bias trick can always improve the quality of the set distribution learned from data.

Sequence-to-Set Generative Models
Longtao Tang', Ying Zhou² and Yu Yang* 1,3
School of Data Science, City University Hong Kong, Hong Kong. China
2 DDpparmennt of Economics and Finance. City University Hong Kong, Hong Kong, China 3 Hong Kong Institute for Data Science, City University of Hong Kong, Hong Kong, China Longttang2-con c8my. cityu. edu. hk, {ying. zhou, yuyang)@cityu.com edu. hk
Abstract
In this paper, we propose requence-to-set method that can transform any sequence generative model based on maximum likelihood to set generative model where evaluate the utility/probability of any set. An efficient importance sampling algorithm is devised to tackle the computational challenge of learning our sequence- to-set model. We present GRU2Set, which is instance of our sequence-to-ses to-set method and employs the famous GRU model as the sequence generative model. To further obtain permutation invariant representation of sets, we devise the SetNN model which is also an instance the sequence-to- to-enneee-t--etteeeee set model. A direct application of our models is learn order/set distribution from collection of e-commerce orders, which is an essential step in many important operational decisions such as inventory arrangement for fast delivery. Based the intuition that small-sized sets are usually easier to learn than large sets, we propose a size-bias trick that can help learn better distributions with respect the -distance evaluation metric. Two e-commerce order datasets, TMALL and HKTVMALL, are used to conduct extensive experiments show the effectiveness of our models. The experimental results demonstrate that our models can learn better set/order distributions from order data than the baselines. Moreover, no matter what model we use, applying the size-bias trick can always improve the quality of the set distribution learned from data.
Introduction
Learning a generative model [Goodiellow et al. 2014, Kingma and Welling 2014) from data to generate random samples is a fundamental unsupervised learning task. In many real applications, the data, such as customer orders and drug packages, can be represented by sets. Learning a set generative model can boost many important applications such simulating users' demand of e-commerce orders for inventory arrangement [Wu et al. 2021] However, literature, little attention has been paid to generative models for sets.
In this paper, we study the problem of learning set generative model from data. Specifically, we study sets containing some categorical items from a finite ground set. Some previous studies [Benson al. 2018 Wu al. 202T proposed traditional statistical models to learn the set generative model, while how to employ deep learning design set generative models remains largely untouched. Meanwhile, deep generative models for sequence data (Hochreiter and Schmidhuber 1997 Schäler and Zimmermann 2006 Cho al. 2014 Devlin et al. 2018 have been extensively studied. Since the only difference between a sequence and a set is that the order items does not matter a set, we propose to leverage sequence generative model to build set generative model.
Yu Yang the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A natural idea of converting random sequence to random set is to just ignore the order of items. However, for set, the number of possible sequences that can be regarded as equivalent to the set is exponential to the number of items. This brings us serious computational challenge where we need to enumerate all the equivalent sequences for a set when learning the parameters of the model. Traditional statistical models [Benson et al. 2018 Wu et al., 202T] along this line either make unrealistic independent assumptions on items or run a learning algorithm of complexity exponential to set sizes to only deal with small sets. These constrained treatments undermine the model's 's ability to capture the complex correlations among items in sets.
To tackle the computational challenge while still well capturing the correlations among items, in this paper, we make the following technical contributions.
In Section S we first show that we can convert any sequence generative model based on maximum
likelihood, such RNN (Schäfer and Zimmermann] 2006 and DeepDiffuse [[slam et al. 2018 to set generative model. We call our method Sequence-to-Set To address the computational challenge, we propose an efficient importance sampling algorithm to estimate the gradient of the log-likelihood function of our Sequence-to-Se model.
In Section 4 to capture the complex item correlations, we propose two deep models GRU2Set and SetNN, which are instances of our Sequence-tor seuuencc-to-Stttt model. GRU2Set directly uses the GRU as an essential building block, while SetNN is designed for obtaining permutation invariant set embeddings when learning a set generative model. We discuss that GRU2Set and SetNN can express any set distributions long as we increase the model capacity.
To learn better set distribution from data, Section 5 we reveal that we often need to modify the empirical size distribution of the training data and propose a heuristic to do so. This size-bias trick can be applied to all set generative models aiming at learning a set distribution.
We conduct extensive experiments using two e-commerce datasets TMALL and HKTVMALL, and report the experimental results in Section C The experimental results clearly show the superiority of our models to the baselines and the effectiveness of our size-bia bias trick.
2 Related Work
Learning a set generative model is often regarded as learning a choice model for subset selection. Along this line, Benson et al. (2018) proposed a discrete choice model (DCM) to model the util- ity/probability that users select a specific set. [W et al. [202T] utilized representation learning to design parameterized Markov Random Walk to model how users select random sets from an item
graph. Both [Benson et al. 2018 and [Wu et al., 202T are set generative models, though they are not deep models. To achieve practical computational efficiency, both [Benson et al. 2018 and (Wu et al. 202T] only handle small sets and compromise on modeling high-order item correlations.
To model the permutation invariant nature of sets, Ruiz et al. [207)] used the sum or max of item embedding as the set embedding. Zaheer et al. [201]] systematically investigated the theory of
permutation invariant functions. [Seezzner et al. 2020 proposed a generative model for point clouds
based on GAN [Goodfellow et al. 2014 and Set Transformer [Lee et al. 2019 Although a point cloud is set of 2D coorddnisssssss we cannot use the point cloud generative model [Stelzner et al. 2020) to solve our problem as the model outputs continuous random variables instead of categorical data. Kosiorek et al. [2020] presented another generative model for sets of real-valued vectors based on Set Transformer Lee et al. 2019] and their generating process based on VAE [Kingma and Welling 2014). A disadvantage of using GAN or VAE to train generative model is that it would be difficult to evaluate the probability density of a random sample. Normalizing Flows are powerful generative models based on solid statistical theories. However, the study of Discrete Flows is still in its infancy (Hoogeboom et al. 2019 Tran et al. 2019] and no existing works have investigated using Discrete Flows to learn set distributions.
Multi-labe classification also needs to model the permutation invariance of sets of labels. Yang et al. (2019) applied reinforeement learning with permutation cnrmutation-invaiian reward function to remove the impact of label orders. Yazici et al. 2020) proposed to align the labels of a sample with its predicted labels before calculating the loss of the sample. However, our task in this paper is an unsupervised
learning task. We cannot apply the loss functions in (Yang et al. 2019 Yazici et al. 2020) since we
do not have "labels" (or feature vectors if one regards a set as a set of labels) for training samples.

---

To sum up, existing works on set generative models either deal with sets of real-valued vectors which are different from categorical sets, or adopt non-deep models which need to compromise on capturing item correlations to make the computation efficient. It is still urgent to design deep generative models for categorical sets and propose efficient algorithms to deal with computational challenges.
Sequence-to-Set Generative Models
In this section, we introduce how to convert sequence generative model to a set generative model and how to optimize the model parameters.
3.1 Converting Sequences to Sets
We first characterize the sequence generative model that we want to convert to set generative model.
Definition (Sequence Generative Model) A sequence generative model is a model with a se- quential generating process, which can be represented by sequence of states and actions 1s, ag $1, 01, .... ST-1, ST-1.0T-1; dr-1, ST.. where 8, the state time i, a, is the action taken at time i and ST is the end state. We call (sp, ao, $1, Q1,..,,T -..1. 11.111.1 ST] -: a generating path. Like finite-state state machine, the sequence generative model uses each action a; to deterministically update the current state 8; to 8141- For each state s, the sequence generative model has corresponding probability distribution p(a s: 0) indicating the probability of choosing action a when the current state is where 0 is the parameters of the sequence generative model.
Our definition of the sequence generative model can cover a wide range models that can generate items in a sequential manner. For example, the commonly used sequence model RNN is an instance, where the state is the history vector H and an action is to choose the next word/item. Moreover, influence diffusion models such the Independent Cascade (IC) model [Goldenberg et al.) 2001] and the DeepDiffuse model [Islam et al. (0188) are also covered by Definition LJ since in these models the state at time is the set of activated nodes until time / and the action d, is to select the group of nodes to be activated at time + 1.
To convert a sequence generative model in Definition I to a set generative model, we need to associate each state with state-associated set S., For example, in an RNN, we can define Ss, as the set of items generated in the first steps. In the IC model, we can define Ss, as the set of nodes activated at or before time
We say that a generating path 1 = [so, 40 -T-1, 1151155] can induce set S if S Say. Let L(S) = {i can induce S} be the path set of S. As the sequence generative model has the probability distribution p(a 8; 0) indicating the probability of taking a as the next action given the
current state s, we can derive the probability of having a path [so. ag. .o......pp ST OT -1, sT) as
p(1: 0) p(ao so; 0) x p(a1 $1; 0) x ... x p(ar ST; 0).
Therefore, given a sequence generative model with parameters the probability induce set S is
p(S: 0) = Σ 0(l;
(EL(S)
3.2 Parameter Learning
Suppose we have collection (multiset) of observed sets S = (S1, Sx,, ). We want to use S as the training data to learn sequence model which can generate the observed sets. We adopt maximum likelihood estimation to learn the parameters O. Specifically, the loss function is
(
Σ log Pogp(S;:0 0)
Σ log
p(1:0) IEL(S.)
Σ
We apply stochastic gradient descent based algorithms to minimize the loss function. The stochastic gradient w.r.t. observation S is V (S 0) = V log (EiELts) p():6)). As in the log-sum term of V logp(S: 0), the number of paths that can induce S is often exponential to ||] (in some
3

---

sequence generative model the number of such paths is even infinite), it is very challenging to evaluate v log p(S exactly.
To tackle the computational challenge in computing V (o; 0), we first find that V Vog can be regarded as expectation as follows.
Vlogp(S;0) EIEL(S) Vp(1; EIEL(S) p(l; 0) log pog 0)
Famp((|s:0)
V pog 0)),
0)
p(S;0)
where p(1 [5:0) p(l:0) is posterior distribution indicating the probability that a given set S p(S:0)
generated by the path Therefore, V log p(S; 0) can be seen as the expected value of Vp(1: 0)
where the random variable / follows the posterior distribution p(1 S: 0)
We apply Monte Carlo method to estimate V log ((S; 0) (1)=p(1]::0) V log p(1: 0)), where the key point is to sample generating paths from p(|]S; 0). A naive idea is to use rejection sampling, where we use the model with parameters 0 to generate random paths and reject those that cannot induce S. Obviously such naive implementation would be extremely inefficient as the rejection rate would be very high.
To sample from p() ; 0) more efficiently, we devise an important sampling method as follows. Let 7 (a, 8, 5) be an indicator function, where aa,, 8, S) = if 31 [so; 0o,,... . 8T 1, 11,-1, 87] such 8 A = In our sampling algorithm, for a state St, we only consider actions or such that T 8, S) = 1. We take such an action a, with probability
that induces S and 7i E 1........ T
1),
p(a,1 8; 0)
g(a: 8; s; Σ =a: p(a 0;
Therefore, the proposal distribution 0)) for 1 (so, ................................ -1. 0T- ST] in our importance sampling is
<(1: S) 4(a0 so; 0, x q(ai The (unnormalized) importance weight of / is
82; 0, x q((T-1 (T-110,
rl::0) ())
p(a so: 0)
Σ
p(a (1;0s x....
Σ
p(a sT-1:
1=7(0.77-155)-
Clearly, we have
r(1: (11 O, x p( s0: 0)p(a1)(a)))))))))))))))) |s: 1al/arr1----1100 = 0(1;)).
This shows the correctness of the sampling method.
An EM perspective Using our importance sampling algorithm to estimate V log p(Spp(S00 0) in min- imizing the loss function L()) can be interpreted from an Exspetation-MMxxiniattii
algorithm's perspective. Our method is actually a Monte Carlo EM algorithm if we treat a generating path / as a latent variable. In the E-step, we sample the latent variable / from the posterior distribution p[[]S; gold). Our importance sampling can boost the efficiency of the E-step. In the M-step, we reconstruct object function which is ΣM M r(4))= p(,; @old), where M is the number of samples and A = Σ r(4). To maximize this function, we update by gradient based meth- ods, such as new = Oold + 1) "M R log;(1;; gold), where 1) is the learning rate and EM rl.) Va po,; gold) is estimated by our importance sampling algorithm.
MRW as an instance of Sequence2Set We show that the Markov Random Walk (MRW) model [Wu et al. 202T] also an instance of our Sequence2Set model. The sequence model in MRW is random walk on the item graph [Wu et al. 2021 where node is an item and we have a special stop item k. The state s, at time is associated with set Se, which includes all the items visited until time t. An action a, taken at time f given the state s, is to jump to an item i, from i,-1, the item visited at time 1. Once the stop item & visited, the random walk terminates. The probability of jumping from 4 to i, = i is parameterized as p(1+ = i 8t-1; 0), where 0 is the embeddings exp(e, c.,_)) where e; is the embedding
of items. Specifically, in MRW, p(ù i 1,
exple, )

---

vector of item i. When an action a is jumping to an item i * K, the indicator 7(a. S) if S.
When an action a is jumping to the stop item K, 7(a, s, S) if = S. Wu et al. [202] propose an algorithm of complexity exponential to [S] to calculate the gradient V pl 0) for learning the
item embeddings and thus, only small sets of size at most are considered [Wu et al. 2021 If we
adopt our important sampling algorithm, we can deal with large sets.
4 GRU2Set and SetNN
In this section, we propose two deep Sequence-to Set models GRU2Set and Set Neural Networks (SetNN), We first introduce some general settings that will be used in both GRU2Set and SetNN.
Item embedding and stop item We also adopt item embeddings, which should be learned from the training dataset, to parameterize the probability p(a | x; O) in the sequence generative model. We assign an embedding vector e; E I to each item i. Similar to MRW (Wu et al, 2021), we also add a stop item where once we add to the set associated to the current state, we stop the generating process. The stop item x also has an embedding vector ex E R
Sparse item graph In our Sequence-to-Se model, the indicator function T(a, s, S) is key for us efficiently sample generating paths from the posterior distribution p(1 S 0) Intuitively, r(a, S,S) indicates what actions are valid given the current state & and the induced set S. To further boost the efficiency of our importance sampling algorithm, we make extra constraints on the items that can be added in the next step when we given the current state To do so, we build an item graph = (V. E), where V is the set of items plus the stop item K and (i.5) E if there is set S in the training dataset S such that i E S and j E S. The stop item is connected to all the other items. When using our importance sampling algorithm, given the current state s associated with a set S. S S, r(a, 8, = if action a is adding an item E S O Neighbor(S.) where Neighbor(S.) is the set of neighbors to items in S., When starting from the first state su associated with a set S.. 0, set Neighbor(S..)
Neighbors
Figure I: An example of sparse item graph. The item & is the stop item.
V {s:). In real set datasets such TMALL and HKTVMALL used in our experiments, such an item graph built from training datasets is often very sparse. Therefore, we call the item graph the sparse item graph. Clearly, the sparsity of the item graph can help boost the efficiency of generating random sets from learned model.
Choice vector In our model design, the conditional probability p(a s; 0) is the key to control the generating process. Exploiting the idea of representation learning, we represent each state s with an embedding vector C, E Rd. We call c, the choice vector of Given s, we control the probability of adding as the next item as
explc, ci) Σ (ENNigghor(SS)) exp(c, e;)
p(i s; 0)
The sparsity of the item graph can make computing the normalization term in p(i s; 0) efficient. Note that in MRW (Wu et al. 2021). the choice vector c2 is the embedding of the last item visited so far. We will see that in GRU2Set and SetNN, we use deep neural nets to aggregate the embeddings of all the items visited so far to obtain the choice vector Cy. The initial choice vector Csp is trainable.
4.1 GRU2Set
We leverage GRU Cho et al. 2014 to build our first sequence-to-ser model, since GRU is one of the most popular sequential model based on RNN and its computational cost is low. We set the history vector H of GRU as a d-dimensional vector and H is regarded as the choice vector of the current state. The initial history vector Ho is trainable and is updated according to the design of GRU. of items added in the first steps of the sequence generating process of GRU. When the stop item added, we terminate the generating process. Due to limited space, the overview of GRU2Set is put in Appendix.
The state-associated set S., for the state 8; at time / is just the set
5

---

S
S4, S
[
state 4
Adding item
Update State
state S+3
action s,
Fg
medin
MLP
Fur Set embedding
mean embedding
S
Reached item embedding
Spr
state
choice vector
softmox
PA
Adding item y
P4
action
Neighbbor's embedding
Figure 2: Overview SetNN. We do not show the situation when we add the stop item. If the action adding the stop item, we end the process immediately.
4.2 SetNN
The choice vector c, can be regarded as the embedding of the S., GRU2Set, the choice vector, which is the history vector H, is sensitive to the order of items added to S.. As sets are permutation invariant, hope have sequence-lo-se model which can also produce permutation invariant representations of sets. Therefore, we develop the Set Neural Networks (SeNNN) where the key idea is design permutation invariant aggregation of embeddings of items in set. The following theorem by Zaheer et al. [2017] guides our model design.
Theorem (Zaheer al. 2017 A function f(X) operating on a set X having elements from countable universe, is a set function, i.e., invariant to the permutation of instances in X, iff it can be decomposed in the form P(Exex ((z)) for suitable transformations and p.
(x:) can be viewed the embedding of and the over (xx) can be replaced by other aggregate operators such as mean or pooling. In SetNN, we use an MLP to express the function & in Theorem obtain the embedding (choice vector) c, of S, for any state s. Fig shows the overview of SctNNN
4.3 Expressive Power of GRU2Set and SetNN
Compared to SetNN, our GRU2Set model uses the history vector By the expressive power of RNN (Schäfer and Zimmermann, 2006), as we increase the embedding dimension d, hidden neural units and layers in GRU module, the history vector H of GRU2Set can approximate any function finite history path arbitrarily close. Then, of course, H can express any permutation invariant function on sets which implies that the expressive power of GRU2Set is more than SetNN.
To explore the expressive power of SetNN, we first give recursive definition of the probability that set S is generated by SetNN, denoted by p(S), as follows.
P(S)
(n is the stop item)
7(S) Σ p (a)))()) (z))
sES
700) 1,
where p(x[S) (=)) conditional probability of adding item x to the set S {x) and 7(S) is the probability of reaching state such that S, = S in SeINN. The following theorem shows that Eq. is general enough to cover possible set distributions.
For any distribution q(S) on all subsets of a ground set of items, there exists a group of transition probubilities p(ar) S) and p( 15 S), such that q(S) = p(5) holds for any S, where p(S) is defined in Eq. II.
Theorem

---

The proof of Theorem 2 can be found in Appendix B As we increase the capacity of the MLP in SetNN, SetNN can represent any conditional probability p(z|S) and p(x]S), and as result of Theorem SetNN and GRU2Set can express any set distributions.
5 Size-Bias Trick for Improving Learned Set Distributions
In this section, we propose a size- bias trick that can help reduce the distance between the ground truth set distribution p" (S) and the learned distribution p(S), where p(S) can be learned by any generative model not limited to our Sequence-tor Soquencc-to-Set models. We first decompose the ground truth distribution (S) when S = k as
p p0(S) - p p*(k)
where P = -A:(A|=>P(AA is the probability of generating a set of size k, and (5( = S(S
2%
the probability that a random size-k set is S. Suppose we know the real size distribution Pk
and we have (5) as an estimation of p" (S) for each How can we construct a distribution ((5) close to p" (S)? An intuitive way is combine (S) and P such that g(S) q (S)
[S] = k [Stelzner et al. 2020 Benson et al. 2018 However, we argue that this intuitive way is not always the optimal plan as follows.
Let K be the largest possible set size. To find the best size distribution qu to collaborate with our estimation (5), we consider the following optimization aiming at minimizing the KL--divereecc between the distribution ((5) = (S) and the ground truth p" (S).
KL(gllp") = Σ (S) log ((S)
min q.....9
p*(S)
8
s.t. (S) = qk x q (S) A|S =
(2)
0 S 9k < 1,
4k
Since we do know p" (S), we cannot solve Eq.
2 exactly. However, we still can tell if setting
Qse Pi. is the optimal solution to some extend.
Let fa(gx) E = (S) ()) log (" We have (L(( (gk). Note that fe(gs) only
g(S)
depends on dk- We can interpret Eq. 2 a portfolio optimization problem. There K projects. gu
is the investment for project k and JR (gx) ) is the revenue of project k. To decide the best portfolio
Tee we check the derivative of f& (xk), which indicates the marginal revenue. We have
((S)
K((x))
(S) log
(S)
S:|S|=
By taking 9k = Ph. we have
g (S) (())
fa(t) 11
Σ S:|S|=k
q (S)
1 + KL ( (S)I0))))))))
As the estimation quality of (S) often varies for different k the derivative (pf) probably also varies for different kt making qk pf not a stationary point. Therefore, probably need to set qk different from P for solving Eq. We first make the following reasonable assumption.
Assumption Small-sized sets easier learn than large sets. In other words, (S) estimates (S( better as the set size k decreases.
Based on Assumption I compared to pi. we should "invest" more small set sizes to construct the "portfolio" (size distribution) (k- Therefore, we set ge slightly bigger than Pf for small lt and set ys smaller than p for big k. To address the issue of not knowing PE. we resort to the empirical size
7

---

Table 1: Basic statistics
dataset
fitem
#training sets
average #testing sets
average #edge
TMALL
1,363
100,000
393,226
60,956
HKTVMALL
782
200,000
616,652
53,853
Table 2: Size distribution of TMALL and HKTVMALL
3
4
5
6
2
7
TMALL
38%
18%
14%
10%
7%
5%
8% 12%
HKTVMALL
42%
17%
11%
8%
6%
5%
distribution of the training dataset, which is a good estimation of the real size distribution pi- We construct biased size distribution qs by adopting the following heuristic.
Constructing Biased Size Distribution Suppose the empirical size distribution of the training data
S is (PI.P2.... Pk]. We first calculate the rest proportion by ты pa/(ps + Pk+1 + + pk). which means pt (1 ri)...(1 rı-1)r;- Then calculate r/g. max (rk + kv [V 1}. where V is the set of all items. We construct a biased size distribution [gt, (h..92,---30 [K] by setting q. r1)...(1 r/
6 Experiments
Datasets We did our empirical study on two real-world datasets of customer orders from online e-commerce platforms. This first dataset is TMALL (https://ww)) tmall. com) which contains 1363 items and orders from August 2018 to March 2019. We treated the collection of orders each month as instance and in total we have instances for the TMALL dataset. The second dataset is HKTVMALL (https: //www hktvmall.......m....mmmmmmmmmmml com) which contains 728 items form the supermarket sector and orders from February 2020 to September 2020. Similar TMALL, we also treated HKTVMALL as 8 instances where each instance is the collection of orders in each month. For both datasets, We split all the orders in a month to a training dataset Strain and a testing dataset Steet- where the size of Strein is 100,000 for Tmall and 200,000 for HKTVmall. The datasets can be found in the source code. We report the sparsity of item graphs and statistics of the datasets in Table I The set size distribution of each dataset is shown in Table 2
Experiment Setting and Evaluation We treat the method of directly using the histogram of Strain as the benchmark method, as it is a natural and unbiased method to estimate the set distribution. Note that this benchmark method has major disadvantage that its cannot model probabilities of any sets not showing in Strain We also compare our models with two baselines which are DCM (Discrete Choice Model) [ennson et al. 2018 and MRW [Wu al. 2021.
Following [wu et al. 2021), for each method (except the benchmark), we generated a collection Spred of 10,000,000 random sets and used the empirical set distribution of Spred approximate the set distribution learned by the model. We regarded the empirical distribution of the testing data Steat the pseudo ground truth. Since the bigger Stest is, the more accurate the pseudo ground truth is, we set the size of Stest bigger than Strain as shown in Table I Denote by Niest(S) the number of sets that are S in Stest- Let NSrea(S) be the number of sets that are S in Spred- We used the l -distance between model's (approximate) distribution and the pseudo ground truth to evaluate the effectiveness of the model. Specifically, the l -ddistance can be calculated as follows.
(S) Nprea(S) [Spred|
(Stests Spred)
Σ SESt
We set the embedding dimension of all methods as 10. For each training and testing dataset, the biased size distribution was the same for all methods and was obtained by the heuristic introduced at the end of Section 5 For all experiments, the MLP SetNN only has one hidden layer which contains 50 neural units. All the experiments were ran CPU with 10 cores. The optimizer used by us is RMSProp with default parameter in PyTorch. The source code and data used in our experiments can be found at https: /

---

Table 3: The performance of cach groups on TMALL. GO stands for the first group of data.
GO
G1
G2
G3
G4
G5
G6
G7
Average
STDEV
Benchmark
1.11
1.02
0.91
0.90
0.94
0.94
0.93
0.85
0.95
0.07
+Size Bias
1.06
0.97
0.87
0.86
0.90
0.90
0.89
0.81
0.91
0.07
DCM
1.17
1.08
0.95
0.94
1.00
0.99
0.99
0.89
1.00
0.08
Size Bias
1.10
1.01
0.90
0.89
0.94
0.93
0.93
0.84
0.94
0.07
MRW
1.14
1.07
0.99
0.96
1.02
1.02
1.03
0.92
1.02
0.06
Size Bias
1.06
0.97
0.88
0.87
0.90
0.92
0.95
0.84
0.92
0.06
SeINN
1.12
1.01
0,92
0.89
0.90
0,91
1.01
0.89
0.96
0.08
+Size Bias
1.04
0.93
0.84
0.82
0.85
0.87
0.89
0.79
0.88
0.07
GRU2Set
1.11
1.04
0.89
0.90
0.94
0,94
1.00
0.90
0.97
0.07
+Size Bias
1.02
0.93
0.83
0.82
0.85
0.86
0.89
0.80
0.87
0.06
Table 4: The performance of each groups on HKTVMALL. GO stands for the first group of data.
GO
G1
G2
G3
G4
G5
G6
G7
Average
STDEV
Benchmark
0.73
0.87
0.91
0.84
0.81
0,87
0.81
0.80
0.83
0.05
+Size Bias
0.70
0.83
0.88
0.81
0.79
0.84
0.78
0.77
0.80
0.05
DCM
0.76
0.89
0.93
0.88
0.84
0.89
0.84
0.82
0.86
0.05
Size Bias
0.72
0.84
0.89
0.83
0.80
0.85
0.79
0.78
0.81
0.05
MRW
0,80
0.91
0.94
0.88
0.88
0.90
0.85
0.85
0.88
0.04
+Size Bias
0.74
0.84
0.88
0.82
0.83
0.85
0.79
0.80
0.82
0.04
SetNN
0.82
0.86
1.00
0.84
0.89
0.92
0.84
0.84
0.88
0.05
+Size Bias
0.71
0.82
0.87
0.79
0.79
0.83
0.76
0.78
0.79
0.04
GRU2Set 0.83
0.87
0.94
0.84
0.93
0.94
0.86
0.82
0.88
0.05
+Size Bias
0.70
0.80
0.85
0.78
0.79
0.82
0.75
0.75
0.78
0.04
Experiment Results Due to limited space, we only report the main experimental results. More
detailed experimental results be found in Appendix D and E
Table 3 and Table Ssow the experimental results. We place the results of using the size-bias trick below the original results. We find that both DCM and MRW never beat the benchmark method, no matter the size-bias trick is played or not. A possible reason is that in our experiments we did not make size constraints
sets, while Wu al. [021]] constrained the set size to be at most 4 and Benson et al. [2188) set the maximum size of sets to be 5. Big sets in our data may have negative effects on learning both DCM and MRW.
We can see that applying the size-bias trick can always reduce the -ddistance significantly, which demonstrates the effectiveness of this trick. Before applying the size-bias trick, the benchmark method has the best performance which is slightly better than our models GRU2Set and SetNN. However, after applying the size-bias trick, our GRU2Set model becomes the best and it outperforms the benchmark method by 4% on TMALL and 2.5% on HKTVMALL. SetNN also outperforms the benchmark method after using the size-bias trick. This suggests that GRU2Set and SetNN may learn probabilities of small sets better than the benchmark method.
The SeINN model performs slightly worse than GRU2Set but the gap between them is small. A possible reason is that GRU2Set has stronger expressive power than SetNN as illustrated in Section 4.3 GRU2Set records the order of items added to the set, while SetNN ignores such order to achieve permutation invariant set embeddings. This shows that GRU2Set includes more information the generating process than SetNN. However, we want to emphasize that SeNNN can produce permutation invariant set embeddings while GRU2Set cannot. If we have a downstream task that needs set embeddings, SetNN may have more advantages in this case.

---

7 Conclusion
In this paper, we present a Sequence-to-Set method to build generative models for set data and the SOTA method MRW is an instance of our method. To utilize deep learning in learning set distributions, we further design two models, GRU2Set and SetNN, which are two instances of our Sequence-to-Se model. To learn better set distributions from data, also propose size-bias trick. Experimental results on two e-commerce order datasets clearly show that our models outperform the baselines including two SOTA methods in this line of research.
For future work, we will explore using more sophisticated deep learning modules in our model, such as replacing the MLP in SetNN with Set Transformer and substituting the RNN in GRU with more advanced sequence models. For potential negative societal impacts, learning the embeddings of items and sets might cause user privacy leakage as many other embeddings might do. Research on how to hide personal information on set data could help protect users' privacy.
Acknowledgments and Disclosure of Funding
Tang and Yang's research is supported part by the Hong Kong Research Grants Council under ECS grant 21214720, City University of Hong Kong under Project 9610465, and Alibaba Group through Alibaba Innovative Research (AIR) Program. Zhou's research is supported in part by City University of Hong Kong under Project 7200694. The authors thank the HKTVmall Open Databank for providing the HKTVMALL dataset. All opinions, findings, conclusions, and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.
References
Austin R. Benson, Ravi Kumar, and Andrew Tomkins. A discrete choice model for subset selection. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018. ACM, 2018
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, 10s/1440.1008,, 2014.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Jacob Goldenberg, Barak Libai, and Erika Muller. Using complex systems analysis to advance marketing theory development. 2001.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, 2014.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Emiel Hoogeboom, Jorn W.T. Peters, Rianne van den Berg, and Max Welling, Integer Discrete Flows and Lossless Compression. Curran Associates Inc., Red Hook, NY, USA, 2019.
Mohammad Raihanul Islam, Sathappan Muthiah, Bijaya Adhikari, B Aditya Prakash, and Naren Ra- makrishnan. Deepdiffuse: Predicting the' who' and when' in cascades. 2018 IEEE International Conference on Data Mining (ICDM), pages (055-10600 IEEE, 2018.
Sudeep Kamath, Alon Orlitsky, Dheeraj Pichapati, and Ananda Theertha Suresh. On learning distributions from their samples. In Conference on Learning Theory, pages 1066-1100, PMLR, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
10

---

Adam R. Kosiorek, Hyunjik Kim, and Danilo J. Rezende. Conditional set generation with transform- ers. CoRR, 2020.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention- based permutation-ine -nvarriinn neural networks. In ICML 2019. PMLR, 2019.
Francisco Ruiz, Susan Athey, and David M. Blei. SHOPPER: A probabilistic model of consumer choice with substitutes and complements. CoRR, abs/a/11.00300000 2017.
Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In Stefanos D. Kollias, Andreas Stafylopatis, Włodzistaw Duch, and Erkki Oja, editors, Artificial Neural Networks ICANN 2006, pages 632-640, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Generative adversarial set transformers. In Workshop on Object-Oriented Learning at ICML, volume 2020, 2020.
Dustin Tran, Keyon Vafa, Kumar Krishna Agrawal, Laurent Dinh, and Ben Poole. Discrete Flows: Invertible Generative Models of Discrete Data. Curran Associates Inc., Red Hook, NY, USA, 2019.
Tongwen Wu, Yu Yang, Yanzhi Li, Huiqiang Mao, Liming Li, Xiaoqing Wang, and Yuming Deng. Representation Learning for Predicting Customer Orders. Association for Computing Machinery, New York, NY, USA, 2021. ISBN 9781 45038333..
Pengcheng Yang. Fuli Luo, Shuming Ma. Junyang Lin, and Xu Sun. A deep reinforced sequence- to-set model for multi-label classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5252-5258. 2019.
Vacit Oguz Yazici, Abel Sonzalez-Garcia. Arnau Ramisa, Bartlomicj Twardowski, and Joost van de Weijer. Orderless recurrent models for multi-label classification. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 13440-13449, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] was described in Conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss the privacy risk.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proof' of all theoretical results? [Yes] All the proofs can be found in the supplemental material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] The standard deviation among 8 groups and detail results can be find in Table 3 and Table $ in the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All experiments were ran on CPU of 10 cores.
4. If you are using existing assets (e.g., code, data, models) or curating/releasngg new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use the MRW's code compare their results and we cite it.
(b) Did you mention the license of the assets? [Yes] the code license can be find Github
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] In the source code
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] The platforms (Tmall and HKTVMall) where we collect our data have already obtained such consent.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The datasets
already anonymized.
5. If you used crowdsourcing conducted research with human subjects....
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "eu82e41x_iW", "reviewer": "Reviewer_1KAT", "summary": "This paper focuses on the problem of modeling sets via sequence models via generative approaches. Mainly this paper considers all the possible state-action paths that would yield the same set and approximately marginalizes over them for MLE estimation. An approach similar to the popular \"REINFORCE\"/score-matching trick is proposed to define a Monte-Carlo based estimator for the gradient of the problem and importance sampling is done to efficient estimate the gradient for training. Two instantiations of the model are proposed -- GRU2set and SetNN. The first uses an autoregressive decoder's states to represent items in the sets, the second approach defines an equivariant MLP based function to represent items generated so far in the set. Size-bias trick is also introduced which biases the learning of set generation according to the size. The experiments are done on ecommerce order datasets, and compared with histogram based baselines and other relevant models", "questions": "How does “sparse item graph\" constructed form training data generalize to test sets?\nConfused about their definition of \"induce\"\n\n", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper tackles an important problem which would be of great interest to retro fit essentially sequnce-based architecture for modelling sets. The main approach is clearly explained and is sound.\n\nMy main concern is that the empirical results show that the simple histogram-based baseline outperforms most of the models. The proposed approach trails until it applies the size bias trick to see modest gains over the baseline. \n\nAnalysis of performance variation according to the size-wise variation of the set would improve the paper.\n\nWhy not use transformers? A simple transformer model can come close to modelling set properties by disabling positional embeddings. Comparison to transformers/SetTransofrmers is crucial given their ubiquity.\n\nRelated work: multilabel classification literature faces the same issues, a discussion about the approaches in this paradigm would be useful. For example: \"Orderless Recurrent Models for Multi-label Classification\", Yazici et al.\n\nThe paper is awkwardly written in parts – the size bias trick is very difficult to understand.\n\nLine 235/236 is very confusing", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "di1-o85LbNF", "reviewer": "Reviewer_uw8U", "summary": "The authors in this paper propose Sequence-to-Set that aims to convert any likelihood based sequence generative model to a set generative model. The proposed model first introduces a probability (likelihood) over a set $S$ induced from the generating path of the sequence generative model. Then an efficient importance sampling method is devised to tackle the training challenge of maximizing the set likelihood. The authors also present two instance of the proposed model, GRU2Set and SetNN to capture the complex relations among data points. Besides, a size-bias trick is designed to learn better set distribution from data. The proposed model is evaluated on two e-commerce order datasets, and the results show that the proposed model outperforms its baselines.", "questions": "(1) I don't quite understand the definition of the indicator function $\\tau$, is there any minor in $s_i=s \\land a_i = a$ ?\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper clearly identifies the problems with previous works and presents the idea of the proposed method.\n\n(2) The paper is well-written and easy to follow. \n\n(3) The paper provides a strong baseline for the set generative model, and several attempts can be tried base on the proposed method.\n\n(4) The experiments are well-designed.\n\n\nWeaknesses:\n\n(1) More deeper analysis about the experiments. for example, without Size-bias, the proposed model has different performance on the two datasets. The authors should give more discussion.\n\n(2) Empirically, the Size-Bias trick is the key for the proposed method and it can be used as a plug-and-play module to improve any generative model. It seems that the Size-Bias trick improves the proposed two models larger than baselines. The authors should give detailed explanation.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8PUNnLnldSC", "reviewer": "Reviewer_DMSE", "summary": "This paper tackles the challenge of developing generative models for sets, which have the property of permutation invariance. The authors first present a method for converting a sequence generative model to a set generative model based on maximum likelihood.  Additionally, they also propose an importance sampling algorithm to estimate the gradient of the log-likelihood function to improve the efficiency.  The authors further propose two parameterizations of the sequence-to-set model, which are GRU2Set and SetNN.  Finally, the authors propose a size-bias trick that can help reduce the distance between the ground truth set distribution and the learned distribution.\n\nEmpirical evaluations have been performed on two real-world datasets to convey the effectiveness of their method.", "questions": "1. The formats of the citations are incorrect in a few places including the very first sentence in the introduction, making the text hard to read.\n2. What are the authors boldfacing in the tables?\n3. It’s a bit hard for me to understand why the performance for the > 7 group is even better than the smaller sizes.  Learning the set distribution for more items is an easier problem?  What’s the intuition here.", "limitations": "No ethical concerns.  Other limitations discussed in earlier sections.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper studies an important and understudied problem and proposed an interesting method \n3. The empirical results are strong against baseline methods\n\nWeaknesses:\n1. The presentation of the paper is a bit hard to follow.  I would suggest the authors to illustrate the size-bias trick with a running example.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept