## 4 Model Description
### Self-Attentive Subset Learning
\[V_{s}=[\mathbf{p}_{u}^{\prime}\circ\mathbf{q}_{1}^{\prime},\dots,\mathbf{p}_{u}^{\prime} \circ\mathbf{q}_{|S|}^{\prime}]^{\top}.\]  
Following [34], we use scaled dot-product attention to calculate the self-attention score:  
\[Z_{s}=\text{softmax}\Bigg{(}\frac{V_{s}V_{s}^{\top}}{\sqrt{d}}\Bigg{)},\]  
where \(Z_{s}\in\mathbb{R}^{|S|\times|S|}\) is a self-attention score matrix which indicates the similarities among \(|S|\) items in set \(S\). \(\sqrt{d}\) is used to scale the attention score for preventing gradient vanishing. The softmax function is to normalize the self-attention scores. The output of the self-attention layer is the matrix product of \(Z_{s}\) with \(V_{s}\). To stabilize the training as [35], residual connections [36] are obtained as the final output:  
\[V_{s}^{\prime}=V_{s}+Z_{s}V_{s}.\]  
**Policy network. ** After transforming item embedding through the self-attention layer, we obtain the set-aware item embeddings \(\mathbf{v}_{s}^{\prime}\). To learn the user's personalized subset selection strategy, the item embeddings are concatenated with the user embedding, i.e., \([\mathbf{p}_{u}^{\prime};\mathbf{v}_{s}^{\prime}]\). Then, the concatenated embeddings are fed into the policy network. Typically, a policy network outputs the probability distribution of all possible actions. In our case, it gives the probability of being chosen in the subset, \(p(x_{ui}=1)\), or not, \(p(x_{ui}=0)\). Then, the policy network selects items by sampling from the probabilities. However, the sampling of items involves discrete variables, so that the gradient is unable to propagate backwards. Another straightforward solution is to rely on a naive Monte-Carlo gradient estimator [37]. The naive Monte-Carlo gradient estimator samples the discrete variable \(x_{uj}\) from \(p(x\mid\mathbf{p}_{u},\mathbf{z}_{i})\) multiple times to estimate the gradient. It can be understood as the REINFORCE algorithm [38] with a single decision. However, as pointed out in [39], the naive Monte-Carlo gradient estimator exhibits very high variance, which destabilizes the sampling process. To reduce the variance, the reparameterization trick is generally utilized [39]. Rather than sampling from the data-dependent distribution \(p(x\mid\mathbf{p}_{u},\mathbf{z}_{j})\), reparameterization samples a random variable \(\epsilon\) from the data-independent distribution, say, a standard normal distribution. Then, variable \(x_{uj}\) is sampled by a function involving both \(\epsilon\) and \(\mathbf{p}_{u},\mathbf{z}_{j}\). Since the sampled \(\epsilon\) is independent and identically distributed, the variance during estimation can be largely reduced. For reparameterizing categorical variables in our case,Gumbel-softmax is often utilized [33]. Given the probabilities \(p(x_{uj}=1)=\pi_{uj}\) and \(p(x_{uj}=0)=1-\pi_{uj}\), Gumbel-softmax performs reparameterization in the following way:  
\[\epsilon \sim\text{Uniform}(0,1),\] \[g =-\log(-\log(\epsilon)),\] \[x_{uj} =\text{softmax}\big{(}(\log(\pi_{uj})+g)/\tau\big{)}.\]  
Note that \(x_{uj}\) is not exactly binary but close to zero or one. \(\tau\), named \"temperature\", is the hyper-parameter for controlling to what extent \(x_{uj}\) is close to binary. At this point, we get the personalized optimal subset from the self-attentive subset learning module. By doing so, we model the pattern how item-level preferences influence users' set-level feedback. It is the key to address the SPL problem by inferring item-level feedback from set-based preferences.