Title: Gaussian Process Bandits for Top-k Recommendations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian processes

Bandit algorithms

Top-k recommendations

Linear Algebra

Iterative Algorithms
Abstract: Algorithms that utilize bandit feedback to optimize top-k recommendations are vital for online marketplaces, search engines, and content platforms.  However, the combinatorial nature of this problem poses a significant challenge,  as the possible number of ordered top-k recommendations from $n$ items grows exponentially with $k$. As a result, previous work often relies on restrictive assumptions about the reward or bandit feedback models, such as assuming that the feedback discloses rewards for each recommended item rather than a single scalar feedback for the entire set of top-k recommendations. We introduce a novel contextual bandit algorithm for top-k recommendations, leveraging a Gaussian process with a Kendall kernel to model the reward function.
Our algorithm requires only scalar feedback from 
the top-k recommendations and does not impose restrictive assumptions on the reward structure. 
Theoretical analysis confirms that the proposed algorithm achieves sub-linear regret in relation to the number of rounds and arms. Additionally, empirical results using a bandit simulator demonstrate that the proposed algorithm outperforms other baselines across various scenarios.

Gaussian Process Bandits for Top-k Recommendations

Abstract

Algorithms that utilize bandit feedback to optimize top-k recommendations are vital for online marketplaces, search engines, and content platforms. However, the combinatorial nature of this problem poses a significant challenge, as the possible number of ordered top-k recommendations from n items grows exponentially with k . As a result, previous work often relies on restrictive assumptions about the reward or bandit feedback models, such as assuming that the feedback discloses rewards for each recommended item rather than a single scalar feedback for the entire set of top-k recommendations. We introduce a novel contextual bandit algorithm for top-k recommendations, leveraging a Gaussian process with a Kendall kernel to model the reward function. Our algorithm requires only scalar feedback from the top-k recommendations and does not impose restrictive assumptions on the reward structure. Theoretical analysis confirms that the proposed algorithm achieves sub-linear regret in relation to the number of rounds and arms. Additionally, empirical results using a bandit simulator demonstrate that the proposed algorithm outperforms other baselines across various scenarios.

1 Introduction

The topk recommendation problem involves providing a ranked list of k items, such as news articles or products, from a pool of n items [34, 13]. Online algorithms must adapt to dynamic user preferences, making bandit algorithms suitable due to their use of limited feedback [1]. Developing bandit algorithms is challenging due to limited feedback and the need for computational efficiency in real-time recommendation environments. Recent research on user interfaces for recommendations highlights that the overall layout of the recommendation page is crucial for user appeal, as modern UI designs have evolved from simple dropdown lists to complex, visually engaging layouts [17, 13, 18]. Consequently, bandit algorithms must jointly select and display all topk items, rather than simply choosing the most relevant k items and ordering them by decreasing user relevance [31].

The joint consideration of topk items makes the number of arms (possible actions for the bandit algorithm) combinatorially large, i.e., Θ( n k ) . Previous research on bandit algorithms often imposes strict assumptions on feedback models [30, 21]. For instance, semi-bandit feedback provides a scalar reward for each of the top k items, thus decomposing the combinatorial feedback into item-level feedback. However, this type of feedback is frequently unavailable [32]. Another common feedback model is cascade browsing [16], which assumes that users examine items in a predetermined order and cease browsing once a desirable item is found, offering item-specific scalar feedback but failing to capture potential non-linear interactions among items [26]. Figure 1 illustrates the limitations of the cascade model in capturing user interactions within modern topk recommendation interfaces.

These limitations motivate us to adopt a more general full-bandit feedback setting, where only a single scalar value is provided for the entire topk set of recommendations [24].

Beyond feedback assumptions, the reward structure in bandit algorithms must be decomposable into scalar values for individual items to prevent a combinatorial explosion of arms-something that is not always feasible. For example, modern e-commerce platforms value and track metrics such as diversity and fairness [1], which cannot be captured by focusing solely on individual items [15]. This necessitates algorithms for full-bandit feedback settings that operate without specific assumptions about the objectives or reward structures [24].

This work introduces a bandit algorithm that uses Gaussian processes (GPs) to model rewards under full-bandit feedback (i.e., a single scalar value). GPs are selected for their flexibility in accommodating feedback across discrete, continuous, and mixed domains, such as continuous contexts and discrete rankings [33]. Additionally, unlike parametric models that require optimization incorporating accumulated feedback from previous rounds, GP updates are computationally efficient, involving only data updates [24]. Although GP inference may face computational limits, we will develop efficient inference methods tailored to our proposed algorithm. A further challenge in designing GP-based bandit algorithms for topk recommendations is constructing expressive positivedefinite kernels that capture similarities between topk recommendations [9]. This work mitigates these computational and modeling challenges, as illustrated in the following sections.

Broadly speaking, GPs have been previously explored for bandit algorithms [27, 19]. Krause et al. [14] employed GPs for contextual bandits in continuous domains; we focus on the discrete domain of topk recommendations. Vanchinathan et al. [28] used GPs with a position-based feedback model, and Wang et al. [31] used GPs with semi-bandit feedback for recommending topk items. In contrast, our work does not rely on a specific reward model or feedback assumption, and develops an an efficient GP-based bandit algorithm for topk recommendations.

1.1 Contributions

Our primary contribution is the GP-TopK algorithm, a contextual bandit algorithm for recommending topk items. This algorithm operates in a full-bandit feedback setting without relying on assumptions on reward, making it broadly applicable compared to prior works. We leverage GPs with variants of the Kendall kernel [12] to model the reward function and optimize the upper confidence bound (UCB) [27] acquisition function for selecting the next arm. Additionally, we introduce a novel weighted convolutional Kendall kernel for topk recommendations that address pathologies in existing variants of the Kendall kernel when applied to topk recommendations.

Our second key contribution is enhancing the scalability of the GP-TopK algorithm for longer time horizons. Initially, the computational cost for topk recommendations using the GP-TopK algorithm is O ( T 4 ) for T rounds. We first reduce this to O ( T 3 ) by leveraging iterative algorithms from numerical linear algebra [25]. Next, we derive sparse feature representations for the novel weighted

convolutional Kendall kernel, further reducing the compute requirements from O ( T 4 ) to O ( T 2 ) and memory requirements from O ( T 2 ) to O ( T ) . Table 1 summarizes these improvements in time and memory requirement, including their dependence on other parameters.

We provide a theoretical analysis showing that GP-TopK's regret is sub-linear in T , benefiting from the feature representations of the Kendall kernels introduced in this work. Specifically, we establish an upper bound on regret that is nearly quadratic in n , significantly improving over the naive Θ( n k ) bound for topk recommendations without using feature representations [27]. Finally, we empirically validate GP-TopK's regret on real-world datasets, demonstrating improvement over baseline methods.

1.2 Organization

The remainder of this paper is organized as follows: Section 2 introduces Kendall kernels for full and topk rankings, including the novel weighted convolutional Kendall kernel. Section 3 presents faster matrix-vector multiplication algorithms for Kendall kernels, enhancing the efficiency of the proposed bandit algorithm, which is further detailed along with the regret analysis in Section 4. Finally, Sections 5 and 6 present empirical results and concluding discussion, respectively.

2 Kendall Kernels for Full and Top-k Rankings

This section introduces Kendall kernels and their extensions for topk recommendations, forming the foundation of our approach. We first establish key notations and them present Sections 2.1 and 2.2, which introduce Kendall kernels for full rankings and topk rankings, respectively.

Notations: Let [ n ] = 1 , 2 , . . . , n , with π representing a topk ranking-an ordered tuple of k distinct elements from [ n ] . For a full ranking ( k = n ), we use σ and denote the set of all possible topk rankings by Π k , with cardinality | Π k | = Θ( n k ) . To capture ranking positions, the vector p σ ∈ R n corresponds to a full ranking σ with entry p σ i gives the rank of item i . For top-k rankings, p π ∈ R n is similarly constructed by arbitrarily assigning distinct ranks to items not in the top k . For relative ranks, indicator functions p σ i<j and p σ i>j denote whether item i is ranked before or after item j , respectively in σ . Also, p π i<j and p π i>j are similar indicator functions defined for top-k rankings.

2.1 Kendall Kernels for Full Rankings

Jiao et al. [9] showed that the Kendall tau rank correlation coefficient [12] is a positive definite (p.d.) kernel for full rankings, which we refer to as the standard Kendall (SK) kernel. The weighted Kendall (WK) kernel generalizes the SK kernel by differentially weighting item pairs [10]. Specifically, the SK and WK kernels for full rankings σ 1 , σ 2 are defined as:

where η i,j is 1 if the pair ( i, j ) is concordant (ordered the same in both rankings) and -1 otherwise; concretely, η i,j ( σ 1 , σ 2 ) := p σ 1 i<j · p σ 2 i<j + p σ 1 i>j · p σ 2 i>j -p σ 1 i<j · p σ 2 i>j -p σ 1 i>j · p σ 2 i<j ; and w (( p σ 1 i , p σ 1 j ) , ( p σ 2 i , p σ 2 j )) is the value of a positive definite weighting kernel w ( · , · ) : [ n ] 2 × [ n ] 2 ↦→ R that operates on pairs of ranks. The w i,j adds flexibility and can assign varying importance to ranks, similar to the discounted cumulative gain (DCG) metric [7]. Note that both SK and WK kernels are p.d. and right-invariant with respect to Π n [10]. In other words, they compute similarity based only on the relative ranks of pairs, not on the labels of items, as clearly evident from Equations 1 and 2.

2.2 Kendall Kernels for Top-k Rankings

Weighted Kendall and Convolutional Kendall (CK) kernels. To adapt the WK kernel from full rankings to topk rankings, Jiao et al. [10] set the weighting function w ( i, j, σ 1 , σ 2 ) to zero if either item is not in the top-k of either ranking. While this approach yields a p.d. kernel, it disregards

items outside the intersection of topk rankings. In contrast, the convolutional operation provides an alternative for adapting the standard Kendall kernel to topk rankings.

Let B π denote the set of full rankings consistent with the top-k ranking π (i.e., for every item i in π , ∀ σ ∈ B π , p π i = p σ i ). The convolutional Kendall kernel can be defined as follows:

where k sk is the standard Kendall kernel. Since the CK kernel is a convolution of another p.d. kernel, it is also a p.d. kernel [5]. Unlike the WK kernel for top-k rankings, the CK kernel accounts for items not in both top-k rankings. However, computing the CK kernel is expensive, requiring exponentially many evaluations of the kernel k sk in the double summation. Therefore, Jiao et al. [9] developed an efficient algorithm to bypass this double summation, reducing compute to O ( k log k ) time.

Proposed Weighted Convolutional Kendall (WCK) Kernel. To combine the strengths of the WK and CK kernels for top-k rankings, we propose the weighted convolutional Kendall kernel for top-k rankings π 1 and π 2 ∈ Π k :

where k wk represents the weighted Kendall kernel for full rankings σ 1 , σ 2 ∈ Π n .

The proposed WCK kernel combines the flexibility of differentially weighting ranks among the topk items (as in the WK kernel) with the ability to account for items outside the intersection of both topk rankings (as in the CK kernel). Additionally, as a convolution of a p.d. kernel, it is also a p.d. kernel. However, computing the WCK kernel remains challenging, as it requires exponentially many evaluation of the k wk kernel, as given in the RHS of Equation 4. To address this, we focus on a specific form of rank weights of the k wk kernel, called as product-symmetric rank weights:

where, w s ( i, j ) : [ n ] × [ n ] ↦→ R is a symmetric function, i.e., w s ( i, j ) = w s ( j, i ) . Notably, the WCK kernel can be computed efficiently for the case of these weights (see Claim 1 below).

The WCK kernel, even with the relatively simple w ps weights, exhibits notable properties, as shown in Table 2. In this table, we use w s ( i, j ) = 1 log( i +1) · 1 log( j +1) , inspired by the DCG metric commonly applied in recommendation systems [7]. Notably, the WKkernel ranks two rankings with no overlap ( π 0 and π 1 ) as more similar than two rankings with the same items in reversed order ( π 0 and π 2 ), indicating a clear pathology. Further, the CK kernel fails to distinguish between reversed pairs at different ranks ( k ck ( π 0 , π 3 ) = k ck ( π 0 , π 4 ) ), presenting another limitation if known variants of Kendall kernels for top-k rankings. By using product-symmetric ranking weights, the WCK kernel addresses these shortcomings, providing a more nuanced similarity comparison for topk rankings.

Table 2: Comparison of Kendall kernel similarities for topk rankings. The table shows kernel values k ( π 0 , · ) for the top-k ranking π 0 = [1 , 2 , 3] with other rankings ( π 1 , π 2 , π 3 , π 4 ) for n = 7 and k = 3 . Rankings are arranged left to right by increasing similarity to π 0 . The similarity values provided by the proposed kernel increase from left to right as expected, demonstrating the desirable behavior of the WCKkernel with DCG rank weights, unlike other variants. All kernels are unit-normalized. See text for further details.

Claim 1. The weighted convolutional Kendall kernel (Equation 4) with product-symmetric rank weights (Equation 5) can be computed in O ( k 2 ) time.

Appendix A provides the proof that leverages the structure of product-symmetric rank weights w ps to establish the existence of a feature representation for the WCK kernel, as formally stated below in

Claim 3 below. We then demonstrate that the inner product of these features, and hence the WCK kernel, can be computed in O ( k 2 ) time (Algorithm 2 in the appendix). Similar to the result of Jiao et al. [9] for the CK kernel, this approach avoid exponentially many evaluations of k wk on the RHS of Equation (4) by enabling a direct computation of the WCK kernel.

3 Fast Matrix-Vector Multiplication with Kendall Kernel Matrices

In Gaussian processes, inference can be accelerated by using iterative algorithms that take advantage of fast matrix-vector-multiplications (MVMs) with the kernel matrix [3]. This section introduces fast algorithms for kernel MVMs by exploiting the implicit structure of Kendall kernel matrices.

Let mvm ( K X t ) denote the runtime required to multiply the t × t kernel matrix K X t = ( k ( x i , x j )) x i ,x j ∈ X t by any admissible vector. In the naive approach, this runtime is mvm ( KX t ) = O ( t 2 ) . However, if k ( x i , x j ) = ϕ a ( x i ) T ϕ b ( x j ) for any arbitrary x i and x j , where the vectors ϕ a ( x i ) and ϕ b ( x j ) are sparse and contain only z non-zero entries, then mvm ( K X t ) reduces to O ( z · t ) , which is a significant improvement over O ( t 2 ) when z ≪ t . When ϕ a = ϕ b , we refer to ϕ a as the linear feature vector for the kernel k . Before focusing on top-k ranking kernels, we provide a linear feature vector for the WK kernel on full rankings (given earlier in Equation 2).

Claim 2. Let ϕ wk ( σ ) : Π n ↦→ R ( n 2 ) be a vector indexed by unique item pairs ( i, j ) , defined as:

where w s is the symmetric weighting function in product-symmetric weights. Then, ϕ wk is a linear feature vector for the weighted Kendall kernel with product-symmetric weights w ps .

Using Claim 2, the linear feature vector for the WK kernel can be extended to the WK topk ranking kernel by utilizing the structure of product-symmetric weights, which allows weights to be set to zero for items outside of the topk rankings, as described in Section 2.2. Precisely, such a feature vector for the topk ranking kernel is sparse; specifically, the feature vector ϕ wk ( π ) contains only O ( k 2 ) non-zero entries due to the WK kernel's focus on item pairs within the topk . Consequently, the runtime for mvm ( K X t ) in the WK kernel matrix is reduced to O ( k 2 · t ) .

Moving forward, we focus on deriving a sparse feature vector for the WCK kernel, enabling fast MVMs with the WCKkernel, which includes the CK kernel as a special case. Notably, any convolutional kernel inherits linear features from its constituent kernel. Specifically, ∑ σ ∈ B π ϕ wk i,j ( σ ) forms a feature vector for the WCK kernel, which follows from Equation 4 and However, computing this feature vector explicitly is computationally challenging, as it requires summing over all σ ∈ B π , which includes an exponential number of terms, i.e., Θ( n k ) .

In response to this challenge, Claim 3 shows that the summation can be computed analytically and provides explicit linear feature vectors for the WCK and CK kernels. It also shows that ϕ wck has only O ( k 2 +2 nk ) nonzero entries among its O ( n 2 ) total entries. Consequently, mvm ( K X t ) for

Claim 3. Let ϕ wck ( π ) : Π k ↦→ R ( n 2 ) be a vector indexed by unique item pairs ( i, j ) given as: ϕ wck i,j ( π ) := 1 √ ( n 2 ) · w wck i,j ( π ) · ( p π i<j -p π i>j ) , where w wck i,j ( π ) is determined as follows:

where w s denotes symmetric weights and w s ( ℓ, · ) = 1 n -k ∑ n j = k +1 w s ( ℓ, j ) . Then, the vector ϕ wck is a linear feature vector for the WCK kernel k wck . By uniformly setting w s ( · , · ) ≡ 1 in the definitions above, ϕ wck i,j ( π ) specializes to a linear feature vector for the CK kernel.

the WCK kernel requires O (( k 2 +2 nk ) · t ) operations, which improves from O ( t 2 ) to linear in t . However, this introduces a dependence on n , the number of items, which poses a serious limitation and is beneficial only when n ≤ t . In the following theorem, we leverage redundancy in ϕ wck to eliminate this dependence on n , leading to the following main theorem about the mvm ( K X t ) .

Theorem 1. For the WCK kernel with product-symmetric weights w ps , the computational complexity of multiplying the kernel matrix K X t with any admissible vector is O ( k 2 t ) , i.e., mvm ( K X t ) = O ( k 2 t ) , where X t is any arbitrary set of t top-k rankings.

Appendix A provides the proof in two steps. First, we utilize the values of ϕ wck from Claim 3 and categorize ϕ wck ( π 1 ) T ϕ wck ( π 2 ) based on item pairs, as summarized in Table 4. Next, we show that only five combinations yield non-zero values, i.e., ϕ wck ( π 1 ) T ϕ wck ( π 2 ) = ∑ 5 i =1 s i ( π 1 , π 2 ) . Each term s i ( π 1 , π 2 ) is a dot product of vectors ϕ a i ( π 1 ) T ϕ b i ( π 2 ) , which contains at most O ( k 2 ) non-zero entries. Thus, for the WCK and CK kernels, mvm ( K X t ) = O ( k 2 t ) , since these vectors across all five terms include only O ( k 2 ) non-zero entries. Consequently, Theorem 1 demonstrates that employing these vector representations for top-k rankings leads to faster MVMs, i.e., mvm ( K X t ) = O ( k 2 t ) ≪O ( t 2 ) .

4 Proposed GP-TopK Bandit Algorithm

In this section, we begin by formally defining the topk recommendation problem within a bandit framework and introduce a generic contextual bandit algorithm, detailed in Algorithm 1. We then explain how the components of the algorithm are instantiated using the proposed GP approach, followed by an analysis of its computational complexity and cumulative regret.

Let T denote the number of rounds. Contexts C are represented in a finite c -dimensional space, i.e., C ⊆ R c . In the t th round, we receive a context c t ∈ C and select a topk ranking π t ∈ Π k . Subsequently, a noisy reward y t = ˆ f ( c t , π t ) + ϵ t is observed, where ˆ f is the true reward function and ϵ t is round-independent noise. The regret is defined as r t := max π ′ ∈ Π k ˆ f ( c t , π ′ ) -ˆ f ( c t , π t ) , with cumulative regret R T := ∑ T t =1 r t . The accumulated data at the t th round is D t = ( c i , π i , y i ) t i =1 . Below, the Algorithm 1 provides provides a generic schematic of the bandit algorithm.

Algorithm 1 Contextual Bandit Algorithm for Top-k Recommendations

Input: Total rounds T , initial reward model M 0 , and acquisition function AF .

1: for t = 1 , · · · , T do

2: Observe a context c t from the context space C .

3: Select a topk ranking π t that maximizes AF ( M t -1 ( c t , π )) for the context c t .

4: Obtain the scalar reward y t .

5: Update the reward model M t using the accumulated feedback D t .

6: end for

We aim to design the components of above Algorithm 1 with the objectives of minimizing cumulative regret and ensuring computational efficiency. It requires two key components: (a) a reward model M t that estimates the reward for any context and topk ranking utilizing the accumulated data D t and (b) an acquisition function AF for selecting π t given the reward model M t and observed context c t .

Reward model M and acquisition function AF . The proposed GP-TopK bandit algorithm leverages GP regression to model the reward function over the domain of contexts and top-k rankings. Section B.1 briefs GP regression for the completeness. Essentially, the reward model M maintains a distribution over functions f , i.e., f ∼ N (0 , k ( · , · )) , where k is a product kernel function over both contexts and top-k rankings ( C ⊗ Π k ). Specifically, the kernel function k is defined as follows:

where k c ( c 1 , c 2 ) = c T 1 c 2 is the dot-product kernel and k r is a kernel for top-k rankings. We use variants of the Kendall kernel for k r from Section 2. Updating the reward model M t at the t th

round involves adding new data points to our GP regression, which is computationally inexpensive compared to the fine-tuning steps required by parametric models to incorporate the latest feedback.

We use the UCB function as the acquisition function, balancing exploration and exploitation by selecting actions that maximize the upper confidence bound of the estimated reward [27]. The UCB acquisition function is AF ( M t ( c t , π )) := µ f |D (( c t , π ))+ β 1 2 · σ f |D (( c t , π )) , where σ f |D (( c t , π )) = √ k f |D (( c t , π ) , ( c t , π )) and β controls the trade-off between exploration and exploitation. Here, µ f |D and k f |D are the GP posterior mean and covariance functions, as detailed in Section B.1. At the t th round, the algorithm selects the top-k ranking π ∈ Π k that maximizes AF ( M t ( c t , π )) , which is performed using local search [19], as detailed further in Appendix B.

Computational complexity. The GP-TopK bandit algorithm does not require compute for model updates. In other words, updating M t , i.e., in the Line 5 of the Algorithm 1 requires only updating the list of accumulated feedback data D t . The GP-TopK relies on local search to optimize AF , so the computational demands stem solely from AF evaluations within the local search. As shown in Section B.1, computing the GP variance term for evaluating AF , i.e, σ (( c , π )) involves solving

[ K X t + σ 2 I ] v for a vector v , where X t = [( c 1 , π 1 ) , · · · , ( c t , π t )] . Naively, this operation requires O ( t 3 ) time per round, amounting to total O ( T 4 ) over T rounds. Iterative algorithms, however, can expedite the process by leveraging fast MVMs with kernel matrices, as discussed in Section 3. Below, Theorem 2 formalizes the computational demands of the GP-TopK algorithm.

f |D t -1

Theorem 2. Assuming a fixed number of iterations required by the iterative algorithms, the total computational time for running the GP-TopK bandit algorithm for T rounds of top-k recommendations, using the contextual product kernel (Equation 6), is O ( k 2 cℓT 2 ) . This applies to WK, CK, and WCK top-k ranking kernels, where ℓ is the number of local search evaluations.

The proof of Theorem 2, provided in Appendix B, demonstrates efficiency gains from combining feature representations with iterative algorithms, reducing computational time from O ( T 4 ) to O ( T 2 ) . This is a substantial improvement, as even a single MVM with the matrix K X t using the full kernel matrix at each round would require O ( T 3 ) compute time. Additionally, the theorem shows that the running time of the GP-TopK algorithm does not explicitly depend on the number of items n .

Regret analysis. The cumulative regret is R T = ∑ T t =1 max π ′ ∈ Π k ˆ f ( c t , π ′ ) -ˆ f ( c t , π t ) , where π t is the ranking chosen at round t . Optimizing cumulative regret for topk recommendations is challenging, as it requires learning the context-arm relationship and matching the best possible mapping. To bound cumulative regret, regularity assumptions are essential, as noted in prior works [27, 14]. We consider the following two assumptions, either of which suffices . Also, X := C ⊗ Π k for below assumptions.

Assumption 1. X is finite, meaning that only finite contexts are considered ( |C| < ∞ ), and the reward function ˆ f is sampled from the GP prior with a noise variance of ξ 2 .

Assumption 2. X is arbitrary and the reward function ˆ f has a bounded RKHS norm for the kernel k , i.e., ∥ f ∥ k ≤ B . The reward noises ϵ t form an arbitrary martingale difference sequence (i.e., reward noise does not systematically depend on its past values) and are uniformly bounded by ξ .

The following theorem proves the regret bound for the GP-TopK algorithm under Assumption 1 or 2.

Theorem 3. If either Assumptions 1 or 2 hold, setting β t as 2 log ( |C|·| Π k |· t 2 · π 2 6 δ ) and 300 γ t ln 3 ( t δ ) respectively, the cumulative regret R T of the GP-TopK bandit algorithm for top-k recommendations can, with at least 1 -δ probability, be bounded by ˜ O ( n √ C 1 Tc (log |C| + k +log( T 2 π 2 / 6 δ ))) under Assumption 1, and ˜ O ( n √ C 1 (2 B 2 c +300 n 2 c 2 ln 3 ( T/δ )) T ) under Assumption 2. Here, C 1 = 8 log(1+ ξ -2 ) , and ˜ O excludes logarithmic factors related to n , k , and T .

Appendix B.4 provides the proof, leveraging the insight that log det | I + ξ -2 · K X T | for any set X T can be effectively bounded using the finite-dimensional feature vectors introduced in this work.

Specifically, Proposition 2 utilizes the feature vectors from Section 2. Building on Proposition 2, Theorem 3 establishes that the cumulative regret of the GP-TopK bandit algorithm grows sublinearly in T with high probability for both assumptions. Furthermore, this result also underscore the importance of using top-k ranking kernels, which improve the asymptotic order in terms of n by factors of n k/ 2 -1 and n k -1 under Assumptions 1 and 2, respectively, compared to Srinivas et al. [27]. This improvement is substantial even for small values of k , such as k = 6 , as shown in Table 3.

5 Experiments

This section empirically evaluates the proposed GP-TopK bandit algorithms for the top-k recommendations using a simulation based on the MovieLens dataset [4]. The reliance on simulation for evaluating bandit algorithms is prevalent in the literature. It stems from the difficulty of conducting online evaluations in real-world bandit scenarios, mainly when there are combinatorial arms [28]. Next, we provide details of the simulation setup and considered reward settings. Following that, we present results for the empirical regret for small and large numbers of arms below, respectively.

Simulation setup and reward settings. The bandit simulation setup follows the framework outlined by Jeunen et al. [8], utilizing real-world datasets on user-item interactions. Specifically, we train user and item embeddings using a collaborative filtering approach [6]. The user embeddings are accessed by the bandit algorithms as context embeddings, while the item embeddings remain hidden. In the non-contextual setup, the first user from the dataset is chosen as a fixed context throughout the bandit algorithm run, allowing us to use the same reward functions as the contextual bandit algorithm.

For setting up the reward functions, we utilize a similarity function s ( c , θ ) := ς ( a · ( c T θ ) -b ) to measure similarity between any user and item embeddings, where a and b are similarity score and shift scalars, respectively. The sigmoid function ς maps similarity scores to a range between 0 and 1 , enhancing the interpretability of the reward signal [31]. We set a and b to 6 and 0 . 3 , respectively, to fully utilize the range of the similarity function, as assessed by evaluating its value for many arms.

We set up two preliminary reward functions based on the similarity function s . The first is the DCG metric, ˆ f dcg ( c , π ) = ∑ k i =1 1 log 2 ( i +1) s ( c , θ π i ) , where c and θ π i represent the context and item embeddings, respectively. The second is the diversity measure, ˆ f div ( π ) = 1 k 2 ∑ k i =1 ∑ k j =1 θ T π j θ π i . These metrics quantify the relevance and diversity of top-k recommendations, respectively.

We use these functions in two contextual reward settings. The first setting focuses on normalizedDCG (n-DCG), ˆ f ndcg ( c , π ) = ˆ f dcg ( c ,π ) max π ′ ˆ f dcg ( c ,π ′ ) [7]. The second setting combines ˆ f ndcg and ˆ f div as ˆ f ndcgdiv ( c , π ) = λ · ˆ f ndcg ( c , π ) + (1 -λ ) · ˆ f div ( π ) , evaluating the aggregate effect of relevance and diversity. We set λ = 0 . 75 to emphasize relevance over diversity.

Evaluation for small arm space. This section presents empirical results for the cumulative regret of bandit algorithms with a limited number of arms. Specifically, with n = 20 and k = 3 , there are 6 , 840 top-k rankings, allowing for an exhaustive search to optimize the acquisition function. All bandit algorithms run in batch mode, updating every five rounds. We consider both reward settings for contextual and non-contextual scenarios, using a subset of five users for the contextual setting.

Several baselines are set to assess the benefits of ranking (Kendall) kernels. Section C details the remaining hyper-parameter configurations and details of other baseline bandit algorithms.

0

100

200

T

0

200

400

RT

Random

-greedy

MAB

WK

CK

WCK

(a)

0

100

200

T

(b)

0

100

200

T

(c)

0

100

200

T

0

150

300

(d)

The Random algorithm randomly recommends any k items. The ϵ -greedy algorithm alternates between recommending a random top-k ranking with a probability of ϵ and choosing the top-k ranking with the highest observed mean reward. In contextual settings, ϵ -greedy differentiates arms for each unique context. Similarly, MAB-UCB conceptualizes each ranking as an independent arm, an equivalent of using a direct delta kernel approach for GPs along with UCB AF . In contextual scenarios, MAB-UCB also treats arms distinctly per context. Each variant of the top-k ranking kernel yields one variation of the proposed GP-TopK algorithm, namely, WK, CK, and WCK. Figure 2 presents empirical values of the cumulative regrets for the above baseline and the proposed GP-TopK algorithms. In all cases, across both reward settings and in both contextual and non-contextual setups, the variants of the proposed GP-TopK algorithm outperform baselines that do not use Kendall kernels, highlighting the significance of top-k ranking kernels for full bandit feedback. Specifically, the CK and WCK kernels significantly outperform the WK kernel regarding the converged values of the regret, with the WCK kernel further improving on the CK kernel variant.

Evaluation for large arm space.

We evaluate bandit algorithms in a large arm space scenario with n = 50 and k = 3 and k = 5 , resulting in 1 . 1 × 10 5 and 1 . 1 × 10 10 possible top-k rankings, respectively. Using local search, we focus on the nDCG reward. The remaining configuration is consistent with the small arm space setup. We use 10 restarts and 5 steps in each search direction for the local search, starting with 1000 initial candidates.

Figure 3 shows that the regret for the GP-TopK variants remains consistently lower even with a large arm

0

50

100

0

800

1600

RT

Random

-greedy

MAB

WK

CK

WCK

0

50

100

T

T

space, despite the use of local search. The WCK approach significantly outperforms the CK, especially for k = 5 , as illustrated in the right plot of Figure 3. Additional empirical results on the effectiveness of local search in a large arm space and other rewards are given in Appendix C.

6 Discussion

This work develops a contextual bandit algorithm for topk recommendations using Gaussian processes with Kendall kernels in a full-bandit feedback setting, without restrictive assumptions about feedback or reward models. Gaussian processes provide computationally efficient model updates

for accumulated feedback data, although inference can be challenging. We address this by deriving features for Kendall kernels tailored to topk rankings, resulting in a faster inference algorithm that reduces complexity from O ( T 4 ) to O ( T 2 ) . While demonstrated here for the product kernel between contexts and topk rankings, these computational improvements extend naturally to other kernel types, such as additive kernels. Additionally, we address limitations of known variants and propose a more expressive Kendall kernel for topk recommendations. Finally, we provide both theoretical and empirical results demonstrating the improved performance of the proposed GP-TopK algorithm.

Future Directions and Limitations. This work opens several research avenues. Efficient matrixvector multiplication with Kendall kernel matrices can enable faster bandit algorithms with various acquisition functions, such as Thompson sampling and expected improvement. Exploring other kernels, like Mallow kernels, for top-k rankings and developing efficient algorithms for them is an intriguing direction, especially since the effectiveness of our algorithm depends on the function space induced by the RKHS of the underlying kernel. Assessing how well these kernels approximate various reward functions for top-k recommendations would provide valuable insights.

Exploring other bandit problem settings, such as stochastic item availability or delayed feedback, would enhance the applicability of this work to more complex scenarios. Extending the finitedimensional GP framework to other acquisition functions using local search is another promising direction. One limitation of our regret analysis is that it does not account for approximations in the arm selection step due to local search [20]. This limitation is common in continuous domains, where optimizing acquisition functions often involves non-convex optimization [27].

Impact. This research advances bandit algorithms for top-k item recommendations. By improving recommendation efficiency and accuracy, our algorithms can enhance user experiences across platforms, promoting content relevancy and engagement. However, they may reinforce implicit biases in training data, limiting content diversity and entrenching prejudices. Therefore, monitoring over time is essential when deploying these algorithms in real-world environments.

References

Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, and Shie Mannor. Multi-objective bandits: Optimizing the generalized gini index. In Proceedings of the 34th International Conference on Machine Learning (ICML) , pages 625-634. PMLR, 2017.

Aryan Deshwal, Syrine Belakaria, Janardhan Rao Doppa, and Dae Hyun Kim. Bayesian optimization over permutation spaces. In Proceedings of the 25th Conference on Artificial Intelligence (AAAI) , pages 6515-6523, 2022.

Jacob R. Gardner, Geoff Pleiss, David Bindel, Kilian Q. Weinberger, and Andrew Gordon Wilson. GPyTorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. Advances in the 31st Neural Information Processing Systems (NeurIPS) , pages 7576-7586, 2018.

F Maxwell Harper and Joseph A Konstan. The Movielens datasets: History and context. In Transactions on Interactive Intelligent Systems , volume 5, pages 1-19. ACM, 2015.

David Haussler. Convolution kernels on discrete structures. Technical report, Technical report, Department of Computer Science, University of California Santa Cruz, 1999.

Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets. In 8th IEEE International Conference on Data Mining (ICDM) , pages 263-272. IEEE, 2008.

Kalervo Jarvelin and Jaana Kekalainen. Cumulated gain-based evaluation of ir techniques. In ACM Transactions of Information Systems , volume 20, pages 422-446, 2002.

Olivier Jeunen and Bart Goethals. Top-k contextual bandits with equity of exposure. In Proceedings of the 15th ACM Conference on Recommender Systems (RecSys) , pages 310-320, 2021.

Yunlong Jiao and Jean-Philippe Vert. The Kendall and Mallows kernels for permutations. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , pages 1935-1944. PMLR, 2015.

Yunlong Jiao and Jean-Philippe Vert. The weighted Kendall and high-order kernels for permutations. In Proceedings of the 35th International Conference on Machine Learning (ICML) , volume 80, pages 2314-2322. PMLR, 2018.

Michael N. Katehakis and Arthur F. Veinott. The multi-armed bandit problem: Decomposition and computation. In Mathematics of Operations Research , volume 12, pages 262-268, 1987.

Maurice G Kendall. A new measure of rank correlation. Biometrika , 30(1/2):81-93, 1938.

Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. Explaining the user experience of recommender systems. In User Modeling and User-Adapted Interaction , volume 22, pages 441-504, 2011.

Andreas Krause and Cheng Ong. Contextual gaussian process bandit optimization. Advances in the 24th Neural Information Processing Systems (NeurIPS) , page 2447-2455, 2011.

Matevvz Kunaver and Tomavz Povzrl. Diversity in recommender systems - a survey. In Knowledge Based Systems , volume 123, pages 154-162, 2017.

Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learning to rank in the cascade model. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , pages 767-776. PMLR, 2015.

E. Lex, Dominik Kowald, Paul Seitlinger, Thi Ngoc Trang Tran, Alexander Felfernig, and Markus Schedl. Psychology-informed recommender systems. In Foundations and Trends in Information Retrieval , volume 15, pages 134-242, 2021.

Zeyang Liu, Yiqun Liu, Ke Zhou, Min Zhang, and Shaoping Ma. Influence of vertical result in web search examination. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 193-202, 2015.

Changyong Oh, Jakub Tomczak, Efstratios Gavves, and Max Welling. Combinatorial bayesian optimization using the graph Cartesian product. Advances in the 32nd Neural Information Processing Systems (NeurIPS) , 2019.

My Phan, Yasin Abbasi Yadkori, and Justin Domke. Thompson sampling and approximate inference. Advances in the 32nd Neural Information Processing Systems (NeurIPS) , 32, 2019.

Lijing Qin, Shouyuan Chen, and Xiaoyan Zhu. Contextual combinatorial bandit and its application on diversified online recommendation. In In Proceedings of the 2014 SIAM International Conference on Data Mining , pages 461-469, 2014.

Carl Edward Rasmussen. Evaluation of Gaussian processes and other methods for non-linear regression . PhD thesis, University of Toronto Toronto, Canada, 1997.

Carl Edward Rasmussen. Gaussian Processes in Machine Learning. In Advanced Lectures on Machine Learning , pages 63-71. Springer, 2004.

Idan Rejwan and Yishay Mansour. Topk combinatorial bandits with full-bandit feedback. In Algorithmic Learning Theory , pages 752-776. PMLR, 2020.

Yunus Saatçi. Scalable inference for structured Gaussian process models . PhD thesis, University of Cambridge, 2012.

Dong-Her Shih, Kuan-Chu Lu, and Po-Yuan Shih. Exploring shopper's browsing behavior and attention level with an eeg biosensor cap. In Brain Sciences , volume 9, page 301, 2019.

Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on Machine Learning (ICML) , pages 1015-1022, 2010.

Hastagiri P Vanchinathan, Isidor Nikolic, Fabio De Bona, and Andreas Krause. Exploreexploit in top-n recommender systems via Gaussian processes. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys) , pages 225-232, 2014.

Richard S. Varga. Geršgorin and his circles. 2004.

Siwei Wang and Wei Chen. Thompson sampling for combinatorial semi-bandits. In Proceedings of the 35th International Conference on Machine Learning (ICML) , pages 5114-5122, 2018.

Yingfei Wang, Hua Ouyang, Chu Wang, Jianhui Chen, Tsvetan Asamov, and Yi Chang. Efficient ordered combinatorial semi-bandits for whole-page recommendation. In Proceedings of the 20th Conference on Artificial Intelligence (AAAI) , volume 31, page 2746-2753, 2017.

Yue Wang, Dawei Yin, Luo Jie, Pengyuan Wang, Makoto Yamada, Yi Chang, and Qiaozhu Mei. Beyond ranking: Optimizing whole-page presentation. Proceedings of the Ninth ACM International Conference on Web Search and Data Mining , pages 103-112, 2016.

Christopher K. I. Williams and Carl Edward Rasmussen. Gaussian Processes for Regression. Advances in the 9th Neural Information Processing Systems (NeurIPS) , pages 514-520, 1996.

Dong Xin, Jiawei Han, and Kevin Chen-Chuan Chang. Progressive and selective merge: computing top-k with ad-hoc ranking functions. In ACM SIGMOD Conference , pages 103-114, 2007.

A Kendall Kernels for Full and Top-k Rankings - Omitted Details

This section includes the proofs that were omitted from Section 2, presented in the following order:

In Section A.1, we present proofs for Claims 2 and 3, which concern the feature representations of Kendall kernels.

In Section A.2, we provide Algorithms 2 and a proof of its correctness for computing the WCKkernel in O ( k 2 ) time, thereby proving Claim 1. Additionally, we extend this proof to cover the proof of correctness for Algorithm 3, which can compute the CK kernel in O ( k log k ) , initially introduced by Jiao et al. [9]. The original paper presented the algorithm without a formal proof of correctness, a gap we address and fill in this section.

Section A.3 details the proof for Theorem 1, discussing the matrix-vector multiplications with the Kendall kernel matrix for top-k rankings. This proof builds on the Algorithm 2 given for computing the WCK kernel for top-k rankings.

A.1 Feature Representation for Kendall Kernels for Top-k Rankings

This section revisits the claims regarding the feature representations of the weighted Kendall kernel and the weighted convolutional Kendall kernel, subsequently providing the proofs for these claims mentioned earlier.

Claim 2. Let ϕ wk ( σ ) : Π n ↦→ R ( n 2 ) be a vector indexed by unique item pairs ( i, j ) , defined as:

where w s is the symmetric weighting function in product-symmetric weights. Then, ϕ wk is a linear feature vector for the weighted Kendall kernel with product-symmetric weights w ps .

Proof. Following the definition of linear feature representation, we need to prove that k wk ( σ 1 , σ 2 ) = ϕ ( σ 1 ) T ϕ ( σ 2 ) for the product-symmetric weight kernel as given in Equation 5. Recalling from Equation 2, we have k wk ( σ 1 , σ 2 ) as follows:

where the second line incorporates the use of the product-symmetric weight kernel. Next, our focus shifts to the simplification of η i,j ( σ 1 , σ 2 ) , which is elaborated as follows:

Combining the above factorization of η i,j with Equation 7, we get:

Claim 3. Let ϕ wck ( π ) : Π k ↦→ R ( n 2 ) be a vector indexed by unique item pairs ( i, j ) given as: ϕ wck i,j ( π ) := 1 √ ( n 2 ) · w wck i,j ( π ) · ( p π i<j -p π i>j ) , where w wck i,j ( π ) is determined as follows:

where w s denotes symmetric weights and w s ( ℓ, · ) = 1 n -k ∑ n j = k +1 w s ( ℓ, j ) . Then, the vector ϕ wck is a linear feature vector for the WCK kernel k wck . By uniformly setting w s ( · , · ) ≡ 1 in the definitions above, ϕ wck i,j ( π ) specializes to a linear feature vector for the CK kernel.

Proof. The main idea revolves around leveraging the feature representation of the Weighted Kendall kernel for a full ranking and the linearity of the convolution operation. It is already established that k wk ( σ 1 , σ 2 ) = ϕ wk ( σ 1 ) T ϕ wk ( σ 2 ) , as demonstrated in Claim 2. Recall that the WCK kernel requires a double summation over pairs of rankings from B π 1 and B π 2 , which represent the sets of full rankings consistent with their respective top-k rankings, as described in Equation 4. We simplify the WCK kernel as follows:

The simplification above reveals that the feature representation, ϕ wck , for the WCK kernel, is a ( n 2 ) dimensional vector and can be indexed by unique pairs of items ( i, j ) , much like the ϕ wk . However, the double summation is over an exponentially large number of pairs of rankings. Moving forward, we shift our focus to the individual entries of this representation involving this summation, elucidating the analytical values within the summation by exploring four unique cases, each dependent on whether these specific items fall within the top-k rankings.

In Case 1, we examine the scenario where items i and j are within the top-k ranking π . Here, the focus is on the feature representation of the pair, specifically when both elements are ranked among the top-k positions.

The simplification in lines 3rd and 4th follows from the fact that any full ranking σ ∈ B π , consistent with the top-k ranking π , the relative ranks and weights of items i and j remains unchanged, given p π i ∈ [ k ] and p π j ∈ [ k ] . Concretely, this implies p σ i<j = p π i<j for all σ ∈ B π and similar with the other term.

In Case 2, we analyze when item i is in the top-k ranking while item j is not.

Case 2: p π i ∈ [ k ] and p π j / ∈ [ k ] .

Next, every possible consistent ranking is considered jointly while fixating on a specific rank outside top-k elements, leading to ( n -k -1)! different rankings. Given that | B π | = ( n -k )! , we can refine the above expression as follows:

In Case 3, we analyze when item i is not in the top-k ranking while item j is.

Case 3: p π i / ∈ [ k ] and p π j ∈ [ k ] . Similar to case 2, the simplification follows analogously, with the only change being 1 p σ i<j -1 p σ i>j = -1 instead of 1 . Thus, by symmetry between i and j , we have the following:

Lastly, in Case 4, we analyze when items i and j are not in the top-k ranking.

Case 4: p σ i / ∈ [ k ] and p σ j / ∈ [ k ] .

The result of zero arises from symmetry. Since p σ i and p σ j are not in the topk ranking, they are treated symmetrically in the summation overall rankings in B π . For any ranking σ , suppose p σ i = l and p σ j = m , there exists a corresponding ranking σ ′ such that only the items i and j are swapped. Therefore, jointly, these two rankings yield w s ( l, m ) and -w s ( l, m ) . Since w s is symmetric, the overall contribution from each pair of such rankings is zero. Hence, the entire summation nets to zero.

Thus, with the explanation provided for each case and combining results from Equations 8, 9, 10 and 11, it's trivial to validate the Claim 3, i.e., ϕ wck i,j ( π ) = 1 √ ( n 2 ) · w wck i,j ( π ) · ( p π i<j -p π i>j ) for all unique pair of items. From Case 4, we have O (( n -k ) 2 ) entries leaving at max only O ( k 2 +2 nk ) non-zero entries.

A.2 Algorithms for Computing Kendall Kernels for top-k Rankings

In this section, we provide and delve into the proofs of Algorithms 2 and 3 for the weighted convolutional Kendall kernel and the convolutional Kendall kernel, as previously discussed in Section 2. Section A.2.1 for valid both the correctness and computational complexity of Algorithm 2 as given earlier in Claim 1. Following this, Section A.2.2 revisits Algorithm 3, initially introduced by Jiao et al. [10]. The original publication presented the algorithm without formal proof of its correctness, which we rectify and offer in Section A.2.2.

A.2.1 Efficiently Computing the Weighted Convolutional Kendall Kernel

This section provides a proof to Claim 1 to establish the efficiency and accuracy of Algorithm 2 in computing the weighted convolutional Kendall kernel, as specified in Equation 4, with a focus on its computational complexity.

Claim 1. The weighted convolutional Kendall kernel (Equation 4) with product-symmetric rank weights (Equation 5) can be computed in O ( k 2 ) time.

Proof. The claim is proven through Algorithm 2, where we establish its correctness and demonstrate its computation requirement is O ( k 2 ) . The essence of our proof centers on analyzing the feature representation of the WCK kernel, ϕ wck , as outlined in Claim 3. The feature vectors of ϕ wck reside in a ( n 2 ) dimensional space, indexed by pairs of items. Our approach is to demonstrate that Algorithm 2 accurately computes the right-hand side (RHS) of the equation k wck ( π 1 , π 2 ) = ϕ wck ( π 1 ) T ϕ wck ( π 2 ) . This involves a summation over item pairs, expressed as k wck ( π 1 , π 2 ) = ∑ l<m ϕ wck l,m ( π 1 ) T ϕ wck l,m ( π 2 ) .

Our proof analyzes various scenarios: cases where pairs of items, namely l and m , fall within the top-k, scenarios with one item within the top-k and the other outside, and situations where neither item is within the top-k. Each of these cases contributes distinctively to the computation of the overall kernel, resulting in different terms in the algorithmic computation. This is encapsulated in Algorithm 2, where k wck ( π 1 , π 2 ) = ∑ 5 i =1 s i ( π 1 , π 2 ) , and each s i corresponds to the terms given earlier in Algorithm 2 from Section 2.

Before proceeding with the cases of this summation as given in Table 4, we recall the notations utilized by Algorithm 3 in Definition 1. Also, remember that we will be proving for product-symmetric weights as given in Equation 5, where, w s : [ n ] × [ n ] ↦→ R n and its one-dimensional marginals are w s ( ℓ, · ) = 1 n -k ∑ n j = k +1 w s ( ℓ, j ) Table 4 shows how these cases are organized and relate to different s i terms required for computing the WCK kernel. The key strategy involves breaking down the kernel's computation into cases based on the positioning of item pairs within the top-k rankings. In case 1, we consider all the scenarios when both indices are within the set of items in both top-k rankings, i.e., all items in the set I 1 ∪ I 2 .

Definition 1. Algorithm 2 and 3 and utilize following notations.

I 1 and I 2 are the sets of items in rankings π 1 and π 2 , respectively.

σ 1 ∈ Π | I 1 | and τ 1 ∈ Π | I 1 ∩ I 2 | are the full rankings of I 1 and I 1 ∩ I 2 , both consistent with the input top-k ranking π 1 . I.e., relative ranks of items is same yielding ∀ l, m ∈ I 1 ∩ I 2 , p π 1 i>j = p τ 1 i>j .

Analogously, σ 2 and τ 2 are constructed utilizing the set I 2 and ranking π 2 .

Algorithm 2 Computing Weighted Convolutional Kendall Kernel

Input: Two permutations π 1 , π 2 ∈ Π k . Ranking weighting function w s : [ n ] × [ n ] ↦→ R n and its one dimensional marginals are w s ( ℓ, · ) = 1 n -k ∑ n j = k +1 w s ( ℓ, j ) .

Output: Convolutional Weighted Kendall kernel k wck ( π 1 , π 2 ) .

-Let I 1 and I 2 be the sets of items in rankings π 1 and π 2 , respectively.

1:

if

|

I

∩

3: end if

4:

if

|

I

∩

I

I

2

1

then and

|

I

\

I

1

then

1

6: end if

9: end if

12: end if

13:

if

|

I

1

∩

15: end if

2

I

2

1

and

|

[

1

n

]

\

2

(

I

1

∪

I

2

)

1

then

Case 1-a: Two items in I 1 ∩ I 2 , meaning both l and m belong to I 1 ∩ I 2 . Using Claim 3 regarding the feature vector ϕ wck , we simplify s 1 as follows:

Case 1-b: When one item is in I 1 ∩ I 2 , the other must reside either in I 1 \ I 2 or I 2 \ I 1 , thus leading to two distinct sub-cases. This is specified in Table 4. Concretely, if the other item is in I 1 \ I 2 , it contributes to the s 2 terms, whereas if it's in I 2 \ I 1 , it contributes to the s 3 terms.

Corresponding to Case 1-b-i, when the other item is in I 1 ∩ I 2 , i.e., s 2 is the term corresponding to indices where l is in I 1 ∩ I 2 and m in I 1 \ I 2 , or the reverse, represented by partial sums u and v . For the partial sum u , with l in I 1 ∩ I 2 and m in I 1 \ I 2 , we find that p π 2 l is in [ k ] , while p π 2 m is not. The simplification of u proceeds using Claim 3 as follows:

Similarly, the partial sum v can be simplified as follows:

In the above, the first two lines use results from Claim 3 and use similarity of w s . In the following line, l and m are exchanged. Lastly, the negative sign is pushed into the indicator functions to make the summand function of this partial sum v similar to the partial sum u , and the similarity of the w s is utilized. The above partial sums simplify s 2 as follows:

Analogously, in Case 1-b-ii, we deduce the corresponding term s 3 for the pair of indices as described in Table 4 through symmetry. Specifically, the term s 3 can be outlined as follows:

Case 1-c: Both items are outside I 1 ∩ I 2 , specifically, l ∈ I 1 \ I 2 and m ∈ I 2 \ I 1 or the reverse. Like Case 1-b-i, we divide s 4 into partial summations u and v . Now, we calculate u under the condition that l ∈ I 1 \ I 2 and m ∈ I 2 \ I 1 .

Similarly, we can estimate partial sum v for the set l ∈ I 2 \ I 1 & m ∈ I 1 \ I 2 . Using calculations similar to Case-1-b-i for summing u and v , we have:

Case 2: One item exists in I 1 ∩ I 2 , the other in [ n ] \ ( I 1 ∩ I 2 ) . It branches into two sub-cases: Case 2-a with one item in I 1 ∪ I 2 , and Case 2-b, where one item outside I 1 ∩ I 2 but is in I 1 ∪ I 2 . Focusing on Case 2-a, represented by s 5 , we simplify as follows. This involves two index scenarios, either l ∈ I 1 ∩ I 2 and m / ∈ I 1 ∪ I 2 or vice versa, represented by partial sums u and v . We now simplify u below:

Using steps similar to the previous case, we get the following value for s 5 :

For Case 2-b, l or m are absent from I 1 or I 2 , leading to two sub-scenarios. Consequently, either ϕ wck l,m ( π 1 ) is zero or ϕ wck l,m ( π 2 ) is zero. Therefore, these terms don't contribute to the overall WCK kernel value.

After covering all configurations of l and m , we incorporate results from Equations 12, 13, 14, 15, and 16. This integration yields the expression k wck ( π 1 , π 2 ) = ∑ 5 i =1 s i ( π 1 , π 2 ) , where, each term s i matches precisely with its corresponding expression in Algorithm 2. The proof for the correctness of Algorithm 2 is complete, as each term s i corresponds to its respective expression in the algorithm. Regarding the time complexity of Algorithm 2, each term s i sums at most k 2 quantities, and each quantity summed can be computed in O (1) time. Therefore, the computation time required for Algorithm 2 is O ( k 2 ) .

A.2.2 Efficiently Computing the Convolutional Kendall Kernel

This section provides Algorithm 3 for computing the convolutional Kendall kernel, as specified in Equation 3. Later, its efficiency and accuracy are proved in Claim 4.

Algorithm 3 Computing Convolutional Kendall Kernel [10] Input: Two top-k rankings π 1 , π 2 ∈ Π k . Output: Convolutional Kendall kernel k ck ( π 1 , π 2 ) . -Let I 1 and I 2 be the sets of items in rankings π 1 and π 2 , respectively. -Let σ 1 ∈ Π | I 1 | and τ 1 ∈ Π | I 1 ∩ I 2 | be the full rankings of I 1 and I 1 ∩ I 2 , both consistent with the input top-k ranking π 1 . -Analogously, construct σ 2 and τ 2 utilizing the set I 2 and ranking π 2 . 1: if | I 1 ∩ I 2 | ≥ 2 then 2: s 1 ( π 1 , π 2 ) = 1 ( n 2 ) · ( | I 1 ∩ I 2 | 2 ) · k sk ( τ 1 , τ 2 ) 3: end if 4: if | I 1 ∩ I 2 | ≥ 1 and | I 1 \ I 2 | ≥ 1 then 5: s 2 ( π 1 , π 2 ) = 1 ( n 2 ) · ∑ l ∈ I 1 ∩ I 2 2 · ( σ 1 ( l ) -τ 1 ( l )) -k + | I 1 ∩ I 2 | 6: end if 7: if | I 1 ∩ I 2 | ≥ 1 and | I 2 \ I 1 | ≥ 1 then 8: s 3 ( π 1 , π 2 ) = 1 ( n 2 ) · ∑ l ∈ I 1 ∩ I 2 2 · ( σ 2 ( l ) -τ 2 ( l )) -k + | I 1 ∩ I 2 | 9: end if 10: s 4 ( π 1 , π 2 ) = -1 ( n 2 ) · | I 1 \ I 2 | · | I 1 \ I 2 | 11: s 5 ( π 1 , π 2 ) = 1 ( n 2 ) · | I 1 ∩ I 2 | · | [ n ] \ ( I 1 ∪ I 2 ) | 12: k ck ( π 1 , π 2 ) = s 1 ( π 1 , π 2 ) + s 2 ( π 1 , π 2 ) + s 3 ( π 1 , π 2 ) + s 4 ( π 1 , π 2 ) + s 5 ( π 1 , π 2 )

Claim 4. Algorithm 3 computes the convolutional Kendall kernel (as given in the Equation 3) with a computational complexity of O ( k 2 ) .

Proof. To establish the correctness of Algorithm 3, we will adopt the same proof approach as the one used for Claim 1 concerning Algorithm 2. Specifically, we will adhere to the earlier categorization in Table 4 and notations given in Definition 1. Since the CK kernel can be derived by uniformly setting the weight function w s ( i, j ) = 1 , we will insert them in s i terms as given in Algorithm 2. These cases will be revisited and simplified by applying the condition w s ( i, j ) = 1 . Note that this also implies its one-direction marginal weights to be 1 , i.e., w s ( i, · ) = 1

Simplifying the s 1 Term: For the WCK kernel, Case 1-a leads to the expression of s 1 as stated in Equation 12. In this case, when two items, specifically l and m , are both in the intersection I 1 ∩ I 2 , it implies that p π 1 l , p π 1 m , p π 2 l , and p π 2 m all rank within the top-k, denoted as [ k ] . We simplify the s 1 term for CK kernel as follows:

The simplification process begins by assigning unit rank weights in the first line, i.e., w i = 1 . Following this, by relabeling the items in I 1 ∩ I 2 and using τ 1 and τ 2 , which are the rankings of π 1 and π 2 limited to the set I 1 ∩ I 2 as defined in Definition 1, it is established that η l ′ ,m ′ ( τ 1 , τ 2 ) = η l,m ( π 1 , π 2 ) . This is because the relative order of any pair of items is maintained in τ 1 and τ 2 . Consequently, this leads to the final simplification to a scaled value of the standard Kendall kernel k sk , as given in Equation 1.

Simplifying the s 2 and s 3 Terms: The s 2 and s 3 terms are obtained for Case 1-b, which is for case when one item is in I 1 ∩ I 2 and the other item is either in I 1 \ I 2 or I 2 \ I 1 . We divide this into two sub-cases. Case 1-b-i: The other item is in I 1 \ I 2 , with s 2 representing the summation terms derived from the CK's inner product. Case 1-b-ii: The other item is I 2 \ I 1 , where s 3 denotes the summation terms. We simplify the s 2 term for the CK kernel as follows:

Next, we examine the terms u and v in detail, starting with u . The term u , which corresponds to p π 1 l<m , signifies instances where item l is ranked before item m in the top-k ranking π 1 . This can be derived from the observation that σ 1 ( l ) -1 items are positioned before item l in the set I 1 . Out of these items, τ 1 ( l ) -1 also belong to the intersection I 1 ∩ I 2 . This follows from the definition of the full rankings σ 1 and τ 1 on the set I 1 and the intersection I 1 ∩ I 2 , respectively. Consequently, it can be concluded that σ 1 ( l ) -τ 1 ( l ) items from the set difference I 1 \ I 2 are ranked before item l . The second term, v , corresponds to p π 1 l>m and involves a calculation that takes into account the items ranked after the l -th item in the set I . Specifically, there are k -σ 1 ( l ) items following the l -th item. Within the intersection I 1 ∩ I 2 , the number of items before l is given by | I 1 ∩ I 2 | -τ 1 ( l ) . Therefore, the expression ( k -σ 1 ( l )) -( | I 1 ∩ I 2 | -τ 1 ( l )) represents the count of elements that are positioned after l in the set difference I 1 \ I 2 .

Combining the above calculations for both terms u and v , the s 2 term for the CK kernel can be simplified as follows:

Using the symmetry between Case 1-b-i and Case 1-b-ii, we can simplify s 3 for the CK kernel as follows:

Simplifying the s 4 and s 5 Terms: We simplify the s 4 and s 5 terms for the CK kernel starting from Equation 15 and Equation 16, respectively, as follows:

We have obtained the values of all the simplified s i terms for the CK kernel in Equations 17, 18, 19, 20, and 21. By combining these terms, we get k ck ( π 1 , π 2 ) = ∑ 5 i =1 s i ( π 1 , π 2 ) , where each term s i precisely matches its corresponding expression in Algorithm 3. This completes the proof of the correctness of Algorithm 3. Regarding its time complexity, each term s i sums at most k 2 quantities, and each quantity can be computed in O (1) time. Therefore, the time required for Algorithm 3 to compute the CK kernel is O ( k 2 ) .

A.3 Fast Matrix-Vector Multiplication with Kendall Kernel Matrix on Top-k Rankings

This section revisits Theorem 1 about the fact matrix-vector multiplication time for the Kendall kernel matrix for top-k rankings. Specifically, we aim to eliminate the mvm ( K X ) 's dependence on the number of items, i.e., n on and linear dependence in the number of rounds, i.e., T , as claimed in Theorem 1.

Theorem 1. For the WCK kernel with product-symmetric weights w ps , the computational complexity of multiplying the kernel matrix K X t with any admissible vector is O ( k 2 t ) , i.e., mvm ( K X t ) = O ( k 2 t ) , where X t is any arbitrary set of t top-k rankings.

Proof. The cornerstone of this proof lies in the computation of the WCK kernel, as delineated in Algorithm 2. This algorithm requires only O ( k 2 ) computation. For brevity, we write X to represent X T , and the proof follows for any X t , i.e., any value of t , not just T .

As also suggested previously, we will demonstrate through the equation K X = (Φ a X ) T Φ b X , where both matrices Φ a X and Φ b X have columns with only O ( k 2 ) non-zero entries. Consequently, this leads to the computational complexity of matrix-vector multiplication, denoted as mvm ( K X ) , being O ( k 2 · T ) .

From Algorithm 2, we know that each entry of the kernel matrix k ( π 1 , π 2 ) , can be expressed as a sum ∑ 5 i =1 s i ( π 1 , π 2 ) . Assuming each s i ( π 1 , π 2 ) equals ϕ a i ( π 1 ) T ϕ b i ( π 2 ) , and considering that all vectors ϕ a i and ϕ b i exhibit this property, we can express K X as (Φ a X ) T Φ b X . Here, the i th row of (Φ a X ) T and the j th column of Φ b X are represented by [ ϕ a 1 ( π i ) T , · · · , ϕ a 5 ( π i ) T ] and [ ϕ b 1 ( π j ) , · · · , ϕ b 5 ( π j )] , respectively. Therefore, the overall mvm complexity can be characterized by the sparsity of the vectors ϕ a i and ϕ b i , as is formalized in the claim presented below.

Claim 5. Consider a kernel matrix K X corresponding to any set X of cardinality T . Each entry of K X , denoted as k ( x 1 , x 2 ) , is defined by the sum ∑ 5 i =1 s i ( x 1 , x 2 ) , where each s i ( x 1 , x 2 ) is the result of the dot product ϕ a i ( x 1 ) T ϕ b i ( x 2 ) , where, ϕ a i and ϕ b i are vectors characterized by having O ( z ) non-zero entries. Given this structure, the matrix-vector multiplication complexity for K X is O ( nnz · T ) , i.e., mvm ( K X ) = O ( z · T ) .

Proof. We will demonstrate this in the following discussion by concentrating on the k th entry of the output vector, specifically K X v , for any arbitrary vector v :

Given that for all i , ϕ b i possesses only O ( z ) non-zero entries for any π j , the computation of ∑ j ϕ b i ( π j ) v j requires O ( z ) operations. This implies that the expression ∑ j ϕ b i ( π j ) v j also necessitates O ( z ) computation. Applying a similar rationale to ϕ a i , it follows that computing ( K X v ) k demands only O ( z ) operations. Extending this argument to all entries of the output vector, it is evident that computing K X v requires only O ( z · T ) computation

Utilizing Claim 5, it suffices to complete the proof by showcasing that these exist vectors ϕ a i and ϕ b i , each with only O ( k 2 ) non-zero elements, corresponding to each s i as specified in Algorithm 2. Additionally, these vectors ensure that s i ( π 1 , π 2 ) = ϕ a i ( π 1 ) T ϕ b i ( π 2 ) . We will next establish such vectors for all s i terms. Starting with the s 1 term below.

Showcasing s 1 ( π 1 , π 2 ) = ϕ a 1 ( π 1 ) T ϕ b 1 ( π 2 ) for sparse ϕ a 1 ( π 1 ) and ϕ b 1 ( π 2 ) vectors. We begin by manipulating s 1 , as defined in Equation 12. For the sake of brevity, their scalar factors will be omitted in the following explanation.

Both ϕ a 1 and ϕ b 1 are sparse by design, taking non-zero values only when l and m appear in the top-k rankings. This demonstrates the existence of sparse vectors for the s 1 term. Next, we will establish the same for the s 2 and s 3 terms.

Showcasing sparse vectors for s 2 and s 3 . We begin by manipulating s 2 , as defined in Equation 13, while ignoring its scalar factor. We will exploit symmetry between s 2 and s 3 terms.

Equation 25 demonstrates the existence of vectors ϕ a 2 and ϕ b 2 for the s 2 term. The vectors ϕ a 21 and ϕ a 22 , possessing O ( k ) and O ( k 2 ) non-zero entries respectively, are defined in Equations 23 and 24. Consequently, the ϕ a 2 vector has O ( k 2 ) non-zero entries. Similarly, it can be shown that ϕ b 2 contains O ( k 2 ) non-zero entries, thus fulfilling the proof requirements for proving the s 2 term. For the s 3 term, we observe a symmetry between s 2 and s 3 , namely s 3 ( π 1 , π 2 ) = s 2 ( π 2 , π 1 ) . This symmetry makes it trivial to satisfy the requirements, as further highlighted by the following equation:

Showcasing sparse vectors s 4 ( π 1 , π 2 ) = ϕ 4 a ( π 1 ) T ϕ 4 b ( π 2 ) . We begin by manipulating the s 4 term without scalar, as defined in Equation 15.

Observing that w := ∑ m ∈ I 2 w s ( p π 2 m , · ) represents a constant value that does not depend on I 2 , we can further simplify the above expression for s 4 as follows:

Next, to simplify the above equation, we first focus on the second term and have the following:

Next, we simplify the third and last term in the Equation 27 as follows:

Next, combining the results from Equations 27, 28, and 29, we obtain the following:

Equation 30 showcases both ϕ 4 a and ϕ 4 b has three components with having only O ( k 2 ) non-zero entries, thus fulfilling the requirements for the s 4 term. Next, we focus on the s 5 term.

Showcasing sparse vectors s 5 ( π 1 , π 2 ) = ϕ 5 a ( π 1 ) T ϕ 5 b ( π 2 ) . We begin by examining the s 5 term, excluding its scalar component, as outlined in Equation 16.

The equation shows that s 5 ( π 1 , π 2 ) = ϕ 5 a ( π 1 ) T ϕ 5 b ( π 2 ) , where both ϕ 5 a and ϕ 5 b possess components with a maximum number of non-zero entries, as indicated in Equations 31 and 32. This completes the proof requirements for the s 5 term.

By combining the results from Equations 22, 25, 26, 30, and 33, we have demonstrated the existence of vectors ϕ a i and ϕ b i , each containing only O ( k 2 ) non-zero elements, and have established that s i ( π 1 , π 2 ) = ϕ a i ( π 1 ) T ϕ b i ( π 2 ) for each i ∈ 1 , 2 , 3 , 4 , 5 . In conjunction with Claim 5, this completes the proof.

B Proposed GP-TopK Bandit Algorithm- Omitted Details

This section includes the proofs that were omitted from Section 4, presented in the following order:

Section B.1 outlines a brief of Gaussian process regression for any domain.

Section B.2 summarizes the committed details about the local search utilized for optimizing the UCB function.

Section B.3 provides the removed proof for the Theorem 2 concerning the overall time for the bandit algorithm.

Section B.4 provides the proof for Theorem 3 concerning regret analysis of the proposed bandit algorithm.

B.1 Gaussian Process Regression

In GP regression [22], the training data are modeled as noisy measurements of a random function f drawn from a GP prior, denoted f ∼ N (0 , k ( · , · )) , where k : X × X → R is a kernel function over any domain X . The observed training pairs ( x i , y i ) are collected as X = [ x 1 , . . . , x T ] and y = [ y 1 , . . . , y T ] ∈ R T , where, for an input x i , the observed value is modeled as y i = f ( x i ) + ϵ , with ϵ i ∼ N (0 , σ 2 ) . The kernel matrix on data is K X = [ k ( x i , x j )] T i,j =1 ∈ R T × T . The posterior mean µ f |D and variance σ f |D functions for GPs are:

where k x ∈ R T has as its i th entry k ( x , x i ) , z = ( K X + σ 2 I ) -1 y , and I is an identity matrix. For GP regression on an arbitrary domain X , the kernel function must be a p.d. kernel [23].

Naive approaches rely on the Cholesky decomposition of the matrix K X + σ 2 I , which takes Θ( T 3 ) time [23]. To circumvent the Θ( T 3 ) runtime, recent works use iterative algorithms such as the conjugate gradient algorithm, which facilitate GP inference by exploiting fast kernel matrix-vector multiplication (MVM) algorithms, i.e., v ↦→ K X v [3]. In practice, these methods yield highly accurate approximations for GP posterior functions with a complexity of Θ( p · T 2 ) for p iterations of the conjugate gradient algorithm, as mvm ( K X ) = T 2 , and mvm ( M ) is the operation count for multiplying matrix M by a vector. p ≪ T proves to be efficient in practical application [3].

B.2 Contextual GP Reward Model

Optimizing the AF , i.e., UCB function, poses a significant challenge due to its enormous size of Π k . Drawing inspiration from prior research on Bayesian optimization within combinatorial spaces, we employ a breadth-first local search (BFLS) to optimize the UCB acquisition function [2, 19]. The BFLS begins with the selection of several random top-k rankings. Subsequently, each specific top-k ranking is compared with the UCB values of its neighboring rankings, proceeding to the one with the highest UCB value.

The neighbors of a top-k ranking include all its permutations and the permutations of modified top-k rankings obtained by swapping one item with any of the remaining items. For any top-k ranking, there are ( n -k ) · k ! + k ! neighbors, which is often not huge as k is often ≤ 6 . This search continues until no neighboring top-k ranking with a higher value is discovered. Although BFLS is a local search, the initial random selection and multiple restart points help it evade local minima, a strategy that previous studies have corroborated [19].

B.3 Assessing GP-TopK Compute Requirements

Theorem 2. Assuming a fixed number of iterations required by the iterative algorithms, the total computational time for running the GP-TopK bandit algorithm for T rounds of top-k recommendations, using the contextual product kernel (Equation 6), is O ( k 2 cℓT 2 ) . This applies to WK, CK, and WCK top-k ranking kernels, where ℓ is the number of local search evaluations for selecting the next arm in every round.

Proof. The proof can be straightforwardly derived by combining the results presented in Table 1, which succinctly summarizes the time complexities for each step of computing the UCB using both feature and kernel approaches. It is important to emphasize that iterative algorithms enhance results from O ( T 4 ) to O ( T 3 ) in computational complexity. Furthermore, these algorithms can further reduce complexity to O ( T 2 ) when used with the feature approach.

The results presented in Table 1 can be validated through straightforward observations and by leveraging findings from previous Sections 2. Specifically, Section 2 offers proof for the mvm ( K X ) row explicitly. For the compute K X t row, the complexity of kernel approaches is deduced from Algorithms 2 and 3. For feature approaches, the compute K X t row is inferred from the sparsity of the feature representations as stated in Claim 3. Lastly, the memory row is straightforwardly deduced for the kernel approach by counting its entries. For the feature approach, it is derived from the sparsity of the feature representations.

B.4 Regret Analysis

In this section, we revisit Theorem 3 and provide its proof. The proofs build on the work by Krause et al. [14], delivering results for bounding the contextual regret in the context of the top-k ranking problem. To set the stage for our regret analysis, let's first define the critical term maximum mutual information , denoted by γ t , is given below:

where I ( y X ; f ) quantifies the reduction in uncertainty (measured in terms of differential Shannon entropy) about f achieved by revealing y A [27]. In Gaussian observation case, the entropy can be computed in closed form: H ( N ( µ, Σ)) = 1 2 log | 2 πe Σ | , so that I ( y X ; f ) = 1 2 log | I + ξ -2 K X | , where K X = [ k ( x, x ′ )] x,x ′ ∈ X is the Gram matrix of k evaluated on set X ⊆ X . For the contextual bandit algorithm, X represents contexts and arms considered until round t .

Before proving Theorem 3, we align the Krause et al. [14] results with our notation for consistency. Furthermore, we modify β t to accommodate embeddings encompassing negative values, aligning with the fact that contextual embeddings may exhibit negative dimensions.

Proposition 1 (Theorem 1, [14]) . Let δ ∈ (0 , 1) , and the unknown reward function ˆ f be sampled from the known GP prior with known noise variance σ 2 . Suppose one of the following holds:

Assumption 1 holds and set β t = 2log( |X| t 2 π 2 / 6 δ ) .

Assumption 2 holds and set β t = 2 B 2 +300 γ t ln 3 ( t/δ ) .

Then the cumulative regret R T of the contextual GP bandit algorithm with the UCB acquisition function is bounded by ˜ O ( √ C 1 Tγ T β T ) w.h.p. Precisely,

Pr { R T ≤ √ C 1 Tγ T β t ) + 2 ∀ T ≥ 1 } ≥ 1 -δ , where, C 1 = 8 / log(1 + σ -2 ) and the notation ˜ O hides logarithmic factors in n , 1 δ and T .

Proposition 1 shows that the regret R T for the contextual GP bandit algorithm, utilizing the UCB acquisition function is bounded with high probability within ˜ O ( √ C 1 Tγ T β T ) , where the notation ˜ O hides logarithmic factors in n , 1 δ and T . To ascertain the ˜ O order for R T , it is imperative to first bound the ˜ O order of γ T β t . We begin by examining the γ T term in the subsequent proposition.

Proposition 2. Under the assumptions of Theorem 3, γ T can be succinctly characterized as γ T = O ( n 2 c log( n 2 T ) + c log T ) , which also simplifies to ˜ O ( n 2 c ) , where the ˜ O notation omits logarithmic factors in n and T .

Proof. For the GP bandit algorithm with the UCB acquisition function, γ T = C · log ( | I + σ -2 K X T | ) , where C equals (1 / 2) · (1 -1 /e ) -1 and K X T represents the kernel matrix computed over contexts and arms across T rounds [27, 14]. Precisely, K X T is calculated using the contextual kernel defined in Equation 6. It is applied to contexts and top-k ratings from the feedback data D t , corresponding to Line 6 of the generic contextual bandit Algorithm 1.

Next, we leverage the characteristic of the contextual kernel being a product kernel. Consequently, the maximum mutual information term for the joint kernel, γ T , can be upper bounded by c · ( γ π T +log T ) , where c denotes the dimensionality of contexts and γ π T represents the maximum information gain in a non-contextual setting [14]. Specifically, γ π T is computed similarly but is confined to top-k rankings. That is, γ π T = C · log ( | I + σ -2 K X π | ) , with K X π T being calculated exclusively using the top-k kernels on the top-k rankings as selected by the bandit algorithm. X π T represents the top-k rankings selected by the bandit algorithm, i.e., excluding the contexts from the collected feedback.

Recalling the formulation for top-k rankings kernels, we have K X T = Φ T X π T Φ X π T , where Φ X π ∈ R ( n 2 ) × T comprises feature columns pertinent to the topk ranking kernels, as elucidated in Section A. Utilizing the Weinstein-Aronszajn identity, γ π T is expressed as C · log ( | I + σ -2 Φ X π T Φ T X π T | ) . Further, we deduce that γ π T ≤ C · ∑ ( n 2 ) i =1 log ( | 1 + σ -2 λ i | ) , where λ i is an eigenvalue of Φ X π T Φ T X π T . Given the Gershgorin circle theorem, which bounds all eigenvalues of a matrix by the maximum absolute sum of its rows, therefore we can conclude that γ π T = O ( n 2 log( n 2 T )) , as for all the columns of the Φ X π have bounded normed as given in Claims 2 and 3, i.e., || ϕ ( π ) || 2 2 ≤ 1 [29].

By combining γ π T = O ( n 2 log( n 2 T )) with the contextual product kernel, we obtain γ T = O ( n 2 c log( n 2 T ) + c log T ) , thereby providing the claimed bound in the proposition.

Next, we build on Propositions 1 and 2 to prove the main theorem regarding the regret of the proposed GP-TopK bandit algorithm for top-k recommendations.

Theorem 3. If either Assumptions 1 or 2 hold, setting β t as 2 log ( |C|·| Π k |· t 2 · π 2 6 δ ) and 300 γ t ln 3 ( t δ ) respectively, the cumulative regret R T of the GP-TopK bandit algorithm for top-k recommendations can, with at least 1 -δ probability, be bounded by ˜ O ( n √ C 1 Tc (log |C| + k +log( T 2 π 2 / 6 δ ))) under Assumption 1, and ˜ O ( n √ C 1 (2 B 2 c +300 n 2 c 2 ln 3 ( T/δ )) T ) under Assumption 2. Here, C 1 = 8 log(1+ ξ -2 ) , and ˜ O excludes logarithmic factors related to n , k , and T .

Proof. We will prove the above theorem for both cases separately.

For Assumption-1. Given |C| is finite and β T = 2log( |D| T 2 π 2 / 6 δ ) . First, we focus on bounding β T as follows:

As ( n k ) ≤ n k and k ! ≤ k k , we also have log | Π k | = log (( n k ) k ! ) ≤ log ( n k k k ) = O ( k log( nk )) , which implies that β T = O (log |C| + k log( nk ) + log( T 2 π 2 / 6 δ )) . Combining this with Proposition 2, we have following:

Thus, we showcase the asserted bound for the regret R T as ˜ O ( √ C 1 Tγ T β T ) = ˜ O ( n √ C 1 Tc (log |C| + k +log( T 2 π 2 / 6 δ )) ) .

For Assumption-2. Given ∥ f ∥ k ≤ B and β t = 2 B 2 +300 γ t ln 3 ( t/δ ) . First, we bound the β T term using Proposition 2 as follows:

Using the above result, we have the following:

Comparison with Srinivas et al. (2010). Using the identity kernel for top-k rankings, we can develop a finite-dimensional feature for the contextual kernel and apply Theorem 5 by Srinivas et al. (2010). Given that γ T = O ( n k c log T ) , the regret bounds are as follows under both assumptions. For instance, the calculations for the O ( √ C 1 Tγ T β T ) under the Assumption 2 are as follows:

Similarly, we can analogously perform the analysis for Assumption 1 and combine it with Proposition 1 to obtain the regret bounds mentioned in the Table 3.

C Experiments - Omitted Details

This section presents omitted details from the main body of the text.

C.1 Compute resources

We utilized multiple NVIDIA Tesla M40 GPUs with 40 GB RAM on our in-house cluster for our experiments. The experiments in Section 5 required approximately 5 GPU-hours for small arm space and 24 GPU-hours per iteration for large arm space. We conducted about 50 to 100 iterations throughout the project. The results reported in Section C.3 required the same computational resources as the large arm space experiments.

WK

CK

WCK

0

50

100

(a)

WK

CK

WCK

0.0

0.5

1.0

(b)

C.2 Bandit Simulation and Hyper-parameter Configurations - Omitted Details

To set up the simulation, we utilized embeddings trained on the MovieLens dataset using a collaborative filtering approach [6]. We consider a 1 M variant of the MovieLens dataset, which contains 1 million ratings from 6040 users for 3677 items. Specifically, we train user embeddings c u and item embeddings θ i such that the user's attraction to the items are captured by the inner product of the user embedding with the item embeddings, respectively. Both context and item embeddings, i.e., c u and θ i , are 5 -dimensional, optimized by considering the 5 -fold performance on this dataset. The reward provided in our experiments is contaminated with zero mean and standard deviation equals 0 . 05 .

For the ϵ -greedy baselines, we considered various values of ϵ are considered, specifically ϵ = { 0 . 01 , 0 . 05 , 0 . 1 } . The outcomes are presented for the configuration that demonstrates optimal performance. For MAB-UCB baseline, the algorithm has an upper confidence score ucb ( i ) = µ i + β mab √ 2 ln( t +1) n i [11]. Here, µ i represents the average reward, n denotes the total number of rounds, and n i signifies the frequency of arm i being played. β mab is a hyper-parameter. We evaluate β mab values within the set { 0 . 1 , 0 . 25 , 0 . 5 } and disclose results for the best-performing configuration. For the parameters of proposed GP-TopK bandit algorithms, we set β t = β gp · log( |X| · t 2 · π 2 ) with β gp ∈ { 0 . 05 , 0 . 1 , 0 . 5 } , reporting results the value that yields the best performance. The choice of β t is informed by prior work in GP bandits [27]. The selection of σ for all variants is determined by optimizing the log-likelihood of the observed after every 10 rounds by considering values in the set { 0 . 01 , 0 . 05 , 0 . 1 } .

C.3 Additional results

Local search results for optimizing combinatorial objectives in Π k for n = 50 and k = 6 . Specifically, π ⋆ = max π ϕ r ( π ) T ϕ r ( π ′ ) , where ϕ r ( π ′ represents the feature vector for Kendall kernels on top-k rankings. Notably, for this optimization problem, it is known that the optimal value is 1 obtained by only π ′ . Figure 4 shows results for this optimization problem when applied to WK, CK, and WCK kernels.

Reward results for large arm space for the nDCG + diversity reward. Similar to Figure 3, a large setup with n = 50 for k = 3 and k = 6 , is considered. For k = 6 , the possible arms are over 1 . 1 × 10 10 possible top-k rankings. Given the vastness of this arm space, computing the optimal arm for the diversity reward is not straightforward. Therefore, we focus on reporting the cumulative reward in Figure 5. We implement this setup using a Local search in batch mode, updating every 5 round and considering a substantial horizon of T = 100 rounds. Specifically, we use 5 restarts, 5 steps in every search direction, and start with 1000 initial candidates. Figure 5 shows that the WCK approach demonstrates superior performance, continuing to learn effectively even after extensive rounds.

0

50

100

T

0

800

1600

RT

Random

Opt

-greedy

MAB

WK

CK

WCK

0

50

100

T

0

800

1600

RT

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Section 1 briefs both contributions and scope of this work.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 6 reflects on the limitations of this work.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs of all Claims and Theorems are provided in the Appendix.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 5 provides necessary details of bandit simulator and experimental setups considered in this work.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code can be accessed using this hyper-link.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 5 provides experimental details and a few remaining details are given in the Appendix C.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All figures reported in this work have errorbars with them.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section C provides relevant information about compute resources.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Section 6 reflects on the impact of this work.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: this work does not release any such resource or asset.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: this work release only code with instructions for its usage.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of top-k recommendation with bandit feedback, where the goal is to recommend a list of k items and receive feedback only on the overall quality of the list, not individual items. This captures the notion that user satisfaction depends on the overall quality of the recommendations, not just the single most relevant item. To tackle this, the authors propose a novel algorithm combining Gaussian Process Upper Confidence Bound (GP-UCB) with a specifically designed weighted Kendall kernel. The paper proposes faster inference exploiting features of the Kendall kernel, improving computational complexity from $O(T^4)$ to $O(T^2)$. An upper bound on cumulative regret is derived, showing sublinear regret. Finally, synthetic experiments show that the proposed algorithm can outperform a few baselines.\n\nDisclosure: I have reviewed a previous version of this paper.\n\nSTRENGTHS:\nStrengths:\n* Extends the literature on top-k recommendations with a GP-UCB algorithm\n* Well presented\n\nWEAKNESSES:\nWeaknesses:\n* The approach does not seem scalable to real-world problems\n* The theoretical contribution could be further clarified\n\nQUESTIONS:\n* The theoretical analysis builds on the analysis of Krause & Ong (2011), adapted to the Kendall kernel. Can the authors please clarify the technical contribution (novelty) compared to Krause & Ong (2011).\n* The approach does not seem practical. The empirical evaluation is done with a small number of items (n<=50) and short horizon (T<500). In real-world applications the number of items is often much larger and the number of rounds can also be large.\n* The acquisition function is not applied to all possible top-k rankings, rather a local search approach is used. The regret analysis does not account for the approximation due to the local search. The implementation also uses an approximate posterior rather than an exact posterior. This creates a discrepancy between the implementation of the algorithm and the theoretical analysis. The authors are somewhat upfront about this.\n* L304 (Evaluation for large arm space):\nFirst, I wouldn’t say that n=50 is large. In real-world applications n is in the millions (e.g., music, books) or billions (e.g., videos).\nSecond, when showing the regret in Figrue 3 you should actually show all iterations (1-500), not just the ones that include a model update (every 5 steps).\n\n* A few comments from the previous review that are still not fixed:\n  * You still have an overloaded notation $\\sigma$ for both a ranking (L88) and the sigmoid function (L273). In the previous version you had a three-way overload of $\\sigma$ and now it’s only two, but this is easy to fix.\n  * Another example from the rebuttal: “CG is the conjugate gradient algorithm. We will clarify this in the future.”\nIn the current version this part is moved to the appendix (L707), but still appears as “CG”, without clarification.\n  * L283: “We set $\\lambda = 0.25$ to emphasize relevance over diversity.” This value actually gives a 0.25 weight to ndcg and a 0.75 weight to diversity so the weight for diversity is higher. I also mentioned this in my review to the previous version, did I not understand this correctly and this is why you ignored the comment?\n\nLIMITATIONS:\nNo societal issues.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a contextual bandit algorithm for top-k recommendations, leveraging Gaussian processes with a Kendall kernel to model the reward function. The proposed method utilizes full-bandit feedback without assumptions such as semi-bandit feedback or cascade browsing. Theoretical analysis demonstrates a sublinear regret with the number of rounds and arms, and empirical results from simulations show superior performance compared to other baselines. The authors also improve the computational efficiency and memory requirements of the algorithm.\n\nSTRENGTHS:\n1. The paper is well-organized and well-written, with clear explanations and detailed experimental results.\n2. The proposed algorithm achieves state-of-the-art performance in top-k recommendation tasks compared with various baselines. The theoretical analysis provides a solid foundation for the algorithm's performance guarantees.\n3. The improvements in computational efficiency and memory requirements are novel, and technical, making the algorithm more practical for real-world applications.\n\nWEAKNESSES:\n1. The paper claims that existing bandit algorithms impose strict assumptions about feedback models, e.g., semi-bandit feedback or cascade browsing. I agree with this statement. However, the paper addresses this problem by directly using Gaussian processes to model the reward function. Isn't this also a restrictive assumption? The authors should clarify this point.\n2. The paper lacks a detailed discussion on the choice of the Kendall kernel. The authors should provide more insights into why this kernel was chosen and its advantages over other kernels.\n3. There are existing works about top-k combinatorial bandits with full-bandit feedback, such as Rejwan and Mansour (2020) [1]. However, the paper does not compare its method with these existing works. The authors should compare their method with these existing works to demonstrate the difference and novelty of their approach.\n\n[1] Rejwan, Idan, and Yishay Mansour. \"Top-$ k $ combinatorial bandits with full-bandit feedback.\" Algorithmic Learning Theory. PMLR, 2020.\n\nQUESTIONS:\nQuestions:\n1. Why did you choose the Kendall kernel for the Gaussian process? What are the advantages of this kernel over other kernels?\n2. How does your algorithm's performance compare to existing works that use full-bandit feedback, such as Rejwan and Mansour (2020)?\n\nLIMITATIONS:\nLimitations are adequately discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new bandit algorithm for top-k recommendations that utilizes a Gaussian Process to model the reward of the exponentially large set of arms, and provide regret bounds which improve upon the naive approach of modeling each arm independently. The authors also present a new kernel and show that the associated kernel matrix displays a particular structure that can be exploited to efficiently compute GP predictions. Finally, experimental validation show the improved regret with respect to baseline algorithms and other similar kernels.\n\nSTRENGTHS:\n- The paper is very well presented and easy to read, and the proofs are also very clear.\nThe authors provide meaningful insight into the strengths of their new kernel through small examples and the results of the experiments.\n- The computational costs of a Gaussian Process using the new kernel are extensively addressed, and their approach is significantly more efficient than a standard procedure.\n\nWEAKNESSES:\n- The authors claim the new algorithm to be their primary contribution, though it is a simple adaptation of a contextual GP-UCB to top-k recommendations, and a lot of attention is dedicated instead to the new kernel and its computational costs.\n- Despite the strengths of the new kernel, it is not particularly novel as it is a straightforward combination of existing kernels.\n- Figures in the Experiments section should be more clear about the confidence ranges (the confidence level in Figure 2 is not mentioned, and the ranges appear to be missing entirely in Figures 3 and 5).\n\nI found a few typos and confusing passages, but overall, they didn't affect the reading experience. I will report them here as a feedback:\nline 99: typo ww(...) -> w(...)\nlines 254-256: this sentence sounds like the new terms involving n in the bounds are n^{k/2-1} and n^{k-1}, maybe the intended meaning was to highlight the improvement with respect to the cited results?\nlines 281-283: \\lambda = 0.25 emphasizes diversity over relevance\nline 751: typo shoes -> shows\nline 821: typo very -> every (also possibly missing a subject after \"observed\"?)\n\nQUESTIONS:\n1) The number of local searches is a substantial factor in the overall time complexity of the proposed algorithm, and the number of local searches seems to be particularly high for the large arm space scenarios. Since optimizing the computational costs of the algorithm is a big part of this work, I would like to ask the authors whether they have considered and whether it is possible and/or practically useful to further exploit the structure of the kernel to avoid recomputing the predictor entirely when moving between neighbors.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the slate recommendation problem where k items are simultaneously recommended to a user at the same time (in a banner or \"slate\").  In order to solve this problem the authors adapt existing Gaussian process methods to the top-k setting by modifying Kendall kernels to the top-k setting.  They also introduce some computational tricks that allow for the algorithm to be quadratic in the embedding size.  They also provide a regret analysis and empirical studies to defend their algorithm.\n\nSTRENGTHS:\nThe paper is extremely readable and well presented, I particularly like the boxes highlighting key claims and the examples e.g. Table 2 and Figure 1.  The mathematical notation is clean and consistently used, and the authors are clearly in command of the literature they are building on.  This is particularly evident when the authors are able to draw a clean distinction between existing work and their contributions.\n\nThe experiments are well designed, cleanly implemented, well executed and presented immaculately.\n\nWEAKNESSES:\nI would like to provide some commentary on this paper and how it might relate to production recommender systems without in any way wanting to diminish the excellent work done in this paper.  NeurIPS is a primarily academic research venue and this paper is very well executed and it is very pleasing to see a recommender systems paper of such quality.\n\nI would however be very surprised if this method or a method derived from it was applied in a real system for a number of reasons.\n\nFirst, I like the authors use of Figure 1 in order to highlight limitations of the cascade model (the even simpler position based model has similar problems), there have however been models proposed for this case including the Probabilistic Rank and Reward model - https://arxiv.org/pdf/2208.06263 .  The \"full bandit framework\" while useful does discard the useful preference information about what was clicked (as well as saying the banner was clicked somewhere), this information is usually quite valuable.\n\nThe covariance functions of the model (equation 6) is very well explained and the idea is interesting.  A further useful note is that using this multiplied form the reward is correlated if both the context is similar _and_ the actions (top-k) are similar, due to multiplication if either is approximately zero the covariance is zero.  The covariance in the action space is rather limited however as it has no notion for some actions being similar.  For example if you are going to recommend action movies then you might fill the top-k with totally different action movies yet still have a correlated reward.  The model in this paper has some similarities, including the factorized covariance matrix https://arxiv.org/abs/2008.12504\n\nFinally, this paper learns a unique reward for every top-k ordering.  This results in a combinatorial explosion both in learning and in delivering recommendations.  While the assumption is interesting it does not allow for finding the best recommendations performing an argsort often approximated using a fast maximum inner product search (e.g. LSH or HNSW), it seems difficult to relax these assumptions in real world systems.\n\nQUESTIONS:\nDo you agree with the limitations discussed above?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-20"}