Title: Thompson Sampling for Multi-Objective Linear Contextual Bandit
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Objective Multi-armed Bandit

Linear Contextual Bandit

Pareto Optimality

Thompson Sampling
Abstract: We study the multi-objective linear contextual bandit problem, where multiple possible conflicting objectives must be optimized simultaneously. We propose $\texttt{MOL-TS}$, the first Thompson Sampling algorithm with Pareto regret guarantees for this problem. Unlike standard approaches that compute an empirical Pareto front each round, $\texttt{MOL-TS}$ samples parameters across objectives and efficiently selects an arm from a novel effective Pareto front, which accounts for repeated selections over time. Our analysis shows that $\texttt{MOL-TS}$ achieves a worst-case Pareto regret bound of $\widetilde{O}(d^{3/2}\sqrt{T})$, where $d$ is the dimension of the feature vectors, $T$ is the total number of rounds, matching the best known order for randomized linear bandit algorithms for single objective. Empirical results confirm the benefits of our proposed approach, demonstrating improved regret minimization and strong multi-objective performance.

Thompson Sampling for Multi-Objective Linear Contextual Bandit

Abstract

We study the multi-objective linear contextual bandit problem, where multiple possible conflicting objectives must be optimized simultaneously. We propose MOL-TS , the first Thompson Sampling algorithm with Pareto regret guarantees for this problem. Unlike standard approaches that compute an empirical Pareto front each round, MOL-TS samples parameters across objectives and efficiently selects an arm from a novel effective Pareto front , which accounts for repeated selections over time. Our analysis shows that MOL-TS achieves a worst-case Pareto regret bound of ˜ O ( d 3 / 2 √ T ) , where d is the dimension of the feature vectors, T is the total number of rounds, matching the best known order for randomized linear bandit algorithms for single objective. Empirical results confirm the benefits of our proposed approach, demonstrating improved regret minimization and strong multi-objective performance.

1 Introduction

The multi-objective multi-armed bandit (MOMAB) problem [8, 5, 15, 17, 11, 18, 10, 7] generalizes the classical, single-objective multi-armed bandit to settings with multiple, potentially conflicting objectives. Pulling an arm yields a vector of objective-specific rewards, so a single 'best' arm is often ill-defined and optimality must account for trade-offs across objectives.

One simple way to handle trade-offs is scalarization [14, 21, 20], which maps reward vectors to a scalar via, e.g., weighted sums, minimax, or other nonlinear transforms, thereby reducing MOMAB to a single-objective bandit. However, selecting a suitable scalarization is itself nontrivial, and an arm optimal under one scalarization can be markedly suboptimal under another. An alternative is to reason directly in the vector space via Pareto optimality [8, 5, 11, 18, 10, 7]: the Pareto front comprises arms whose mean reward vectors are not dominated component-wise. Because it compares reward vectors objective-wise, Pareto optimality is strictly more general than committing to a fixed scalarization, and we adopt it throughout.

All of the prior MOMAB work in Pareto regret propose and study UCB-based algorithms [8, 11, 18, 10]. To the best of our knowledge, the Pareto regret of Thompson Sampling (TS) [4, 2] has not been studied for MOMAB. This gap is noteworthy: in many single-objective contextual and non-contextual bandits, TS and its variants are empirically competitive or superior to UCB methods [16, 6, 4, 2], yet their worst-case analyses are typically more delicate. Extending TS to MOMAB introduces additional challenges, including coordinating randomized samples across multiple objectives and handling the possibility of multiple Pareto-optimal arms.

In this work, we develop a TS algorithm for the multi-objective linear contextual bandit and analyze its worst-case Pareto regret. Our method samples separate parameters for each objective and evaluates arms via an optimistic sampling mechanism that ensures a nontrivial probability of being jointly optimistic across all objectives (Section 5.3), which underpins the regret analysis.

We also revisit the notion of performance measurement. The standard Pareto regret compares perround mean reward vectors; consequently, repeatedly playing a single Pareto-optimal arm can yield zero regret even when an alternative arm selection would strictly larger cumulative rewards (see Section 3.3). To address this, we introduce the notion of an effective Pareto-optimal arm (Definition 5): an arm that is Pareto-optimal per round and remains undominated when evaluated through the lens of cumulative rewards under any number of repeated selections. Building on this, we define a corresponding regret notion that penalizes policies vulnerable to such cumulative inefficiencies; our algorithm targets the effective Pareto front to avoid these failures.

Our main contributions are summarized as follows:

We propose an algorithm for the multi-objective linear contextual bandit problem: Thompson sampling for Multi-objective Linear Bandit ( MOL-TS ). To the best of our knowledge, this is the first randomized algorithm for multi-objective contextual bandits with Pareto regret guarantees. Unlike the existing multi-objective algorithms, MOL-TS does not explicitly compute an empirical Pareto front each round, but rather randomly selects an arm from that Pareto front, which is much more computationally efficiently.

We propose the concept of a effective Pareto optimal arm (Definition 5), which satisfies the condition of Pareto optimal arm, and also the total rewards for every objective with any number of its repeated selection satisfying Pareto optimality. Any arm that is not effective Pareto optimal has an alternative selection of arms over the same total number of rounds, resulting higher total rewards in all objectives. Our proposed algorithm, MOL-TS , operates on this new notion of the effective Pareto front and samples an arm from the estimated effective Pareto front. As a result, MOL-TS produces higher cumulative rewards compared to the methods that selected from the plain Pareto front.

We establish that MOL-TS is statistically efficient, achieving the Pareto regret bound of ˜ O ( d 3 / 2 √ T ) , where d is the dimension of the feature vectors, T is the total number of rounds. In order to ensure the provable guarantees of the randomized exploration for multiple objectives, MOL-TS adopts the optimistic sampling strategy (Section 5.3).

Numerical experiments demonstrate the effectiveness of our proposed approach, showing improved performance in regret minimization, and objective-wise total reward maximization.

2 Related works

Multi-objective multi armed bandit setting was first explored by Drugan and Nowe [8], who proposed UCB algorithms for MOMAB by applying two representative approaches: using Pareto order and scalarized order. Subsequently, Auer et al. [5] proposed algorithms that identify all Pareto optimal arms with high probability. More recently, the upper and lower bounds of Pareto regret in the MOMABsetting have been studied in both stochastic and adversarial settings by Xu and Klabjan [18]. There are also several studies on multi-objective contextual bandits. For example, Tekin and Tur˘ gay [15] studied MOMAB in a contextual setting where a dominant objective exists, but we do not assume any dominance among objectives. Turgay et al. [17] developed the PCZ algorithm, which identifies the Pareto front using the idea of contextual zooming and proved its regret bound. However, the algorithm is complex, and the paper does not provide specific details on its implementation. Lu et al. [11] studied the multi-objective generalized linear bandit (MOGLB) problem and analyzed the upper bound of Pareto regret using the ParetoUCB algorithm. Additionally, Kim et al. [10] explored Pareto front identification in linear bandit settings. The studies mentioned thus far proposed complex algorithms that calculate the empirical Pareto front. In contrast, Zhang [20] introduced a hypervolume scalarization method in stochastic linear bandit settings, which uses random scalarization to explore the entire Pareto front.

While these studies address significant challenges in multi-objective bandits, surprisingly, although the practical effectiveness of randomized methods is widely recognized, research on randomized algorithms in multi-objective bandits has been rare. To the best of our knowledge, only Yahyaa and Manderick [19] proposed a Thompson Sampling (TS) algorithm for MOMAB, but no theoretical analysis of this approach has been conducted. Separately, there has been significant research on randomized scalarization in the multi-objective Bayesian optimization problem [14, 21], including theoretical analyses of TS algorithms. However, Zhang and Golovin [21] and Paria et al. [14] analyzed the "Bayes regret" with known Gaussian prior setting, increasing with the number of objectives.

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

/lscript

= 1

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

/lscript

= 2

µ

(1)

t,a

µ

(2)

t,a

a

(1)

µ

t,a

(1)

= [0

.

2

,

0

.

9]

a

(2)

µ

t,a

(2)

= [0

.

4

,

0

.

4]

a

(3)

µ

t,a

(3)

= [0

.

8

,

0

.

2]

a

(4)

µ

t,a

(4)

= [0

.

1

,

0

.

2]

t

= 1

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

/lscript

= 1

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

/lscript

= 2

µ

(1)

t,a

µ

(2)

t,a

a

(1)

µ

t,a

(1)

= [0

.

1

,

0

.

7]

a

(2)

µ

t,a

(2)

= [0

.

3

,

0

.

4]

a

(3)

µ

t,a

(3)

= [0

.

7

,

0

.

2]

a

(4)

µ

t,a

(4)

= [0

.

6

,

0

.

6]

t

= 2

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

/lscript

= 1

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

/lscript

= 2

µ

(1)

t,a

µ

(2)

t,a

a

(1)

µ

t,a

(1)

= [0

.

1

,

0

.

5]

a

(2)

µ

t,a

(2)

= [0

.

8

,

0

.

2]

a

(3)

µ

t,a

(3)

= [0

.

3

,

0

.

3]

a

(4)

µ

t,a

(4)

= [0

.

2

,

0

.

8]

t

= 3

In the single-objective case, the theoretical analysis of Thompson sampling was first introduced by Agrawal and Goyal [3], and then was extended to the stochastic linear bandit setting in Agrawal and Goyal [4], where arms were categorized as either saturated or unsaturated to derive theoretical bounds. From a different perspective, Abeille and Lazaric [2] analyzed Thompson Sampling under the assumption of fixed probabilities for sampling optimistic parameters. Additionally, Chapelle and Li [6] showed that, although the theoretical guarantees of Thompson Sampling are weaker than those of UCB, empirical results have consistently demonstrated that TS algorithms outperform UCB algorithms in practice. However, there has been a clear gap in extending these theoretical guarantees from single objective settings to multi-objective bandits.

Weprovide the first theoretical analysis of a randomized algorithm in the multi-objective bandit setting. To the best of our knowledge, this is the first work to propose a TS algorithm for multi-objective linear contextual bandits and to analyze it theoretically.

3 Preliminaries

3.1 Notations

Throughout this paper, we use notations that distinguish between different objectives. For any positive integer N ∈ N , let [ N ] := { 1 , 2 , . . . , N } . We denote L as the number of objectives, and for ℓ ∈ [ L ] , any real number u corresponding to the ℓ th -objective is denoted as u ( ℓ ) . The vector u ∈ R L comprises all u ( ℓ ) values and is represented in bold notation, i.e., u = [ u (1) , u (2) , . . . , u ( L ) ] ⊤ . Otherwise, individual features of any vector x are typically denoted as x ( i ) . For clarity, ∥ · ∥ denotes the Euclidean norm, and for a positive semi-definite matrix V , the norm ∥ · ∥ V is defined in the inner product space with the matrix V as ⟨ x, y ⟩ V = √ x ⊤ V y . Finally, we define S n ⊂ R n as the unit ( n -1) -simplex.

3.2 Problem settings

We consider a standard stochastic linear contextual bandit problem, extended to the multi-objective setting. Let A be a finite set of arms. Each arm a ∈ A corresponds to a d -dimensional context vector x t,a ∈ R d which is adversarially given at each round t . For each objective ℓ ∈ [ L ] , there exists a fixed parameter θ ( ℓ ) ∗ ∈ R d , but unknown to agent. In total, there are L parameters θ (1) ∗ , θ (2) ∗ , . . . , θ ( L ) ∗ . At each round t ∈ [ T ] , the agent selects an arm a t ∈ A and receives a L -dimensional reward vector r t,a t = [ r (1) t,a t , r (2) t,a t , . . . , r ( L ) t,a t ] ⊤ ∈ R L , where the reward for each objective ℓ is given by r ( ℓ ) t,a t = x ⊤ t,a t θ ( ℓ ) ∗ + ξ ( ℓ ) t , and ξ ( ℓ ) t is a zero-mean random noise. The mean reward for objective ℓ is defined as µ ( ℓ ) t,a t := E [ r ( ℓ ) t,a t ] . And consequently, the mean reward vector of the chosen arm a t is µ t,a t = [ µ (1) t,a t , µ (2) t,a t , . . . , µ ( L ) t,a t ] ⊤ ∈ R L .

3.3 Pareto optimality

Definition 1 (Pareto order) Let u , v ∈ R L be two distinct vectors. We say that the vector u is dominated by the vector v (i.e., v dominates u ), denoted as u ≺ v , if u ( ℓ ) ≤ v ( ℓ ) for all ℓ ∈ [ L ] and u ( ℓ ) < v ( ℓ ) for some ℓ ∈ [ L ] . Conversely, the vector u is not dominated by the vector v , denoted as u ̸≺ v , if there exists at least one ℓ ∈ [ L ] satisfying u ( ℓ ) > v ( ℓ )

Definition 2 (Pareto optimal arm) An arm is Pareto optimal if its mean reward vector is not dominated by that of any other arm. The set of all Pareto optimal arms is called Pareto Front ( P ∗ t ) ,

In linear setting, where the mean rewards of each arm remain fixed, the Pareto front is denoted as P ∗ .

Definition 3 (Pareto sub-optimality gap) The Pareto sub-optimality gap of an arm a at round t is the minimum scalar value ϵ ≥ 0 for a to be Pareto optimal, i.e.,

The Pareto sub-optimal gap can be defined as ∆ PR t,a := max a ′ ∈P ∗ min ℓ ∈ [ L ] { µ ( ℓ ) t,a ′ -µ ( ℓ ) t,a } . For every Pareto optimal arm a ′ ∈ P ∗ t , the arm that maximizes ∆ PR t,a ′ is a ′ itself, which implies ∆ PR t,a ′ = 0 . Any other arm is dominated by at least one Pareto optimal arm, ensuring that ∆ PR t,a ≥ 0 .

Definition 4 (Pareto regret) Let a 1 , a 2 , . . . , a T be the sequence of arms chosen by agent. The Pareto regret up to round T is defined as PR ( T ) := ∑ T t =1 ∆ PR t,a t .

In the contextual bandit setting, the Pareto front varies dynamically depending on the given context. Hence, we can not apply the algorithm of identifying Pareto front in Auer et al. [5] and Kim et al. [10] as they remove arm from the arm set, which can be Pareto optimal in our setting.

By using the Pareto order relationship, the definition of Pareto regret provides a general measurement of an agent's performance in a multi-objective setting. Previous studies of Pareto optimality [8, 11, 17, 10] adopt this measurement and design algorithms that randomly select arms from the Pareto front, aiming to minimize the Pareto regret. However, this definition of Pareto regret does not fully account for cumulative rewards. For example, consider a case with two objectives and four arms, as illustrated in Figure 1. Suppose two agents follow same policy that randomly selects arm from Pareto front. The first agent sequentially selects a (2) , a (4) and a (3) , while the second agent selects a (1) , a (4) and a (2) . Both agents selected Pareto optimal arms, resulting zero Pareto regret. But total rewards of the first agent is µ 1 ,a (2) + µ 2 ,a (4) + µ 3 ,a (3) = [1 . 3 , 1 . 3] and the second agent is [1 . 6 , 1 . 7] . This example highlights the limitation of Pareto regret, as it does not distinguish between policies that yield different cumulative rewards despite selecting only Pareto optimal arms. Thereby, we propose the concept of a effective Pareto optimal arm, whose mean reward vector is Pareto optimal and contributes to maximizing cumulative rewards across all objectives.

3.4 Effective Pareto optimality

Definition 5 (Effective Pareto optimal arm) An arm is effective Pareto optimal (denoted a ∗ ) if its mean reward vector is either equal to or not dominated by any convex combination of the mean reward vectors of the other arms. Formally, for any β ∈ S |A|1 ,

where β = ( β a ) a ∈A\{ a ∗ } . The set of all effective Pareto optimal arms at round t is called the effective Pareto front, denoted as C ∗ t . In the linear bandit setting, where the mean reward vectors of all arms remain fixed, the effective Pareto front is denoted as C ∗

In this paper, we refer to an arm that is not effective Pareto optimal as sub-optimal. If an arm a ′ ∈ A is sub-optimal, then there exists a convex combination β , such that µ t,a ′ ≺ ∑ a ∈A\{ a ′ } β a µ t,a .

Every effective Pareto optimal arm is also a Pareto optimal arm. This can be easily verified by restricting β to be a one-hot vector, which corresponds to comparing mean reward vectors between two individual arms. However, the converse does not hold. As can be seen from Figure 1, at the first round, arm a (2) is Pareto optimal, but not effective Pareto optimal, because its mean reward vector is dominated by a convex combination of two arms a (1) and a (3) . Hence, for any t ∈ [ T ] , we have C ∗ t ⊂ P ∗ t . This shows that our definition of effective Pareto optimal arm is strictly defined than the standard definition of Pareto optimal arm.

As shown in the example in the previous subsection, two agents following the same policy randomly selected arms from the Pareto front. The second agent consistently selected arms from the effective Pareto front, while the first agent selected arms from the Pareto front but not the effective Pareto front in the first and third rounds. This difference led to the first agent achieving lower cumulative rewards than the second agent for all objectives. Importantly, this disparity becomes increasingly severe as the total number of rounds T grows, leading to significantly worse long-term performance for policies that fail to prioritize the effective Pareto front.

The intuition behind an effective Pareto optimal is that repeatedly selecting such arms leads to Pareto optimal cumulative rewards. In other words, rather than selecting a sub-optimal arm over multiple rounds, it is preferable to select effective Pareto optimal arms for the same total number of rounds, which is expected to yield strictly higher cumulative reward in some objectives without worsening the others. In summary, for large enough total number of rounds T , selecting arms from effective Pareto front C ∗ t achieves higher total rewards than selecting arms from Pareto optimal front P ∗ t . Based on this, we propose a theorem that establishes a relationship between the newly defined effective Pareto optimality and the linear scalarization method.

Theorem 1 For any a ∗ ∈ C ∗ , there exist w ∈ S L satisfying a ∗ = arg max a ∈A w ⊤ µ a . Conversely, for any w ∈ S L , if a ∗ = arg max a ∈A w ⊤ µ a is unique arm, then a ∗ ∈ C ∗ .

The theorem is proved in Appendix A where we refer to the proof from Mangasarian [12]. The theorem shows a one-to-one correspondence: every effective Pareto optimal arm is optimal for some non-negative weight vector, and every non-negative weight vector guarantees to have an effective Pareto optimal arm.

Definition 6 (Effective Pareto sub-optimality gap) Let β = ( β a ) a ∈A be a vector in S |A| and define µ t,β = ∑ a ∈A β a µ t,a . The effective Pareto sub-optimality gap for selecting arm a t at round t is defined as

The effective Pareto sub-optimal gap measures the minimum value ϵ for a t not to be dominated by any convex combination of the mean reward vectors of the other arms. In other words, the gap quantifies how close arm a t is to being effective Pareto optimal. For any effective Pareto optimal arm, this gap is zero. Also, it is easy to verify that the effective Pareto sub-optimality gap is always greater than or equal to the standard Pareto sub-optimality gap, i.e., ∆ PR t,a t ≤ ∆ EPR t,a t , since the Pareto sub-optimality gap corresponds to the special case where β is restricted to be a one-hot vector. As discussed in Section 3.3, the effective Pareto sub-optimality gap can also be expressed as

Definition 7 (Effective Pareto regret) The effective Pareto regret up to round T is defined as EPR ( T ) := ∑ T t =1 ∆ EPR t,a t .

4 Algorithm: MOL-TS

We propose a multi-objective linear Thompson Sampling algorithm, MOL-TS , a generic randomized algorithm designed with multiple regularized MLE, where one need not sample from an actual

Bayesian posterior [2]. Our algorithm adopts an optimistic sampling strategy to avoid the theoretical challenges in worst-case regret analysis [13, 9].

Each round t , the mean reward vector for each arm is estimated based on the history of chosen arms a 1 , a 2 , . . . , a t -1 , and received reward vectors r 1 ,a 1 , r 2 ,a 2 , . . . , r t -1 ,a t -1 up to round t . The true parameter for each objective θ ( ℓ ) ∗ is estimated by regularized least squares (RLS), denoted ˆ θ ( ℓ ) t . Given regularizer λ ∈ R + , the matrix and the RLS estimator is defined as

For each objective ℓ , the parameters ( ˜ θ ( ℓ ) t,m ) m ∈ [ M ] are sampled independently M times from Gaussian posterior distribution N ( ˆ θ ( ℓ ) t , c 2 V -1 t ) , where the tunable parameters c and M are given from the beginning. A total of ML samples are drawn. The reward for each arm and for each objective is then optimistically evaluated using the sample that yields the highest value,

The reward vector for each arm is then constructed as ˜ µ t,a = [ ˜ µ (1) t,a ˜ µ (2) t,a . . . ˜ µ ( L ) t,a ] ⊤ .

The number of samples M controls the probability that the estimated rewards are optimistically evaluated. Increasing M raises the likelihood that the reward estimates are optimistic, which is crucial for ensuring a high theoretical probability of optimism across multiple objectives (see Section 5.3). We approximate the empirical effective Pareto front ˜ C t using the estimated reward vectors, by

Note that this optimistic sampling strategy is different from that proposed in [13, 9]. The setting in [13] considers a dynamic assortment selection problem, and [9] considers a combinatorial selection problem. Unlike multiple arms selection problem in single objective setting, our setting considers receiving multiple rewards from single arm selection problem.

5 Regret analysis

In this section, we analyze the expected effective Pareto regret of our algorithm MOL-TS in the worst-case, where the expectation is taken over all sources of randomness present in the problem setup. We begin with the general assumptions widely used in the linear bandit literature. We then outline the challenges in bounding the effective Pareto regret and explain how the number of samples M affects the worst-case regret bound.

5.1 Assumptions

Let F t = σ ( x 1 ,a 1 , . . . , x t,a t , r 1 ,a 1 , . . . , r t -1 ,a t -1 ) be the filtration up to round t containing all historical information about the selected arms and the received rewards. The following assumptions are commonly used in the stochastic linear bandit literature.

Assumption 1 (Boundedness) For each arm a ∈ A , ∥ x t,a ∥ ≤ 1 . Also, ∥ θ ( ℓ ) ∗ ∥ ≤ 1 for all ℓ ∈ [ L ] .

Assumption 2 (Sub-Gaussian) Each noise ξ ( ℓ ) t is conditionally R -sub-Gaussian, given the filtration F t and for some R ∈ R + .

Under the first assumption, each vector is both fixed and bounded for all rounds. If ∥ x t,a ∥ ≤ C and ∥ θ ( ℓ ) ∗ ∥ ≤ C are bounded for some constant C , then our regret bound would increase by a factor of C . Note that we do not assume any linear independence of the true parameters or noise vectors between objectives. Our assumptions are essentially the same as those used in the single objective stochastic linear bandit setting.

5.2 Challenges in bounding the effective Pareto regret

Previously, there are many papers of multi-objective UCB-type algorithm with Pareto regret analysis [8, 15, 17, 11, 10, 18]. The analysis of UCB algorithm is almost similar to that of single objective setting, attaining Pareto regret bound where the number of objectives depend up to logarithmic factor. By contrast, deriving comparable guarantees for TS in the multi-objective linear contextual setting remains an open problem, and our work is the first to tackle these technical obstacles directly.

Recall the effective Pareto regret Equation (1). For any weight vector w ∈ S L , since ∥ w ∥ 1 = 1 ,

The algorithm MOL-TS optimistically evaluates reward vector ˜ µ t,a for each arm, and selects arm a t randomly from the set ˜ C t . Hence, by Theorem 1, there exist weight vector, denoted w t , satisfying a t = arg max a ∈A w ⊤ t ˜ µ t,a . But for true mean reward vector, let ¯ a ∗ = arg max a ∈A w ⊤ t µ t,a be the effective Pareto optimal arm for given weight vector w t . Then we have

The key insight is that the effective Pareto regret is bounded by the weighted sum of rewards under an arbitrary weight vector, and the same holds for Pareto regret. This analysis generalizes the single objective case, which is recovered by restricting w t to a one-hot vector.

However, since the arm a t is randomly selected from the set ˜ C t , both the weight vector w t , and the corresponding effective Pareto optimal arm ¯ a ∗ are random. Due to the randomness of the vector w t and the optimal arm ¯ a ∗ , analyzing the worst-case regret bound of TS algorithm becomes significantly more difficult. Also, unlike the single objective setting, the multi-objective setting involves multiple true parameters and corresponding RLS estimates. This complicates the problem of ensuring optimism, as there are multiple sampled parameters in TS algorithm (see example in Section 5.3). We resolve these theoretical challenges by adopting an optimistic sampling strategy .

5.3 Why do we need optimistic sampling?

In this section, we explain the necessity of the number of samples M . As we discuss in Section 5.2, the challenges in the worst-case regret analysis for TS algorithms lie in the difficulty of ensuring optimism in randomly selected arm a t . When w t is one-hot vector, the analysis aligns with the single objective setting [4, 2]. However, since w t is random, the analysis requires that the randomly chosen arm is optimistically evaluated under a weighted sum of rewards. This probability can become exponentially small as the number of objectives increases.

Before providing a detailed explanation, we first define the event ˆ E t that the true parameters θ ( ℓ ) ∗ are close enough to the RLS estimate parameters ˆ θ ( ℓ ) t , and define c 1 ,t ( δ ) which is the high probability bound on the distance between the true parameter and the RLS estimate,

By Lemma 8, we have P ( ˆ E t ) ≥ 1 -δ . The O ( √ log L ) dependence in c 1 ,t ( δ ) is inevitable as the confidence bound must hold uniformly over all objectives. We define the event ˙ E ( ℓ ) t,a for a specific arm a and objective ℓ , where the event has at least one parameter following anti-concentration property of being optimistic, i.e.,

As the algorithm MOL-TS optimistically evaluate each arm a with Equation (2), the probability P ( ˙ E ( ℓ ) t,a ) increases as M increases. Suppose, for example, one follows standard TS algorithm by setting M = 1 , that only one parameter is sampled for each objective. Previous studies [4, 2] have shown that the probability of an arm a being optimistically evaluated is at least ˜ p , i.e.,

where ˜ p is constant probability, that depends on the choice of sampling distribution. However, since w t is random, the probability of ensuring this optimism for every objective is at least ˜ p L . Since this probability decreases exponentially with the number of objectives, the regret grows exponentially in L , yielding ˜ O ( 1 / ˜ p L · d 3 / 2 √ T ) .

Optimistic sampling strategy resolves this problem as MOL-TS optimistically evaluate the rewards using M independent parameter samples for each objective. Specifically, the algorithm evaluates the arm according to the sampled parameter that maximizes the evaluation, i.e.,

Then, the probability bound for the optimism event is

To prevent the exponential growth of the probability of ensuring optimism, the number of samples M must depend on the number of objectives L . The next lemma shows the minimum number of samples M for ensuring the event of optimism with constant probability.

Lemma 1 (Optimistic Sampling) For any arm a ∈ A , define the event of anti-concentration property of being optimism ˙ E t,a = ⋂ ℓ ∈ [ L ] ˙ E ( ℓ ) t,a . Then on event ˆ E t , with p = 0 . 15 and M ≥ 1 -log L log(1 -p ) , we have P ( ˙ E t,a ) ≥ p .

The event ˙ E t,a is that the arm a is optimistically evaluated for every objective. Lemma 1 shows that the lower bound on the probability that arm a being optimistically evaluated remains constant by taking optimistic sampling strategy. The proof of this lemma is provided in Appendix B.1.

5.4 Worst-case regret bound

We now present the worst-case (frequentist) regret upper bound of MOL-TS , where the expectation is taken over all sources of randomness present in the problem setup.

Theorem 2 (Effective Pareto regret of MOL-TS ) With Assumptions 1 and 2, with c = c 1 ,t ( δ ) and M = ⌈ 1 -log L log(1 -p ) ⌉ , the effective Pareto regret of the algorithm MOL-TS is upper-bounded by

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Pareto Regret

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 1

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 2

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 3

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 4

0

2500

5000

7500

10000

Round (

t

)

0

5

10

15

20

Pareto Regret

MOL-TS

(

M

= 1)

MOL-TS

(

M

=

O

(log

L

))

0

2500

5000

7500

10000

Round (

t

)

0

5

10

15

20

Effective Pareto Regret

where

Corollary 1 (Pareto regret of MOL-TS ) With same assumptions and initialization, the Pareto regret of the algorithm MOL-TS is upper-bounded by

Discussions of Theorem 2 and Corollary 1. Theorem 2 establishes that the expected effective Pareto regret of MOL-TS is bounded above by ˜ O ( d 3 / 2 √ T ) , where the regret has an additional O (log L ) and O ( √ log M ) dependence on the number of objectives and the number samples, respectively, both of which are minimal. Additionally, Corollary 1 holds since ∆ PR t,a t ≤ ∆ EPR t,a t . The details of the proof are provided in Appendix B. To the best of our knowledge, MOL-TS is the first TS algorithm with the worst-case regret guarantees in both Pareto regret and effective Pareto regret.

6 Experiments

In this section, we empirically evaluate the performance of our algorithm. We measure the Pareto regret and effective Pareto regret over T = 10000 rounds. Each experimental setup contains 10 different instances with fixed number of arms K , objectives L , and feature dimension d . We demonstrate the case where K = 50 , d = 5 , L = 4 . The parameter vector for each objective θ ( ℓ ) ∗ has a norm of 1 . Each round, d -dimensional context vectors are revealed for every arm, bounded by 1 in Euclidean norm. Upon playing an arm, the agent receives a reward vector with an additional noise term, where the noise values are sampled from a zero mean Gaussian distribution with σ = 1 .

We compare the performance of MOL-TS with those basic novel algorithms : the Upper Confidence Bound algorithm, and ϵ -Greedy algorithm. The Upper Confidence Bound algorithm is MOGLM-UCB (represented as MOL-UCB in our experiments), from Lu et al. [11] in linear bandit setting and ϵ -Greedy algorithm MOLϵ -Greedy is basic MOMAB algorithm with ϵ = 0 . 05 . Other algorithms cannot be applied in contextual setting, as they remove sub-optimal arm from the arm set. We also

compare the performance of MOL-TS with and without the optimistic sampling. As shown in Figure 2, our proposed algorithm MOL-TS shows greater performance compared to other algorithms, with minimizing the Pareto regret and effective Pareto regret, and maximizing cumulative rewards in all objectives. Additionally, MOL-TS with optimistic sampling performs better in minimizing regret. Additional experiments with different settings of K,d and L are left in Appendix F, with additional algorithm PFIwR from Kim et al. [10] in linear bandit setting.

We observe that MOLϵ -Greedy yields higher Pareto regret and effective Pareto regret, but also total rewards compared to MOL-UCB . This counterintuitive behavior arises from the averaging of cumulative rewards: although the algorithm selects arms that are Pareto optimal, averaging their outcomes can reduce the overall performance because the algorithm randomly samples from the Pareto front. In other words, by exploring multiple Pareto optimal arms without a consistent preference direction, averaging total rewards may appear smaller despite balanced trade-offs. This issue could be mitigated by guiding the arm selection toward a specific scalarization or optimization direction, allowing the algorithm to maintain both Pareto efficiency and higher total reward.

7 Discussions

In this paper, we study the multi-objective linear contextual bandit problem, where multiple conflicting objectives must be optimized simultaneously. We define the effective Pareto regret, whose definition considers the Pareto optimality of cumulative reward vectors. We propose a Thompson Sampling algorithm with optimistic sampling strategy, MOL-TS , that achieves the Pareto regret and effective Pareto regret of ˜ O ( d 3 / 2 √ T ) , matching the best known order for randomized linear bandit algorithms for single objective setting. Empirical results confirm the benefits of our proposed approach, demonstrating improved regret minimization and strong multi-objective performance.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Marc Abeille and Alessandro Lazaric. Linear thompson sampling revisited. In Artificial Intelligence and Statistics , pages 176-184. PMLR, 2017.

Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory , pages 39-1. JMLR Workshop and Conference Proceedings, 2012.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013.

Peter Auer, Chao-Kai Chiang, Ronald Ortner, and Madalina Drugan. Pareto front identification from stochastic bandit feedback. In Artificial intelligence and statistics , pages 939-947. PMLR, 2016.

Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems , 24, 2011.

Ji Cheng, Bo Xue, Jiaxiang Yi, and Qingfu Zhang. Hierarchize pareto dominance in multiobjective stochastic linear bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 11489-11497, 2024.

Madalina M Drugan and Ann Nowe. Designing multi-objective multi-armed bandits algorithms: A study. In The 2013 international joint conference on neural networks (IJCNN) , pages 1-8. IEEE, 2013.

Taehyun Hwang, Kyuwook Chai, and Min-hwan Oh. Combinatorial neural bandits. In International Conference on Machine Learning , pages 14203-14236. PMLR, 2023.

Wonyoung Kim, Garud Iyengar, and Assaf Zeevi. Learning the pareto front using bootstrapped observation samples. arXiv preprint arXiv:2306.00096 , 2023.

Shiyin Lu, Guanghui Wang, Yao Hu, and Lijun Zhang. Multi-objective generalized linear bandits. arXiv preprint arXiv:1905.12879 , 2019.

Olvi L Mangasarian. Nonlinear programming . Society for Industrial and Applied Mathematics, 1994.

Min-hwan Oh and Garud Iyengar. Thompson sampling for multinomial logit contextual bandits. In Advances in Neural Information Processing Systems , volume 32, 2019.

Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. A flexible framework for multi-objective bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelligence , pages 766-776. PMLR, 2020.

Cem Tekin and Eralp Tur˘ gay. Multi-objective contextual multi-armed bandit with a dominant objective. IEEE Transactions on Signal Processing , 66(14):3799-3813, 2018.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Eralp Turgay, Doruk Oner, and Cem Tekin. Multi-objective contextual bandit problem with similarity information. In International Conference on Artificial Intelligence and Statistics , pages 1673-1681. PMLR, 2018.

Mengfan Xu and Diego Klabjan. Pareto regret analyses in multi-objective multi-armed bandit. In International Conference on Machine Learning , pages 38499-38517. PMLR, 2023.

Saba Q Yahyaa and Bernard Manderick. Thompson sampling for multi-objective multi-armed bandits problem. In ESANN , 2015.

Qiuyi Richard Zhang. Optimal scalarizations for sublinear hypervolume regret. Advances in Neural Information Processing Systems , 37:39963-39999, 2024.

Richard Zhang and Daniel Golovin. Random hypervolume scalarizations for provable multiobjective black box optimization. In International conference on machine learning , pages 11096-11105. PMLR, 2020.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose the first Thompson Sampling algorithm with Pareto regret guarantees in multi-objective linear contextual bandit. Our contributions are clearly summarized in Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in Appendix D.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theorems are presented with assumptions in detail (see Section 5). Details and explanations of the proofs are given in Appendix A and Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All codes of algorithms and experiments are provided in a ZIP file. Experimental results are provided in Section 6 and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All codes of algorithms and experiments are provided in a ZIP file. Experimental results are provided in Section 6 and Appendix F.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details of experimental settings are provided in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In both Section 6 and Appendix F, all the bars in regret plots and reward plots represent the 1 -sigma standard deviation of experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include information about the computing environment used to run experiments in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research in our paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper has no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We clearly mention the sources of the comparator algorithms in Section 6.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper clearly describes our proposed algorithm in Section 4. The codes of our proposed algorithm are provided in a ZIP file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: Our core method development in this research does not involve any usage of LLMs.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Proof of Theorem 1

In this section we present the proof for the necessary properties of effective Pareto optimal arms. Before proving this theorem, we use the following definition that denotes the convex hull of arbitrary set.

Definition 8 Let M⊂ R L be a set of mean reward vector µ a . For all a ∈ A . Define the convex hull of a set M as Conv ( M ) .

By definition, for any β = ( β a ) a ∈A ∈ S |A| , we have

This convex hull covers all the convex combination of mean reward vectors of every arms. We note that, in the convex set Conv ( M ) , the mean reward vector of the effective Pareto optimal arm satisfies the Pareto optimality in Conv ( M ) . In other words, for all a ∗ ∈ C ∗ , we have

However, for the arm a ∈ A \ C ∗ , there exists µ ∈ Conv ( M ) satisfying

The arm that is effective Pareto optimal, satisfies the Pareto optimality in Conv ( M ) (see Definition 5).

Other than the Pareto optimality, the next definition describes the optimality of one specific objective, with constraint on the other objectives.

Definition 9 ( ϵ -constraint optimal) Let ϵ ℓ be L -1 dimensional arbitrary constraint vector

Given the constraint vector ϵ ℓ , the ϵ ℓ -constraint optimal vector among the set Conv ( M ) , denoted µ ℓ ∗ , is defined by

̸

The vector is ϵ -constraint optimal (denoted µ ∗ ) if, for all ℓ ∈ [ L ] , there exists constraint vector ϵ ℓ such that the vector µ ∗ is ϵ ℓ -constraint optimal vector.

The constraint vector ϵ ℓ is the lower bound values, such that the vector µ dominates the constraint vector, except objective ℓ . Among those vector µ satisfying constraint, the ϵ ℓ -constraint optimal vector is the one that has maximum value in objective ℓ . The ϵ -constraint optimal vector is such constraint vector ϵ ℓ exists, as to be ϵ ℓ constraint optimal, for all ℓ ∈ [ L ] .

The next lemma shows the equivalence between ϵ -constraint optimality and Pareto optimality.

Lemma 2 The vector µ ∗ ∈ Conv ( M ) is Pareto optimal if and only if it is ϵ -constraint optimal.

̸

Proof. ( = ⇒ ) Let µ ∗ be Pareto optimal. Assume it is not ϵ ℓ -constraint optimal for some ℓ . Let the constraint vector be ϵ ( k ) = µ ( k ) ∗ for k = 1 , ..., L, k = ℓ . Since it is not ϵ ℓ -constraint optimal, then there exists vector ˙ µ such that µ ( k ) ∗ ≤ ˙ µ ( k ) for k = 1 , ..., L and µ ( ℓ ) ∗ < ˙ µ ( ℓ ) . Since ˙ µ exists and dominates µ ∗ , this contradicts the definition of Pareto optimality.

̸

( ⇐ = ) Let µ ∗ be ϵ -constraint optimal. Suppose the constraint vector is defined as ϵ ( ℓ ) = µ ( ℓ ) ∗ for all ℓ ∈ [ L ] . Since µ ∗ is ϵ ℓ -constraint optimal for every ℓ = 1 , ..., L , there is no other µ ∈ Conv ( M ) satisfying µ ( ℓ ) ∗ < ˙ µ ( ℓ ) and µ ( k ) ∗ ≤ ˙ µ ( k ) when k = ℓ , for every ℓ = 1 , ..., L . This holds the definition of Pareto optimality.

Above lemma demonstrates that every Pareto optimal arm is also ϵ -constraint optimal. This equivalence also holds for non-convex set. But in the convex set Conv ( M ) , the mean reward vector

effective Pareto optimal arm satisfies the Pareto optimality in Conv ( M ) , hence, it also satisfies ϵ -constraint optimal. It is enough to show that, for any ϵ -constraint optimal vector µ ∗ ∈ Conv ( M ) , there exists weight vector w ∈ S L satisfying

To prove above equation, we prove some of the lemmas that are useful for our proof.

Lemma 3 let Ω be non-empty convex set in R L , not containing origin. Then there exists a vector w ∈ S L such that w ⊤ µ ≥ 0 holds for all µ ∈ Ω .

Proof. for µ 1 , µ 2 , ..., µ m ∈ Ω , define matrix and arbitrary vector as

by convexity of set Ω , we have M ⊤ β ∈ Ω , but 0 ̸∈ Ω . So, there is no solution β , satisfying

The solution still do not exist even if we remove the constraint ∥ β ∥ 1 = 1 . By Proposition 1, the second condition of Gordan's theorem does not hold. Hence, there exist L -dimensional vector w that w ⊤ µ i > 0 holds for all i = 1 , ..., m . Since w is non-zero vector, we can take w as ∑ ℓ ∈ [ L ] | w ( ℓ ) | = 1 . Define the set

Then we can write

̸

Each set V µ i is closed and bounded, hence, it is compact set. Since µ i was arbitrary chosen, the collection ( V µ ) µ satisfies finite intersection property. So, we have

̸

Lemma 4 Let Ω be non-empty convex set in R L , such that the vector µ ∈ Ω with all negative entries do not exist. Then, there exist vector w ∈ S L such that w ⊤ µ ≥ 0 holds for all µ ∈ Ω .

Proof. For a vector µ ∈ Ω , define the set

If origin is in B , then there exists µ that 0 > µ ( ℓ ) holds for all ℓ ∈ [ L ] , which contradicts the assumption. If y 1 ∈ B µ 1 , y 2 ∈ B µ 2 , we have

for γ ∈ [0 , 1] . Hence, B is convex set. By Lemma 3, there exist vector w , satisfying w ⊤ y ≥ 0 for all y ∈ B . If the vector has negative entry w ( ℓ ) < 0 , we can choose y ∈ B with large y ( ℓ ) so that w ⊤ y < 0 . Hence, we must have w ( ℓ ) ≥ 0 for all ℓ ∈ [ L ] . Also, since w is non-zero vector, we can restrict the vector in unit L -dimensional simplex. We now prove w ⊤ µ ≥ 0 for all µ ∈ Ω . For any positive real ϵ > 0 , we have µ + ϵ 1 ∈ B . If there exist δ > 0 , µ with w ⊤ µ = -δ , we can choose ϵ < δ , so that

Hence, we must have vector w that satisfies w ⊤ µ ≥ 0 for all µ ∈ Ω , and w ∈ S L

Lemma 5 (Generalized Gordan's Theorem) Let Ω be non-empty convex set. Either one of the following statements holds, but not both.

There exists µ ∈ Ω whose entries are all negative.

There exists L -dimensional vector w ∈ S L satisfying w ⊤ µ ≥ 0 for all µ ∈ Ω .

Proof. ( ¯ 1 = ⇒ 2 ) proof follows by Lemma 4. ( 2 = ⇒ ¯ 1 ) If µ is negative vector, we must have w ⊤ µ < 0 for all w ∈ S L .

Lemma 6 For any w ∈ S L , let A ∗ w be set of optimal arm, that has optimal weight sum reward given weight vector w , i.e.,

Then there exists effective Pareto optimal arm a ∗ , such that a ∗ ∈ A ∗ w .

Proof. Suppose there exist w that a ∗ ̸∈ A ∗ w for any a ∗ ∈ C ∗ . Let ¯ a ∈ A ∗ w / C ∗ that maximizes w ⊤ µ a . Since ¯ a ̸∈ C ∗ , for every effective Pareto optimal arm a ∗ ∈ C ∗ , there exist β a ∗ ≥ 0 satisfying

Since w is vector with non-negative entries, we have

We have, at least, one Pareto optimal arm with

Such existence of is guaranteed with existence of .

a ∗ β a ∗ > 0

Now, we begin the proof of Theorem 1

Proof. Suppose a ∗ is effective Pareto optimal. By Equation (4), for any µ ∈ Conv ( M ) \ { µ a ∗ } , we have µ a ∗ ̸≺ µ , hence, the vector µ a ∗ satisfies the Pareto optimality in Conv ( M ) . By Lemma 2, µ a ∗ is also ϵ -constraint optimal, with ϵ ( ℓ ) = µ ( ℓ ) a ∗ , such that no vector µ ∈ Conv ( M ) \ { µ a ∗ } satisfies µ ( k ) a ∗ -µ ( k ) h ≤ 0 for k = 1 , ..., L and µ ( ℓ ) a ∗ -µ ( ℓ ) h < 0 for some ℓ . By Lemma 5, since the set Conv ( M ) is convex, the first statement does not hold. There exists L -dimensional vector w ∈ S L satisfying w ⊤ ( µ a ∗ -µ ) ≥ 0 for all µ ∈ Conv ( M ) . Hence, we have

Conversly, Suppose a ∗ w = arg max a ∈A w ⊤ µ a is unique arm. By Lemma 6, the existence of effective Pareto optimal arm implies a ∗ w ∈ C ∗ .

B Analysis of MOLB-TS

In this section, we provide the analysis of the worst-case regret of algorithm MOLB-TS . We begin with the proof of Lemma 1.

B.1 Proof of Lemma 1

Proof. The parameters ( ˜ θ ( ℓ ) t,m ) m ∈ [ M ] are sampled from Gaussian distribution N ( ˆ θ ( ℓ ) t , c 2 1 ,t V -1 t ) . Then for any given d -dimensional vector x t,a , we can rewrite this probability distribution as

Also, we can see the probability of the event ˙ E ( ℓ ) t,a as where ( η m ) m ∈ [ M ] is sampled from standard normal distribution N (0 , 1) . For M = 1 , the probability of optimism P ( ˙ E ( ℓ ) t,a ) is bounded below by p . The probability of optimism, satisfying at least one sampled parameter, is bounded by

These optimism events between different objectives are independent. Let the event ˙ E t,a be defined as

The event ˙ E t,a is the optimism satisfying for all objectives ℓ ∈ [ L ] . Hence, we have

To have P ( ˙ E t,a ) ≥ p , we need to choose large enough M so that

Rearrange the equation, we get

With sampling by Gaussian distribution, we have p = 0 . 15 . However, the probability p can be different by choosing different sampling distribution. But, as long as p ∈ [0 , 1) , we have

Hence, by choosing M as we get P ( ˙ E t,a ) ≥ p

Lemma 1 shows the minimum number of M for the inequality P ( ˙ E t,a ) ≥ p to hold.

B.2 Proof of Theorem 2

Dependence on log L . Before proving Theorem 2, we define the events ˆ E t , ˜ E t such that the true parameters θ ( ℓ ) ∗ and all sampled parameters ( ˜ θ ( ℓ ) t,m ) m ∈ [ M ] are close enough to the RLS estimate parameters ˆ θ ( ℓ ) t for all objectives, respectively.

where c 1 ,t ( δ ) and c 2 ,t ( δ ) are defined as

Let ˆ E = ⋂ t ≥ 0 ˆ E t . By Lemma 8, we have P ( ˆ E ) ≥ 1 -δ , and by Lemma 9, on event ˆ E , we have P t ( ˜ E t ) = P ( ˜ E t | F t ) ≥ 1 -δ / T . The high probability bounds c 1 ,t ( δ ) and c 2 ,t ( δ ) increased by a factor of log L due to the union bound over the number of objectives. Consequently, the regret inevitably depends on log L as this factor arises from the concentration bounds required to hold uniformly over all objectives.

Bounding the sub-optimality gap. Wenowprove the following lemma, which bounds the conditional expectation of the regret of MOLB-TS at round t given the historical information up to that point.

Lemma 7 For any filtration F t -1 , on event ˆ E , we have

Proof. Let ( β t,a ∗ ) a ∗ ∈C ∗ t be one that maximizes ∆ EPR a t .

Let w t be the weight vector in unit L -simplex sampled, described in Section 5.2. Then,

By Theorem 1, there exists ¯ a ∗ ∈ C ∗ satisfying

Hence, we have

From M multiple sampled parameters, we define ˜ θ ( ℓ ) a,t as optimal sampled parameter with arm a , i.e.,

At round t , the arm a is evaluated with the sampled parameters ˜ θ ( ℓ ) a,t for all ℓ ∈ [ L ] . As we described in Section 5.2, we can write ¯ a ∗ , a t as

Let c t ( δ ) = c 1 ,t ( δ ) + c 2 ,t ( δ ) . We separated arms into two sets with given weight vector, saturated and unsaturated [4].

B t : set of saturated arms, that is, for all a ∈ B t , we have

B t : set of unsaturated arms, that is, for all a ∈ B t , we have

Note that w t is random variable, since the arm a t is uniform randomly selected from ˜ C t . Hence, those sets of saturated and unsaturated arms ( B t , ¯ B t ) are not fixed. Let ¯ a t = argmin a ∈ ¯ B t ∥ x t,a ∥ V -1 t be arm in ¯ B t with smallest matrix norm. From Equation (6), bounding the sub-optimality gap ∆ EPR a t on event ˆ E and ˜ E t , we have

For any arm a , by the Cauchy-Schwarz inequality of matrix norm, we have

And by triangle inequality of norm, we have

And lastly, on event ˆ E and ˜ E t , we get

In total, we get

Hence, on event ˆ E and ˜ E t ,

Since we have

where the last inequality holds since ¯ a t is unsaturated arm. This inequality holds on event ˆ E and ˜ E t . Define the conditional probability P t = P ( · | F t ) . Then, on event ˆ E , we have

We bound the term E t [ ∥ x t, ¯ a t ∥ V -1 t ] with ∥ x t,a t ∥ V -1 t . We have

The probability P t { a t ∈ B t } has randomness over algorithm selecting arm from empirical effective Pareto front ˜ C t , where the set B t varies on this random selection. More precisely, the set B t and B t change as w t changes. But, for any given w t , the probability P t { a t ∈ B t } bounds below by the probability that at least one unsaturated arm is evaluated higher compared to all saturated arms, i.e.,

where the unsaturated arm exists by ¯ a ∗ ∈ B t . Hence this probability bounds below by the probability that the arm ¯ a ∗ is evaluated higher compared to all saturated arms, i.e.,

On event ˜ E t , those saturated arms a ′ ∈ B t satisfy

and

Subtracting Equation (8) from Equation (7), we get

for all a ′ ∈ B t . Using this inequality, the Probability bounds as

Since ˜ θ ( ℓ ) ¯ a ∗ ,t is objective wise independent, the probability bounds by objective wise probability,

Removing the random vector w t , this inequality holds for any w t . In other words, this inequality holds for any random selection of arms from the set ˜ C t . Hence, we get

As we remove w t , the probability P t { a t ∈ B t } gets exponentially small as the number of objective increases. We remove this by adopting optimistic sampling strategy. With the number multiple samples M , following Lemma 1, we have

This inequality holds for any w t . With M = ⌈ 1 -log L log(1 -p ) ⌉ , we have (1 -(1 -p ) M ) L ≥ p . Finally, we have

Replacing the term ∥ x ¯ a t ∥ V -1 t to ∥ x a t ∥ V -1 t , we get.

Hence, we get

Now we begin the proof of Theorem 2.

Proof. We have

By Lemma 7, bounding the term E t [∆ EPR a t ] , we have

where the last inequality follows by Proposition 2,

C Additional Technical Tools

Proposition 1 (Gordan's Theorem, page 31, Mangasarian 12) For given matrix M ∈ R m × L , either one of the following statements holds, but not both.

There exists L -dimensional vector w , that M w has all positive entries.

M ⊤ β = 0 , β ≻ 0 has solution β ∈ R m .

Proposition 2 (Lemma 11, Abbasi-Yadkori et al. 1) Let λ ≥ 1 . For arbitrary sequence ( x t,a t ) t ∈ [ T ] , we have

Lemma 8 (Theorem 2, Abbasi-Yadkori et al. 1) Let ( F t ) t ≥ 0 be a filtration. Let ( ξ ( ℓ ) t ) be a realvalued stochastic process such that ξ ( ℓ ) t is conditionally R -sub-Gaussian, given filtration F t for any ℓ ∈ [ L ] . Then with probability at least 1 -δ , the event

holds for all t ≥ 1 .

Proof. By Theorem 2 in Abbasi-Yadkori et al. [1], and union bound with L .

Lemma 9 (Definition 1, Abeille and Lazaric 2) On event ˆ E t , with probability at least 1 -δ , all sampled parameters ( ˜ θ ( ℓ ) t,m ) m ∈ [ M ] ,ℓ ∈ [ L ] follow concentration property, i.e.,

for all m ∈ [ M ] , ℓ ∈ [ L ] .

Proof. By Definition 1 in Abeille and Lazaric [2], and union bound with M and L .

D Discussions

Our proposed algorithm demonstrates strong empirical performance with various settings. But its theoretical worst-case regret bound is not tighter than that of UCB-based algorithms. This gap between UCB-type algorithms and TS algorithms is well-known in the regret analysis of previous TS algorithms [4, 2] bounding the worst-case frequentist regret.

Our work is studied under the standard linear contextual bandit setting. Our framework can be readily extended to generalized linear contextual bandits. Extension to more complex function class such as neural networks requires analysis that is beyond the scope of this work, but is certainly the promising avenue for future work.

Yet, our work introduces the first randomized algorithm with Pareto regret guarantees in the multiobjective bandit framework. We hope that our work lays a foundation basis for extending such techniques to follow-up works.

E Computing resources for experiments

All experiments are conducted with INTEL(R) XEON(R) GOLD 6526Y CPU and 4 TB memory. The software environment includes Python 3.12.7, Scipy 1.14.1, and Numpy 1.26.4. The experiments took approximately 4 hours to 1 day, as it takes longer with increasing numbers of arms, dimensions, and objectives.

F Additional experimental results

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Pareto Regret

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 1

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 2

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 3

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 4

0

2500

5000

7500

10000

Round (

t

)

0

15

30

45

60

Pareto Regret

MOL-TS

(

M

= 1)

MOLB-TS

(

M

=

O

(log

L

))

0

2500

5000

7500

10000

Round (

t

)

0

15

30

45

60

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Pareto Regret

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 1

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 2

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 3

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 4

0

2500

5000

7500

10000

Round (

t

)

0

15

30

45

60

Pareto Regret

MOL-TS

(

M

= 1)

MOL-TS

(

M

=

O

(log

L

))

0

2500

5000

7500

10000

Round (

t

)

0

15

30

45

60

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Pareto Regret

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 1

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 2

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 3

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

5000

Cumulative Reward

objective

/lscript

= 4

0

2500

5000

7500

10000

Round (

t

)

0

15

30

45

60

Pareto Regret

MOL-TS

(

M

= 1)

MOL-TS

(

M

=

O

(log

L

))

0

2500

5000

7500

10000

Round (

t

)

0

15

30

45

60

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

50

100

150

200

Pareto Regret

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2500

5000

7500

10000

Round (

t

)

0

50

100

150

200

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

Cumulative Reward

objective

/lscript

= 1

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

Cumulative Reward

objective

/lscript

= 2

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

Cumulative Reward

objective

/lscript

= 3

0

2500

5000

7500

10000

Round (

t

)

0

1000

2000

3000

4000

Cumulative Reward

objective

/lscript

= 4

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Pareto Regret

MOL-TS

(

M

= 1)

MOL-TS

(

M

=

O

(log

L

))

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

50

100

150

200

Pareto Regret

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2500

5000

7500

10000

Round (

t

)

0

50

100

150

200

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 1

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 2

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 3

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 4

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Pareto Regret

MOL-TS

(

M

= 1)

MOL-TS

(

M

=

O

(log

L

))

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

50

100

150

200

Pareto Regret

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2500

5000

7500

10000

Round (

t

)

0

50

100

150

200

Effective Pareto Regret

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 1

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 2

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 3

0

2500

5000

7500

10000

Round (

t

)

0

1500

3000

4500

6000

Cumulative Reward

objective

/lscript

= 4

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Pareto Regret

MOL-TS

(

M

= 1)

MOL-TS

(

M

=

O

(log

L

))

0

2500

5000

7500

10000

Round (

t

)

0

25

50

75

100

Effective Pareto Regret

0

2000

4000

6000

8000

10000

Round (

t

)

0

25

50

75

100

125

150

175

200

Pareto Regret

0

2000

4000

6000

8000

10000

Round (

t

)

0

25

50

75

100

125

150

175

200

Effective Pareto Regret

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 1

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 2

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 3

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 4

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 5

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 6

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 7

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 8

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2000

4000

6000

8000

10000

Round (

t

)

0

25

50

75

100

125

150

175

200

Pareto Regret

0

2000

4000

6000

8000

10000

Round (

t

)

0

25

50

75

100

125

150

175

200

Effective Pareto Regret

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 1

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 2

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 3

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 4

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 5

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 6

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 7

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 8

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2000

4000

6000

8000

10000

Round (

t

)

0

100

200

300

400

500

Pareto Regret

0

2000

4000

6000

8000

10000

Round (

t

)

0

100

200

300

400

500

Effective Pareto Regret

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

/lscript

= 1

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

Cumulative Reward

/lscript

= 2

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

/lscript

= 3

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

/lscript

= 4

MOL-

/epsilon1

-greedy

MOL-UCB

PFIwR

MOL-TS

(

ours

)

0

2000

4000

6000

8000

10000

Round (

t

)

0

25

50

75

100

125

150

175

200

Pareto Regret

0

2000

4000

6000

8000

10000

Round (

t

)

0

25

50

75

100

125

150

175

200

Effective Pareto Regret

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 1

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 2

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 3

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 4

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 5

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 6

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 7

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

5000

6000

Cumulative Reward

objective

/lscript

= 8

MOL-

/epsilon1

-greedy

MOL-UCB

MOL-TS

(

ours

)

0

2000

4000

6000

8000

10000

Round (

t

)

0

100

200

300

400

500

Pareto Regret

0

2000

4000

6000

8000

10000

Round (

t

)

0

100

200

300

400

500

Effective Pareto Regret

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

Cumulative Reward

/lscript

= 1

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

Cumulative Reward

/lscript

= 2

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

Cumulative Reward

/lscript

= 3

0

2000

4000

6000

8000

10000

Round (t)

0

1000

2000

3000

4000

Cumulative Reward

/lscript

= 4

MOL-

/epsilon1

-greedy

MOL-UCB

PFIwR

MOL-TS

(

ours

)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study stochastic linear contextual bandits under the multi-objective setting, in which each arm corresponds to a vector of $L$ objective rewards, whose expected reward is the inner product between the hidden objective value and the arm’s context. As the Pareto front is non-convex, they define the effective Pareto optimality, which is essentially the convex hull of all arms. They measure performance using the cumulative minimal effective Pareto sub-optimality, defined as the $\\ell_\\infty$ distance between the performance and the convex hull.\n\nThey present a Thompson Sampling-based algorithm to solve this problem efficiently. Specifically, for each objective, they sample a logarithmic number of hidden objective parameters and estimate the empirical objective reward of each arm using the maximum inner product between the sampled parameters and the context. The algorithm then randomly plays an action sampled uniformly from the empirical Pareto front. They establish a regret bound of order $\\sqrt{T}$ for their algorithm. Experiments are provided to demonstrate the benefits of the algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- The multi-objective linear contextual bandit setting is definitely significant and interesting. Applying Thompson Sampling to this setting is a natural idea. The authors propose a clean algorithm for this setting, with a regret upper bound that matches the performance of Thompson Sampling in the linear contextual bandit setting.\n\n- I think optimistic sampling is a novel idea, as it allows one to perform optimistic estimation in a computationally efficient way compared to the UCB bounds ($\\mathcal{O}(d^2 M)$ for optimistic sampling per action vs. $\\mathcal{O}(d^3)$ in UCB). The corresponding intuition and proof sketch look correct to me. This analysis may shed light on future work in the bandit literature.\n\nWeakness:\n\n- I feel that the main reason one would prefer a TS-based approach over a UCB-based approach is its computational efficiency, as suggested by Agrawal et al. (2012), especially when there are many arms. Unfortunately, I don't see the authors highlighting this point. I would suggest that the authors add a section discussing the computational complexity of the proposed approach.\n\n\nAgrawal, Shipra and Navin Goyal. “Thompson Sampling for Contextual Bandits with Linear Payoffs.” International Conference on Machine Learning (2012).\n\nQUESTIONS:\n- How does $M$ affect the constant in the regret bound of Theorem 2? For example, what happens if $M$ doubles?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a Thompson sampling (TS) based approach for multi-objective linear bandits, with the goal of optimizing the effective Pareto regret. The authors define this new notion to overcome the issues associated with measuring cumulative regret using the Pareto regret idea. The algorithm (MOL-TS) essentially takes multiple samples from the posteriors to construct a randomized optimistic confidence bound for each arm, based on which the effective Pareto front is determined at that round, and action is taken randomly from this front. The paper establishes a $\\tilde{O}(d^{3/2}\\sqrt{T})$ worst-case regret bound, which matches the lower bound for single-objective linear bandit TS.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written and the results are promising. However, due to the missing appendix, the proofs cannot be verified and experimental details are missing at present.\n\nQUESTIONS:\n1. The optimistic TS using $M$ samples basically behaves like a UCB-type algorithm, by taking large number of samples and taking the maximum, this maximum (for large $M$) behaves like the mean added with an upper confidence term, depending on the Gaussian tail and $M$, however unlike UCB it is slightly randomized (amount of randomization controlled by $M$) - can the authors provide any comment on this connection?\n2. Can you explain why for each individual objective, greedy outperforms UCB, but somehow the Pareto regret is way better for UCB (in Figure 2)? This figure is misleading and leads to question the effectiveness of the Pareto regret (and effective PR) notion.\nAppendix is missing - proofs and other experimental details.\n3. Can you be more explicit in the regret bounds in the orders of dependence on $K$ and $L$? Is the bound tight in these? Also, it might be good to show the dependence on $M$ (or $p$) in theorem 2 and corollary 1.\n4. [Contextual Bandits with Linear Payoﬀ Functions by Chu et al. 2011] shows lower bound for linear contextual bandits (single-objective) of the order $\\Omega(\\sqrt{Td})$ and a matching upper bound (up to log terms). For TS, the recent [Geometry-Aware Approaches for Balancing Performance and Theoretical Guarantees in Linear Bandits by Luo et al] used a slight modification of TS to achieve this regret bound for randomized algorithms. Even in the high-dimensional case, the dependence on dimension and sparsity for TS is $O(s\\log d)$ (e.g. [Thompson Sampling for High-Dimensional Sparse Linear Contextual Bandits by Chakraborty et al.]), demonstrating better dependence on dimension (stochastic setting). - in the current paper, the dependence on $d$ seems to be slightly higher at $d^{3/2}$ - can the authors comment on this? At least the authors should include a brief discussion on single-objective regret bounds in the literature to claim “best known order for randomized linear bandit algorithms for single objective”.\n5. What is the parameter $\\delta$ in the algorithm 1?\n6. From a practician’s perspective, how do you tune all the hyper parameters in algorithm 1. Theoretical results give some indication on $M$, but what about the others, in particular $\\lambda, c$ and $\\delta$? Did you tune these parameters for the numerical experiments? If so, did you tune the associated parameters for greedy and UCB based methods for a fair comparison?\n\nLIMITATIONS:\nSee the questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the multi-objective linear bandit problem, which extends traditional linear contextual bandit problems by requiring optimization across multiple, often conflicting objectives simultaneously. In this paper, the authors proposes MOL-TS as the first randomized algorithm for multi-objective contextual bandits with Pareto regret guarantees, achieving $O(d^{3/2}\\sqrt{T})$ regret. Also, this paper firstly introduces \"effective Pareto optimal arm\" that not only satisfies standard Pareto optimality but also ensures total rewards across all objectives remain Pareto optimal with repeated selections. Numerical experiments demonstrate improved performance in both regret minimization and objective-wise total reward maximization\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n - This research addresses multi-objective bandit literature by introducing the first Thompson Sampling approach with theoretical guarantees. \n\n- The algorithm sample multiple models from the posterior distribution and compute an optimistic reward estimation to adapt to the multi-objective setting, which is novel.\n\n - The effective Pareto optimality concept provides a more robust framework for multi-objective optimization, ensuring better long-term performance across all objectives rather than just instantaneous optimality.\n\nWeaknesses\n- The regret bound achieved by this paper is $O(d^{3/2}\\sqrt{T})$, which is worse than UCB-based approaches for multi-objective linear bandits.\n\n- It would be better if the authors also discuss how to modify the previous approaches so that they could work for the new definition of Pareto optimality.\n\nQUESTIONS:\nIn feel-good TS[1], it is shown that for standard linear bandits, the regret upper bound is $O(d\\sqrt{T})$, which matches with the UCB-based approaches. Can similar techniques be applied in multi-objective linear bandit settings to achieve a tighter bound?\n\n[1] Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the multi-objective linear contextual bandit problem, where a decision-maker (or learner) must simultaneously optimize several potentially conflicting objectives. The goal is to learn a policy for selecting an effective Pareto optimal arm (Definition 5) for each given context, whose mean reward vector is Pareto optimal and maximizes the cumulative rewards across all objectives. The performance of the learned policy is measured in terms of minimizing effective Pareto regret (Definition 7), which is the sum of effective Pareto sub-optimality gap (defined in Eq. (1)).\n\n\nThe authors propose a Thompson sampling-based algorithm,  MOL-TS (Thompson Sampling for Multi-Objective Linear Bandits), for this setting with sub-linear effective Pareto regret guarantees. This paper introduces the notion of effective Pareto front, which consists of arms whose selection maximizes the total reward. The authors have also validated the performance of MOL-TS using synthetic problem instances and show that MOL-TS outperforms baseline algorithms in both regret minimization and cumulative objective-wise rewards.\n\nSTRENGTHS AND WEAKNESSES:\n#### **The following are the strengths of the paper:**\n1. This paper studies the multi-objective multi-armed bandit problem, which has applications in many areas, especially having several potentially conflicting objectives. \n\n2. This paper proposes a Thompson sampling-based algorithm, MOL-TS, to learn a policy to select the best effective Parato optimal arms for each context. They have theoretically shown that MOL-TS enjoys a sub-linear regret guarantee while choosing a better arm that maximizes the total reward.\n \n3. The authors have also empirically validated the performance (lower regret and higher cumulative rewards) of MOL-TS using synthetic problem instances.\n\n\n#### **The following are the weaknesses of the paper:**\n1. **Assumption:** Assuming a linear reward function may not be practical in real-life applications. Since there has been a lot of work considering the non-linear bandit setting, it would be much better to have a more practical algorithm by considering a general reward function.\n\n2. **Selecting the best arm:** Selecting an effective Pareto optimal arm can lead to a better performance than simply selecting a Pareto optimal arm. Since there can be multiple effective Pareto-optimal arms, it is unclear how to choose a single best arm among them. Doing this may need additional constraints, but it may make it more practical, as one can explain why a specific effective Pareto-optimal arm is selected. Otherwise, different effective Pareto-optimal arms may lead to different values of the objectives.\n\n3. **Analysis novelty:** The multi-objective linear contextual bandits and Thompson sampling-based bandit variants are now separately well-studied problems. From the paper, it is unclear what the key challenges are faced in the regret analysis, especially after getting an upper bound on the effective Pareto sub-optimality gap in Section 5.2.\n\nQUESTIONS:\nPlease address the weaknesses raised in **Strengths And Weaknesses***.\n\nLIMITATIONS:\nI have raised a few limitations of the paper in my response to the **Strengths And Weaknesses***. Since the paper is a theoretical contribution to bandit literature, I do not find any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper studies the multi-objective linear contextual bandit problem. The paper introduces a new concept effective Pareto regret and proposes a Thompson sampling based algorithm relying on the principle of optimism, The paper proves the effective Pareto regret (hence the Pareto regret) of the algorithm. The paper concludes with experiments.

STRENGTHS AND WEAKNESSES:
Strength: 
1. Significance: The paper establishes the first TS based algorithm with worst case Pareto regret guarantee, and the regret bound matches the best known order that in the single objective setting. The multi-objective is very common in real word scenarios but less studied.
2. Clarity: The presentation leading up to the introduction of the algorithm is very clear for someone like me who do not have the extensive background in multi-objective settings. 
3. Originality: I really like the optimistic sampling intuition

Weakness: 
1. I think this is minor but I think it would be interesting to see some stretch analysis on the number $L$ in the appendix (or in the main text). Is there any reason that you omitted the comparison between $M=1$ and $M = \log L$when $L=8$ in the appendix? What if $L$ keeps getting larger.

QUESTIONS:
1. This is still on the problem of $L$. I am curious why in the appendix if we keep $L$ the same, the difference between $M=1$ and $M=\log L$ is larger when $d$ is larger? 
2. I am also a bit confused about why $M=1$ does not seem to be that bad given the algorithm relies on optimistic sampling.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03