Title: Adaptive Algorithms for Relaxed Pareto Set Identification
Venue: NeurIPS 2023
Accepted: yes
Keywords: bandit

pure-exploration

pareto front

pareto set
Abstract: In this paper we revisit the fixed-confidence identification of the Pareto optimal set in a multi-objective multi-armed bandit model. As the sample complexity to identify the exact Pareto set can be very large, a relaxation allowing to output some additional near-optimal arms has been studied. In this work we also tackle alternative relaxations that allow instead to identify a relevant \emph{subset} of the Pareto set. Notably, we propose a single sampling strategy, called Adaptive Pareto Exploration, that can be used in conjunction with different stopping rules to take into account different relaxations of the Pareto Set Identification problem. We analyze the sample complexity of these different combinations, quantifying in particular the reduction in sample complexity that occurs when one seeks to identify at most $k$ Pareto optimal arms. We showcase the good practical performance of Adaptive Pareto Exploration on a real-world scenario, in which we adaptively explore several vaccination strategies against Covid-19 in order to find the optimal ones when multiple immunogenicity criteria are taken into account.

Adaptive Algorithms for Relaxed Pareto Set Identification

Abstract

In this paper we revisit the fixed-confidence identification of the Pareto optimal set in a multi-objective multi-armed bandit model. As the sample complexity to identify the exact Pareto set can be very large, a relaxation allowing to output some additional near-optimal arms has been studied. In this work we also tackle alternative relaxations that allow instead to identify a relevant subset of the Pareto set. Notably, we propose a single sampling strategy, called Adaptive Pareto Exploration, that can be used in conjunction with different stopping rules to take into account different relaxations of the Pareto Set Identification problem. We analyze the sample complexity of these different combinations, quantifying in particular the reduction in sample complexity that occurs when one seeks to identify at most k Pareto optimal arms. We showcase the good practical performance of Adaptive Pareto Exploration on a real-world scenario, in which we adaptively explore several vaccination strategies against Covid-19 in order to find the optimal ones when multiple immunogenicity criteria are taken into account.

1 Introduction

In a multi-armed bandit model, an agent sequentially collects samples from several unknown distributions, called arms, in order to learn about these distributions (pure exploration), possibly under the constraint to maximize the samples collected, viewed as rewards (regret minimization). These objectives have been extensively studied for different types of univariate arms distributions [21]. In this paper, we consider the less common setting in which arms are multi-variate distributions. We are interested in the Pareto Set Identification (PSI) problem. In this pure exploration problem, the agent seeks to identify the arms that are (Pareto) optimal , i.e. such that their expected values for all objectives are not uniformly worse than those of another arm.

Weformalize this as a fixed-confidence identification problem: in each round t the agent selects an arm A t using an adaptive sampling rule and observes a sample X t ∈ R D from the associated distribution. It further uses an adaptive stopping rule τ to decide when to stop sampling and output a set of arms ̂ S τ which is her guess for (an approximation of) the true Pareto set S /star . Given a risk parameter δ ∈ (0 , 1) , this guess should be correct with high probability, e.g. satisfy P ( ̂ S τ = S /star ) ≥ 1 -δ for exact Pareto set identification, while requiring a small sample complexity τ . This generalizes the well-studied fixed-confidence Best Arm Identification (BAI) problem [8, 15, 10] to multiple objectives.

Our motivation to study multi-objective adaptive identification stems from the design of adaptive earlyphase clinical trials. In phase I/II trials, the effects of a molecule in humans are explored, and several biological endpoints may be assessed at the same time as indicative markers of efficacy. In particular, in the context of vaccine development, early-phase trials usually assess multiple immunogenicity endpoints (i.e. various markers of the effects of the vaccine on the immune system, such as different facets of antibody responses or other immune parameters). In the absence of a known correlate of protection during early clinical development, these endpoints may not have a clear a priori hierarchy, may not all be correlated, which makes an examination of the Pareto set of different vaccinal strategies particularly relevant. In addition, given the availability of various vaccine platforms (such as mRNA vaccines, viral-vector vaccines, protein vaccines), as exemplified by Covid-19 vaccines, there may be a need to adaptively screen the various resulting vaccine strategies to select the most promising ones. Apart from clinical trials, active Pareto Set Identification can be meaningful in many real-word contexts, and we refer the reader to the various examples given by [28], such as in hardware or software design. Other applications include A/B/n testing for marketing or online recommender systems in which it is common to jointly optimize multiple (possibly conflicting) objectives such as user behavioral metrics (e.g. clicks, streams, dwell time, etc), supplier exposure objectives (e.g. diversity) and platform centric objectives (e.g. promotions) [22].

For many applications, the sample complexity of exact PSI can be prohibitive, either when there are many close to optimal arms or when the Pareto set is very large, and different relaxations have been considered in the literature [4, 27]. Going back to our motivation, in an adaptive trial that aims at pre-selecting a certain number of treatments or vaccine strategies for further investigations in clinical trials, practical constraints (the cost and feasibility of the trials) impose a constraint on the maximal number of interesting arms that can be identified. This motivates the introduction of a new setting where the agent is asked to identify at most k Pareto optimal arms. Interestingly the sampling rule that we propose for this setting can be used to solve (some generalizations of) other relaxations considered in the literature.

Related work The work most closely related to ours is that of Auer et al. [4], who propose a relaxation, which we refer to as ε 1 -PSI: their algorithm returns a set ̂ S that contains w.h.p. all the Pareto optimal arms and possibly some sub-optimal arms, which when increased by ε 1 coordinatewise become Pareto optimal. For arms that have sub-Gaussian marginals, they provide an instancedependent sample complexity bound scaling with some notion of sub-optimality gap for each arm. The work of Zaluaga et al. [28, 27] studies a structured variant of fixed-confidence PSI in which the means are regular functions of arms' descriptors. They use Gaussian process modeling and obtain worse-case sample complexity bounds. In particular [27] considers the identification of an ε -cover of the Pareto set, which is a representative subset of the ( ε ) -Pareto set that will be related to our ( ε 1 , ε 2 ) -PSI criterion. The algorithms of [4] and those of [28, 27] in the unstructured setting 1 have the same flavor: they sample uniformly from a set of active arms and remove arms that have been found sub-optimal (or not representative). Auer et al.[4] further adds an acceptation mechanism to stop sampling some of the arms that have been found (nearly-)optimal and are guaranteed not to dominate an arm of the active set. In this paper, we propose instead a more adaptive exploration strategy, which departs from such accept/reject mechanisms and is suited for different types of relaxation, including our novel k -relaxation.

Adaptive Pareto Exploration (APE) leverages confidence intervals on the differences of arms' coordinates in order to identify a single arm to explore, in the spirit of the LUCB [15] or UGapEc [9] algorithms for Topm identification in (one-dimensional) bandit models. These algorithms have been found out to be preferable in practice to their competitors based on uniform sampling and eliminations [19], an observation that will carry over to APE. Besides the multi-dimensional observations, we emphasize that a major challenge of the PSI problem with respect to e.g. Top m identification is that the number of arms to identify is not known in advance. Moreover, when relaxations are considered, there are multiple correct answers. In the one-dimensional settings, finding optimal algorithms in the presence of multiple correct answers is notoriously hard as discussed by the authors of [5], and their lower-bound based approach becomes impractical in our multi-dimensional setting. Finally, we remark that the k -relaxation can be viewed as an extension of the problem of identifying any k -sized subset out of the best m arms in a standard bandit [25].

Beyond Pareto set identification, other interesting multi-objective bandit identification problems have been studied in the literature. For example [6] propose an algorithm to identify some particular arms in the Pareto set through a scalarization technique [23]. The idea is to turn the multi-objective pure-exploration problem into a single-objective one (unique optimal arm) by using a real-valued preference function which is only maximized by Pareto optimal arms (see e.g [23] for some examples of these functions). In practice, a family of those functions can be used to identify many arms of the Pareto set but it is not always possible to identify the entire Pareto set using this technique (see e.g [7] for weighted sum with a family of weights vectors). In a different direction, the authors of [16] introduce the feasible arm identification problem, in which the goal is to identify the set of arms whose mean vectors belong to a known polyhedron P ⊂ R D . In a follow up work [17], they propose a fixed-confidence algorithm for finding feasible arms that further maximize a given weighted sum of the objectives. In clinical trials, this could be used to find treatments maximizing efficacy (or a weighted sum of different efficacy indicators), under the constraint that the toxicity remains below a threshold. However, in the presence of multiple indicators of biological efficacy, choosing the weights may be difficult, and an examination of the Pareto set could be more suitable. Finally, some papers consider extensions of the Pareto optimality condition. The authors of [1] tackle the identification of the set of non-dominated arms of any partial order defined by an R D polyhedral ordering cone (the usual Pareto dominance corresponds to using the cone defined by the positive orthant R D + ), and they provide worst-case sample complexity in the PAC setting. The work of [3] studies the identification of the set of non-dominated elements in a partially ordered set under the dueling bandit setting, in which the observations consists in pairwise comparison between arms.

Outline and contributions First, we formalize in Section 2 different relaxations of the PSI problem: ε 1 -PSI, as introduced by [4], ε 1 , ε 2 -PSI, of which a particular case was studied by [27] and ε 1 -PSIk , a novel relaxation that takes as input an upper bound k on the maximum number of ε 1 -optimal arms that can be returned. Then, we introduce in Section 3 Adaptive Pareto Exploration, a simple, adaptive sampling rule which can simultaneously tackle all three relaxations, when coupled with an appropriate stopping rule that we define for each of them. In Section 4, we prove high-probability upper bounds on the sample complexity of APE under different stopping rules. For ε 1 -PSI, our bound slightly improves upon the state-of-the-art. Our strongest result is the bound for ε 1 -PSIk , which leads to a new notion of sub-optimality gap, quantifying the reduction in sample complexity that is obtained. Then, Section 5 presents the result of a numerical study on synthetic datasets, one of them being inspired by a Covid-19 vaccine clinical trial. It showcases the good empirical performance of APE compared to existing algorithms, and illustrates the impact of the different relaxations.

2 Problem Setting

In this section, we introduce the Pareto Set Identification (PSI) problem and its relaxations. Fix K,D ∈ N /star . Let ν 1 , . . . , ν K be distributions over R D with means µ 1 , . . . , µ K ∈ R D . Let A := [ K ] := { 1 , . . . , K } denote the set of arms. Let ν := ( ν 1 , . . . , ν K ) and X := ( µ 1 , . . . , µ K ) . We use boldfaced symbols for R D elements. Letting X ∈ R D , u ∈ R , for any d ∈ { 1 , . . . , D } , X d denotes the d -th coordinate of X and X + u := ( X 1 + u, . . . , X D + u ) . In the sequel, we will assume that ν 1 , . . . , ν K have 1 -subgaussian marginals 2 .

Definition 1. Given two arms i, j ∈ A , i is weakly (Pareto) dominated by j (denoted by µ i ≤ µ j ) if for any d ∈ { 1 , . . . , D } , µ d i ≤ µ d j . The arm i is (Pareto) dominated by j ( µ i /precedesequal µ j or i /precedesequal j ) if i is weakly dominated by j and there exists d ∈ { 1 , . . . , D } such that µ d i < µ d j . The arm i is strictly (Pareto) dominated by j ( µ i ≺ µ j or i ≺ j ) if for any d ∈ { 1 , . . . , D } , µ d i < µ d j .

For ε ∈ R D + , the ε -Pareto set S /star ε ( X ) is the set of ε -Pareto optimal arms, that is:

In particular, S /star 0 ( X ) is called the Pareto set and we will simply write S /star ( X ) to denote S /star 0 ( X ) . When it is clear from the context, we write S /star (or S /star ε ) to denote S /star ( X ) (or S /star ε ( X ) ). By abuse of notation we write S /star ε when ε ∈ R + to denote S /star ε , with ε := ( ε , . . . , ε ) .

In each round t = 1 , 2 , . . . , the agent chooses an arm A t and observes an independent draw X t ∼ ν A t with E ( X A t ) = µ A t . We denote by P ν the law of the stochastic process ( X t ) t ≥ 1 and by E ν , the

expectation under P ν . Let F t := σ ( A 1 , X 1 , . . . , A t , X t ) the σ -algebra representing the history of the process. An algorithm for PSI consists in : i) a sampling rule which determines which arm to sample at time t based on history up to time t -1 , ii) a stopping rule τ which is a stopping time w.r.t the filtration ( F t ) t ≥ 1 and iii) a recommendation rule which is a F τ -measurable random set ̂ S τ representing the guess of the learner. The goal of the learner is to make a correct guess with high probability, using as few samples τ as possible. Before formalizing this, we introduce the different notion of correctness considered in this work, depending on parameters ε 1 ≥ , ε 2 ≥ 0 and k ∈ [ K ] . Our first criterion is the one considered by [4].

Definition 2. ̂ S ⊂ A is correct for ε 1 -PSI if S /star ⊂ ̂ S ⊂ S /star ε 1 . To introduce our second criterion, we need the following definition.

Definition 3. Let ε 1 , ε 2 ≥ 0 . A subset S ⊂ A is an ( ε 1 , ε 2 ) -cover of the Pareto set if : S ⊂ S /star ε 1 and for any i / ∈ S either i / ∈ S /star or ∃ j ∈ S such that µ i ≺ µ j + ε 2 .

/negationslash

The ε -accurate set of [27] is a particular case of ( ε 1 , ε 2 ) -cover for which ε 1 = ε 2 = ε . Allowing ε 1 = ε 2 generalizes the notion of ε -correct set and can be useful, e.g., in scenarios when we want to identify the exact Pareto set (setting ε 1 = 0 ) but allow some optimal arms to be discarded if they are too close (parameterized by ε 2 ) to another optimal arm already returned. We note however that the sparse cover of [4] is an ( ε , ε ) -cover with and additional condition that the arms in the returned set should not be too close to each over. Identifying a sparse cover from samples requires in particular to identify S /star ε 1 hence it can not be seen as a relaxation of ε 1 -PSI.

Definition 4. ̂ S ⊂ A is correct for ( ε 1 , ε 2 ) -PSI if it is an ( ε 1 , ε 2 ) -cover of the Pareto set. Definition 5. ̂ S ⊂ A is correct for ε 1 -PSIk if either i) | ̂ S | = k and ̂ S ⊂ S /star ε 1 or ii) | ̂ S | < k and S /star ⊂ ̂ S ⊂ S /star ε 1 holds.

We now introduce two important quantities to characterize the (Pareto) optimality or sub-optimality of the arms. For any two arms i, j , we let

Given a specified objective ( ε 1 -PSI, ( ε 1 , ε 2 ) -PSI or ε 1 -PSIk ), and a target risk parameter δ ∈ (0 , 1) , the goal of the agent is to build a δ -correct algorithm, that is to guarantee that with probability larger than 1 -δ , her guess ̂ S τ is correct for the given objective, while minimizing the number of samples τ needed to make the guess, called the sample complexity .

which have the following interpretation. If i /precedesequal j , m( i, j ) is the minimal quantity α ≥ 0 that should be added component-wise to µ i so that µ i + α ⊀ µ j , α := ( α , . . . , α ) . Moreover, m( i, j ) > 0 if and only if i ≺ j . Then, for any arms i, j , if i ⊀ j , M( i, j ) is the minimum quantity α ′ such µ i ≤ µ j + α ′ , α ′ := ( α ′ , . . . , α ′ ) . We remark that M( i, j ) < 0 if and only if i ≺ j . Our algorithms, presented in the next section, rely on confidence intervals on these quantities.

3 Adaptive Pareto Exploration

We describe in this section our sampling rule, Adaptive Pareto Exploration, and present three stopping and recommendation rules to which it can be combined to solve each of the proposed relaxation. Let T k ( t ) := ∑ t -1 s =1 ( A s = k ) be the number of times arm k has been pulled up to round t and ̂ µ k ( t ) := T k ( t ) -1 ∑ T k ( t ) s =1 X k,s the empirical mean of this arm at time t , where X k,s denotes the s -th observation drawn i.i.d from ν k . For any arms i, j ∈ A , we let

The empirical Pareto set is defined as

3.1 Generic algorithm(s)

Adaptive Pareto Exploration relies on a lower/upper confidence bound approach, similar to singleobjective BAI algorithms like UGapEc [9], LUCB[15] and LUCB++ [26]. These three algorithms identify in each round two contentious arms: b t : a current guess for the optimal arm (defined as the empirical best arm or the arm with the smallest upper bound on its sub-optimality gap), c t : a contender of this arm; the arm which is the most likely to outperform b t (in all three algorithms, it is the arm with the largest upper confidence bound in [ K ] \{ b t } ). Then, either both arms are pulled (LUCB, LUCB++) or the least explored among b t and c t is pulled (UGapEc). The originality of our sampling rule lies in how to appropriately define b t and c t for the multi-objective setting. To define those, we suppose that there exists confidence intervals [ L d i,j ( t, δ ) , U d i,j ( t, δ )] on the difference of expected values for each pair of arms ( i, j ) and each objective d ∈ D , such that introducing

/negationslash

we have P ( E ) ≥ 1 -δ . Concrete choices of these confidence intervals will be discussed in Section 3.2. To ease the notation, we drop the dependency in δ in the confidence intervals and further define

Lemma 1. For any round t ≥ 1 , if E t holds, then for any i, j ∈ A , M -( i, j, t ) ≤ M( i, j ) ≤ M + ( i, j, t ) and m -( i, j, t ) ≤ m( i, j ) ≤ m + ( i, j, t ) .

/negationslash

Noting that S /star ε 1 = { i ∈ A : ∀ j = i, M( i, j ) + ε 1 > 0 } , we define the following set of arms that are likely to be ε 1 -Pareto optimal:

Sampling rule In round t , Adaptive Pareto Exploration samples a t , the least pulled arm among two candidate arms b t and c t given by

/negationslash

/negationslash

The intuition for their definition is the following. Letting i be a fixed arm, note that M( i, j ) > 0 for some j , if and only if there exists a component d such that µ d i > µ d j i.e i is not dominated by j . Moreover, the larger M( i, j ) , the more i is non-dominated by j in the sense that there exists d such that µ d i /greatermuch µ d j . Therefore, i is strictly optimal if and only if for all j = i , M( i, j ) > 0 i.e α i := min j = i M( i, j ) > 0 . And the larger α i , the more i looks optimal in the sense that for each arm j = i , there exists a component d j for which i is way better than j . As the α i are unknown, we define b t as the maximizer of an optimistic estimate of the α i 's. We further restrict the maximization to arms for which we are not already convinced that they are optimal (by Lemma 1, the arms in OPT ε 1 ( t ) are (nearly) Pareto optimal on the event E ). Then, we note that for a fixed arm i , M( i, j ) < 0 if and only if i is strictly dominated by j . And the smaller M( i, j ) , the more j is close to dominate i (or largely dominates it): for any component d , µ d i -µ d j is small (or negative). Thus, for a fixed arm i , argmin j = i M( i, j ) can be seen as the arm which is the closest to dominate i (or which dominates it by the largest margin). By minimizing a lower confidence bound on the unknown quantity M( b t , j ) , our contender c t can be interpreted as the arm which is the most likely to be (close to) dominating b t . Gathering information on both b t and c t can be useful to check whether b t can indeed be optimal.

/negationslash

Interestingly, we show in Appendix E that for D = 1 , our sampling rule is close but not identical to the sampling rules used by existing confidence-based best arm identification algorithms.

/negationslash

/negationslash

/negationslash

Stopping and recommendation rule(s) Depending on the objective, Adaptive Pareto Exploration can be plugged in with different stopping rules, that are summarized in Table 1 with their associated recommendations. To define those, we define for all i ∈ A , ε 1 , ε 2 ≥ 0 ,

/negationslash and let g i ( t ) := g 0 i ( t ) . Introducing

for ε 1 -PSI, our stopping rule is τ ε 1 := inf { t ≥ K : Z ε 1 1 ( t ) > 0 ∧ Z ε 1 2 ( t ) > 0 } and the associated recommendation is O ( τ ε 1 ) where

/negationslash consists of the current empirical Pareto set plus some additional arms that have not yet been formally identified as sub-optimal. Those arms should be ( ε 1 ) -Pareto optimal.

For ( ε 1 , ε 2 ) -PSI we define a similar stopping rule τ ε 1 , ε 2 where the stopping statistics are respectively replaced with

with the convention min ∅ = + ∞ , and the recommendation is OPT ε 1 ( τ ε 1 , ε 2 ) .

To tackle the ε 1 -PSIk relaxation, we propose to couple τ ε 1 with an additional stopping condition checking whether OPT ε 1 ( t ) already contains k arms. That is, we stop at τ k ε 1 := min ( τ ε 1 , τ k ) where τ k := inf { t ≥ K : | OPT ε 1 ( t ) | ≥ k } with associated recommendation OPT ε 1 ( τ k ) . Depending of the reason for stopping ( τ ε 1 or τ k ), we follow the corresponding recommendation.

Lemma 2. Assume E holds. For ε 1 -PSI (resp. ( ε 1 , ε 2 ) -PSI , ε 1 -PSIk ), Adaptive Pareto Exploration combined with the stopping rule τ ε 1 (resp. τ ε 1 , ε 2 , resp. τ k ε 1 ) outputs a correct subset.

Remark 1. We decoupled the presentation of the sampling rule to that of the 'sequential testing' aspect (stopping and recommendation). We could even go further and observe that multiple tests could actually be run in parallel, for free. If we collect samples with APE (which only depends on ε 1 ), whenever one of the three stopping conditions given in Table 1 triggers, for any values of ε 2 or k , we can decide to stop and make the corresponding recommendation or continue and wait for another 'more interesting' stopping condition to be satisfied. If E holds, a recommendation made at any such time will be correct for the objective associated to the stopping criterion (third column in Table 1).

3.2 Our instantiation

We propose to instantiate the algorithms with confidence interval on the difference of pair of arms. For any pair i, j ∈ A , we define a function β i,j such that for any d ∈ [ D ] , U d i,j ( t ) = ̂ µ d i ( t ) -̂ µ d j ( t )+ β i,j ( t ) and L d i,j ( t ) = ̂ µ d i ( t ) -̂ µ d j ( t ) -β i,j ( t ) . We take from [20] the following confidence bonus for timeuniform concentration:

where K 1 := K ( K -1) D/ 2 and C g ≈ x + log( x ) is a calibration function. They result in the simple expressions M ± ( i, j, t ) = M( i, j, t ) ± β i,j ( t ) and m ± ( i, j, t ) = m( i, j, t ) ± β i,j ( t ) . As an

/negationslash

/negationslash

/negationslash example, we state in Algorithm 1 the pseudo-code of APE combined the stopping rule suited for the k -relaxation of ε 1 -PSI, which we refer to as ε 1 -APEk .

In Appendix F, we also study a different instantiation based on confidence bounds of the form U i,j ( t ) = U i ( t ) -L j ( t ) where [ L i ( t ) , U i ( t )] is a confidence interval on µ i . This is the approach followed by LUCB for D = 1 and prior work on Pareto identification [4, 27]. In practice we advocate the use of the pairwise confidence intervals defined above, even if our current analysis does not allow to quantify their improvement. For the LUCB-like instantiation, we further derive in Appendix F an upper bound on the expected stopping time of APE for the different stopping rules.

4 Theoretical analysis

In this section, we state our main theorem on the sample complexity of our algorithms and give a sketch of its proof.

First let us introduce some quantities that are needed to state the theorem. The sample complexity of the algorithm proposed by [4] for ( ε 1 ) -Pareto set identification scales as a sum over the arms i of 1 / ( ∆ i ∨ ε 1 ) 2 where ∆ i is called the sub-optimality gap of arm i and is defined as follows. For a sub-optimal arm i / ∈ S /star ( X ) ,

which is the smallest quantity that should be added component-wise to µ i to make i appear Pareto optimal w.r.t { µ i : i ∈ A } . For a Pareto optimal arm i ∈ S /star ( X ) , the definition is more involved:

where

For x ∈ R , ( x ) + := max( x, 0) . We also introduce some additional notion needed to express the contribution of the k -relaxation. Let 1 ≤ k ≤ K . For any arm i , let ω i = min j = i M( i, j ) and define

with the k -th max and first to k -th argmax operators. Observe that ω k > 0 if and only if |S /star ( X ) | ≥ k .

Theorem 1. Fix a risk parameter δ ∈ (0 , 1) , ε 1 ≥ 0 , let k ≤ K and ν a bandit with 1 -subgaussian marginals. With probability at least 1 -δ , ε 1 -APEk recommends a correct set for the ε 1 -PSIk objective and stops after at most samples, where for each a ∈ A , ˜ ∆ a := max( ∆ a , ε 1 , ω k ) .

/negationslash

/negationslash

First, when k = K , observing that ε 1 -APEK provides a δ -correct algorithm for ε 1 -PSI, our bound improves the result of [4] for the ε 1 -PSI problem in terms of constant multiplicative factors and log log ∆ -1 terms instead of log ∆ -2 . It nearly matches the lower bound of Auer et al.[4] for the ε 1 -PSI problem (Theorem 17 therein). It also shows the impact of the k -relaxation on the sample complexity. In particular, we can remark that for any arm i ∈ S /star \S /star ,k , max( ∆ i , ω k ) = ω k . Intuitively, it says that we shouldn't pay more than the cost of identifying the k -th optimal arm, ordered by the ω ′ i s. A similar result has been obtained for the any k -sized subset of the best m problem [25]. But the authors have shown the relaxation only for the best m arms while our result shows that even the sub-optimal arms should be sampled less.

In Appendix D, we prove the following lower bound showing that in some scenarios, ε 1 -APEk is optimal for ε 1 -PSIk , up to D log( K ) and constant multiplicative terms. We note that for ε 1 -PSI a lower bound featuring the gaps ∆ a and ε 1 was already derived by Auer et al. [4].

Theorem 2. There exists a bandit instance ν with |S /star | = p ≥ 3 such that for k ∈ { p, p -1 , p -2 } any δ -correct algorithm for 0 -PSIk verifies

where ∆ k a := ∆ a + ω k and τ δ is the stopping time of the algorithm.

In Appendix C, we prove that Theorem 1 without the ω k terms also holds for ( ε 1 , ε 2 ) -APE. This does not justifies the reduction in sample complexity when setting ε 2 > 0 in ( ε 1 , ε 2 ) -PSI observed in our experiments but it at least guarantees that the ε 2 -relaxation doesn't make things worse.

Furthermore, since our algorithm allows ε 1 = 0 , it is also an algorithm for BAI when D = 1 , ε 1 = 0 . We prove in Appendix E that in this case, the gaps ∆ ′ i s matches the classical gaps in BAI [2, 18] and we derive its sample complexity from Theorem 1 showing that it is similar in theory to UGap [9], LUCB[15] and LUCB++ [26] but have better empirical performance.

Sketch of proof of Theorem 1 Using Proposition 24 of [20] we first prove that the choice of β i,j in (4) yields P ( E ) ≥ 1 -δ for the good event E defined in (1). Combining this result with Lemma 2, yields that ε 1 -APEk is correct with probability at least 1 -δ .

The idea of the remaining proof is to show that under the event E , if APE has not stopped at the end of round t , then the selected arm a t has not been explored enough. The first lemma showing this is specific to the stopping rule τ k ε 1 used for ε 1 -PSIk .

Lemma 3. Let ε 1 ≥ and k ∈ [ K ] . If E t holds and t < τ k ε 1 then ω k ≤ 2 β a t ,a t ( t ) .

The next two lemmas are more general as they apply to different stopping rules.

Lemma 4. Let ε 1 ≥ 0 . Let τ = τ k ε 1 for some k ∈ [ K ] or τ = τ ε 1 , ε 2 for some ε 2 ≥ 0 . If E t holds and t < τ then ∆ a t ≤ 2 β a t ,a t ( t ) .

Lemma 5. Let ε 1 ≥ 0 and τ be as in Lemma 4. If E t holds and t < τ then ε 1 ≤ 2 β a t ,a t ( t ) .

As can be seen in Appendix B, the proofs of these three lemmas heavily rely on the specific definition of b t and c t . In particular, to prove Lemma 4 and 5, we first establish that when t < τ , any arm j ∈ A satisfies m( b t , j, t ) ≤ β b t ,j ( t ) . The most sophisticated proof is then that of Lemma 4, which relies on a case distinction based on whether b t or c t belongs to the set of optimal arms.

These lemmas permit to show that, on E t if t < τ k ε 1 then ˜ ∆ a t < 2 β a t ,a t ( t ) ≤ 2 β T a t ( t ) , where we define β n to be the expression of β i,j ( t ) when T i ( t ) = T j ( t ) = n . Then we have

A careful upper-bounding of the RHS of (5) completes the proof of Theorem 1.

5 Experiments

We evaluate the performance of Adaptive Pareto Exploration on a real-world scenario and on synthetic random Bernoulli instances. For a fair comparison, Algorithm 1 of [4], referred to as PSI-Unif-Elim and APE are both run with our confidence bonuses β i,j ( t ) on pairs of arms, which considerably improve single-arm confidence bonuses 3 . As anytime confidence bounds are known to be conservative, we use K 1 = 1 in (4) instead of its theoretical value coming from a union bound. Still, in all our experiments, the empirical error probability was (significantly) smaller than the target δ = 0 . 1 .

Real-world dataset COV-BOOST [24] is phase 2 trial which was conducted on 2883 participants to measure the immunogenicity of different Covid-19 vaccines as third dose (booster) in various combinations of initially received vaccines (first two doses). This resulted in a total of 20 vaccination strategies being assessed, each of them defined by the vaccines received as first, second and third dose. The authors have reported the average responses induced by each candidate strategy on cohorts of participants, measuring several immunogenicity markers. From this study, we extract and process the average response of each strategy to 3 specific immunogenicity indicators: two markers of antibody response and one of the cellular response. The outcomes are assumed to have a log-normal distribution [24]. We use the average (log) outcomes and their variances to simulate a multivariate Gaussian bandit with K = 20 , D = 3 . We give in Appendix H.2 some additional details about the processing of the data, and report the means and variances of each arm. In Appendix H.1 we further explain how APE can be simply adapted when the marginals distributions of the arms have different variances. In this experiment, we set ε 1 = 0 , δ = 0 . 1 and compare PSI-Unif-Elim to 0 -APEk (called

APEk in the sequel) for different values of k . The empirical distribution of the sample complexity of the algorithms, averaged over 2000 independent runs, are reported in Figure 1. The results are shown in log-scale ( y -axis is the log of the sample complexity) to fit in the same figure. As |S /star | = 2 , we first observe that, without the relaxation (i.e. for k > 3 ), APE outperforms its state-of-the-art competitor PSI-Unif-Elim . Moreover for k = 1 or k = 2 , the sample complexity of APEk is significantly reduced. For k = 2 when the stopping time τ k is reached some sub-optimal arms have possibly not yet been identified as such, while for k = 3 , even if the optimal arms have been identified, the remaining arms have to be sampled enough to ensure that they are sub-optimal before stopping. This explains the gap in sample complexity between k = 2 and k = 3 . In Appendix H.3, we compare APE to an adaptation of PSI-Unif-Elim for the k -relaxation, showing that APE is always preferable.

Experiments on random instances To further illustrate the impact of the k -relaxation and to strengthen the comparison with PSI-Unif-Elim , we ran the previous algorithms on 2000 randomly generated multi-variate Bernoulli instances, with K = 5 arms and different values of the dimension D . We set δ = 0 . 1 and ε 1 = 0 . 005 (to have reasonable running time). The averaged sample complexities are reported in Table 2. We observe that APE (with k = K ) uses 20 to 25% less samples

than PSI-Unif-Elim and tends to be more efficient as the dimension increases (and likely the size of the Pareto set, since the instances are randomly generated). We also note that identifying a k -sized subset of the Pareto set requires considerably less samples than exact PSI. In Appendix H.3 we also provide examples of instances for which APE takes up to 3 times less samples than PSI-Unif-Elim .

To illustrate the impact of the ε 2 relaxation, setting ε 1 = 0 we report the sample complexity of APE associated with the stopping time τ 0 , ε 2 for 20 equally spaced values of ε 2 ∈ [0 . 01 , 0 . 05] , averaged over 2000 random Bernoulli instances. Figure 2 shows the decrease of the average sample complexity when ε 2 increases (left) and the average ratio of the size of the returned set to the size of the Pareto set (right). Note that for ε 1 = 0 , we have O ( τ 0 , ε 2 ) ⊂ S /star . The average sample complexity reported decreases up to 86% for the instance with K = 5 , D = 2 while the returned set contains more than 90% of the Pareto optimal arms. In Appendix H.3, we further illustrate the behavior of APE with the ε 2 relaxation on a fixed instance in dimension 2.

(a) Average sample complexity vs

ε

2

(b) Average ratio

|O

(

τ

0

,

ε

2

)

|

/

|S

/star

|

vs

ε

2

Instance type

1.00

70000

★

K=5,D=2

+

K=5, D =4

0.99

60000

K=5,D=8

50000

0.98

[0,1

40000

[s*]

0.97

30000

0.96

20000

10000

0.95

0.010

0.0150.020

0.025

0.030

0.035

0.040

0.045

0.050

0.010

0.015

0.020

0.025

0.030

0.035

0.040

0.045

0.050

E2

E2

6 Conclusion and perspective

We proposed and analyzed APE, an adaptive sampling rule in multi-variate bandits models that when coupled with different stopping rules can tackle different relaxations of the fixed-confidence Pareto Set Identification problem. Our experiments revealed the good performance of the resulting algorithms compared to the state-of-the-art PSI algorithm as well as the great reductions in sample complexity brought by the relaxations. In future work, we intend to make our algorithms more practical for possible applications to clinical trials. For this purpose, as measuring efficacy takes time, we will investigate its adaptation to a batch setting, following, e.g. the work of [13] for BAI. We will also investigate the use of APE beyond the fixed-confidence setting, to the possibly more realistic fixed-budget [2] or anytime exploration [14] settings. To the best of our knowledge, no algorithm exists in the literature for PSI in such settings. Finally, following the works of [28, 4], we defined the ε 1 , ε 2 relaxations with scalar values, so that the same slack applies to all components. Although we could easily modify our algorithms to tackle vectorial values ε 1 , ε 2 , so far we could only prove a dependence on min d ε d 1 in the sample complexity. We intend to study the right quantification in the sample complexity when ε 1 and ε 2 are vectorial.

Acknowledgments and Disclosure of Funding

Cyrille Kone is funded by an Inria/Inserm PhD grant. Emilie Kaufmann acknowledges the support of the French National Research Agency under the projects BOLD (ANR-19-CE23-0026-04) and FATE (ANR22-CE23-0016-01).

References

C. Ararat and C. Tekin. Vector optimization with stochastic bandit feedback. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206, pages 2165-2190. PMLR, 2023.

J.-Y. Audibert and S. Bubeck. Best arm identification in multi-armed bandits. In COLT - 23th Conference on Learning Theory - 2010 , page 13 p., 2010.

J. Audiffren and L. Ralaivola. Bandits dueling on partially ordered sets. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

P. Auer, C.-K. Chiang, R. Ortner, and M.-M. Drugan. Pareto front identification from stochastic bandit feedback. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , volume 51, pages 939-947. PMLR, 2016.

R. Degenne and W.M. Koolen. Pure exploration with multiple correct answers. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

M.-M. Drugan and A. Nowe. Designing multi-objective multi-armed bandits algorithms: A study. In The 2013 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2013.

M. Ehrgott. Multicriteria Optimization . Springer, second edition edition, 2005.

E. Even-Dar, S. Mannor, and Y. Mansour. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of Machine Learning Research , 7(39):1079-1105, 2006.

V. Gabillon, M. Ghavamzadeh, and A. Lazaric. Best arm identification: A unified approach to fixed budget and fixed confidence. In Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012.

A. Garivier and E. Kaufmann. Optimal best arm identification with fixed confidence. In 29th Annual Conference on Learning Theory , volume 49, pages 998-1027. PMLR, 2016.

K. Jamieson, M. Malloy, R. Nowak, and S. Bubeck. lil' ucb : An optimal exploration algorithm for multi-armed bandits. In Proceedings of The 27th Conference on Learning Theory , volume 35, pages 423-439. PMLR, 2014.

M. Jourdan, R. Degenne, D. Baudry, R. de Heide, and E. Kaufmann. Top two algorithms revisited. In Advances in Neural Information Processing Systems , volume 35, pages 2679126803. Curran Associates, Inc., 2022.

K.-S. Jun, K. Jamieson, R. Nowak, and X. Zhu. Top arm identification in multi-armed bandits with batch arm pulls. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , volume 51, pages 139-148. PMLR, 2016.

K-S. Jun and R. Nowak. Anytime exploration for multi-armed bandits using confidence information. In Proceedings of The 33rd International Conference on Machine Learning , volume 48, pages 974-982. PMLR, 2016.

S. Kalyanakrishnan, A. Tewari, P. Auer, and P. Stone. PAC subset selection in stochastic multiarmed bandits. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 227-234. Omnipress, 2012.

J. Katz-Samuels and C. Scott. Feasible arm identification. In Proceedings of the 35th International Conference on Machine Learning , volume 80, pages 2535-2543. PMLR, 2018.

J. Katz-Samuels and C. Scott. Top feasible arm identification. In Proceedings of the TwentySecond International Conference on Artificial Intelligence and Statistics , pages 1593-1601. PMLR, 2019.

E. Kaufmann, O. Cappé, and A. Garivier. On the complexity of best-arm identification in multi-armed bandit models. Journal of Machine Learning Research , 17(1):1-42, 2016.

E. Kaufmann and S. Kalyanakrishnan. Information complexity in bandit subset selection. In Conference On Learning Theory , volume 30. JMLR: Workshop and Conference Proceedings, 2013.

E. Kaufmann and W.-M. Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. Journal of Machine Learning Research , 22(246):1-44, 2021.

T. Lattimore and C. CSzepesvári. Bandit Algorithms . Cambridge University Press, 2020.

R. Mehrotra, N. Xue, and M. Lalmas. Bandit based optimization of multiple objectives on a music streaming platform. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , page 3224-3233. Association for Computing Machinery, 2020.

K. Miettinen. Nonlinear multiobjective optimization. In International Series in Operations Research and Management Science , 1998.

A.-P.-S. Munro, L. Janani, V. Cornelius, and et al. Safety and immunogenicity of seven COVID19 vaccines as a third dose (booster) following two doses of ChAdOx1 nCov-19 or BNT162b2 in the UK (COV-BOOST): a blinded, multicentre, randomised, controlled, phase 2 trial. The Lancet , 398(10318):2258-2276, 2021.

A. Roy Chaudhuri and S. Kalyanakrishnan. PAC identification of many good arms in stochastic multi-armed bandits. In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 991-1000. PMLR, 2019.

M. Simchowitz, K. Jamieson, and B. Recht. The simulator: Understanding adaptive sampling in the moderate-confidence regime. In Proceedings of the 2017 Conference on Learning Theory , volume 65, pages 1794-1834. PMLR, 2017.

M. Zuluaga, A. Krause, and M. Püschel. e-pal: An active learning approach to the multiobjective optimization problem. Journal of Machine Learning Research , 17(104):1-32, 2016.

M. Zuluaga, G. Sergent, A. Krause, and M. Püschel. Active learning for multi-objective optimization. In Proceedings of the 30th International Conference on Machine Learning , volume 28, pages 462-470. PMLR, 2013.

Peer reviews:
comments: SUMMARY:
This paper studies a less common setting that arms are multi-variate distributions. In the expiration period, the agent seeks to identify the arm which is Pareto optimal. 

This problem is formulated as a fixed-confidence identification problem. The most novel setting in that, the fixed confidence identification problem s extended for multi-objective settings. 

This paper proposed a novel sampling rule called Adaptive Pareto Exploration. The central idea is to identify the two contentious arms and sample both or one of them. 

The paper also gives the theoretical guarantee that  the proposed method can recommend a correct set whp after most a given number of iterations.

STRENGTHS:
This paper proposes a principled method and provides theoretical guarantees.

WEAKNESSES:
It seems trival to me to extend the single objective best arm  identification problem to the mo setting.

QUESTIONS:
1. Can the proposed method identify the full (relaxed) Pareto front?

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe primary objective of the paper is to address the challenge of identifying a set of arms, consisting of at most $k$ arms, where each arm is either Pareto optimal or close to Pareto optimal. The paper explains the concept of Pareto optimality within the context of bandit problems. To tackle this problem, the paper presents the $\\epsilon_1$-APE-$k$ (Adaptive Pareto Exploration) algorithm and establishes an upper bound on the sample complexity. Furthermore, empirical evidence is provided to demonstrate the effectiveness of the proposed algorithm.\n\nSTRENGTHS:\nOne strength of the paper is its focus on addressing the problem of Pareto optimality in bandit literature, which is an underexplored area. The paper introduces a novel problem setup and clearly defines the goals for different scenarios related to Pareto optimal identification, including the identification of Pareto optimal sets, near Pareto optimal sets, and at-most $k$ near Pareto optimal sets.\n\nThe paper excels in providing a comprehensive discussion on sample complexity upper bounds, highlighting the theoretical aspects of the problem. It also establishes connections to the existing literature on Pareto optimality in bandits, demonstrating a solid understanding of the research landscape.\n\nFurthermore, the paper offers ample experimental evidence to support its claims, including experiments conducted on real-world datasets. This empirical validation strengthens the credibility of the proposed approaches and enhances the practical relevance of the research.\n\nWEAKNESSES:\nOne weakness of the paper is the absence of references to real-world applications that could benefit from the proposed framework. For instance, discussing potential applications such as A/B testing in clinical trials would enhance the practical relevance and broader impact of the research.\n\nAnother weakness is the lack of discussion of lower bounds in the main paper. Given that the problem setup for Pareto set identification is relatively new, it would be valuable to explore more properties of the lower bounds to gain insights into the tightness of the derived upper bounds. This would provide a more comprehensive understanding of the problem and its inherent complexities.\n\nThe paper lacks a thorough discussion on the computational complexity of the $\\epsilon_1$-APE-$k$ algorithm. Considering the importance of computational efficiency in practical applications, it would be beneficial to address\n\nQUESTIONS:\nOne potential weakness is the lack of information regarding scalability issues encountered when running large-scale experiments using the $\\epsilon_1$-APE-$k$ algorithm. It would be valuable to understand whether the algorithm faces any challenges in handling larger datasets or more complex scenarios. Additionally, providing references on the scale of parameters used in A/B testing for applications beyond COVID datasets would further strengthen the practicality of the proposed approach.\n\nAnother point to consider is the limited discussion on the broader scope of utilizing Pareto sets in various applications apart from clinical trials. Exploring and discussing other potential domains where Pareto sets could be beneficial would enhance the paper's impact and shed light on additional practical applications.\n\nLIMITATIONS:\nNo limitations or potential impact of their work discussed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper extends the best arm bandit problem to a multi-objective version to identify the arms that are in the Pareto front.

STRENGTHS:
This problem is new and the author provides a clear practice motivation of this problem.

The theory study seems valid and complete.

WEAKNESSES:
The math notation seems a bit over-complicated and I was wondering can you simplify it?

Overall, extending the best arm bandit problem into multi-object setting seems straightforward, which limits the novelty of this paper.

The experiment result looks a bit hand-wavy. I understand that there might not be any algorithm specifically designed for such problem, but I was wondering how the proposed method compares with some naively modified baselines?

QUESTIONS:
In the definition of Pareto front, why we don't consider the variance of the distribution?

LIMITATIONS:
See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-09

comments: SUMMARY:
This paper studies a relaxed problem of Pareto set identification, in which the learner is required to identify a subset of the optimal arms. A single sampling strategy APE is proposed and then combined with various stopping rules to realize different relaxations of the original problem. In theory, this paper derives the sample complexity of these combinations. The proposed method is also validated empirically on a real-world scenario of Covid-19 vaccination.

STRENGTHS:
1. Relaxed Pareto set identification is an important problem for multi-objective MAB. The proposed new framework may inspire new researches in this field, if properly justified.

2. The proposed sampling strategy sounds novel, and the analysis is technically sound.

3. The experiment on selecting vaccination strategies against Covid-19 is interesting.

WEAKNESSES:
1. One of the main contributions of this paper is to propose a new problem of $\epsilon$-PSI-$k$ and provide an analysis framework for this problem. However, I think the motivation of $\epsilon$-PSI-$k$ should be explained in more detail. I am not sure if we need this new formulation as $(\epsilon_1,\epsilon_2)$-PSI has already dealt with the sparsity issue. 

2. Some detailed explanations of the intuition of the proposed sampling strategy (and why it outperforms previous strategies in principle) may help better understand the novelty in the algorithm design.

QUESTIONS:
See the weaknesses.

LIMITATIONS:
Limitations have been properly discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10