SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cyrille Kone
cyrille.kone@inria.fr
Emilie Kaufmann<sup>1</sup>
emilie.kaufmann@univ-lille.fr
Laura Richert2
laura.richert@u-bordeaux.fr
<sup>1</sup> Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9198-CRISTAL, F-59000 Lille, France
<sup>2</sup> Univ. Bordeaux, Inserm, Inria, BPH, U1219, Sistm, F-33000 Bordeaux, France
In this paper we revisit the fixed-confidence identification of the Pareto optimal set in a multi-objective multi-armed bandit model. As the sample complexity to identify the exact Pareto set can be very large, a relaxation allowing to output some additional near-optimal arms has been studied. In this work we also tackle alternative relaxations that allow instead to identify a relevant subset of the Pareto set. Notably, we propose a single sampling strategy, called Adaptive Pareto Exploration, that can be used in conjunction with different stopping rules to take into account different relaxations of the Pareto Set Identification problem. We analyze the sample complexity of these different combinations, quantifying in particular the reduction in sample complexity that occurs when one seeks to identify at most  $k$  Pareto optimal arms. We showcase the good practical performance of Adaptive Pareto Exploration on a real-world scenario, in which we adaptively explore several vaccination strategies against Covid-19 in order to find the optimal ones when multiple immunogenicity criteria are taken into account.
In a multi-armed bandit model, an agent sequentially collects samples from several unknown distributions, called arms, in order to learn about these distributions (pure exploration), possibly under the constraint to maximize the samples collected, viewed as rewards (regret minimization). These objectives have been extensively studied for different types of univariate arms distributions [21]. In this paper, we consider the less common setting in which arms are multi-variate distributions. We are interested in the Pareto Set Identification (PSI) problem. In this pure exploration problem, the agent seeks to identify the arms that are (Pareto) optimal, i.e. such that their expected values for all objectives are not uniformly worse than those of another arm.
We formalize this as a fixed-confidence identification problem: in each round  $t$  the agent selects an arm  $A_{t}$  using an adaptive sampling rule and observes a sample  $\mathbf{X}_{t} \in \mathbb{R}^{D}$  from the associated distribution. It further uses an adaptive stopping rule  $\tau$  to decide when to stop sampling and output a set of arms  $\widehat{S}_{\tau}$  which is her guess for (an approximation of) the true Pareto set  $S^{\star}$ . Given a risk parameter  $\delta \in (0,1)$ , this guess should be correct with high probability, e.g. satisfy  $\mathbb{P}(\widehat{S}_{\tau} = S^{\star}) \geq 1 - \delta$  for exact Pareto set identification, while requiring a small sample complexity  $\tau$ . This generalizes the well-studied fixed-confidence Best Arm Identification (BAI) problem [8, 15, 10] to multiple objectives.
Our motivation to study multi-objective adaptive identification stems from the design of adaptive early-phase clinical trials. In phase I/II trials, the effects of a molecule in humans are explored, and several biological endpoints may be assessed at the same time as indicative markers of efficacy. In particular, in the context of vaccine development, early-phase trials usually assess multiple immunogenicity endpoints (i.e. various markers of the effects of the vaccine on the immune system, such as different facets of antibody responses or other immune parameters). In the absence of a known correlate of protection during early clinical development, these endpoints may not have a clear a priori hierarchy, may not all be correlated, which makes an examination of the Pareto set of different vaccinal strategies particularly relevant. In addition, given the availability of various vaccine platforms (such as mRNA vaccines, viral-vector vaccines, protein vaccines), as exemplified by Covid-19 vaccines, there may be a need to adaptively screen the various resulting vaccine strategies to select the most promising ones. Apart from clinical trials, active Pareto Set Identification can be meaningful in many real-word contexts, and we refer the reader to the various examples given by [28], such as in hardware or software design. Other applications include A/B/n testing for marketing or online recommender systems in which it is common to jointly optimize multiple (possibly conflicting) objectives such as user behavioral metrics (e.g. clicks, streams, dwell time, etc), supplier exposure objectives (e.g. diversity) and platform centric objectives (e.g. promotions) [22].
For many applications, the sample complexity of exact PSI can be prohibitive, either when there are many close to optimal arms or when the Pareto set is very large, and different relaxations have been considered in the literature [4, 27]. Going back to our motivation, in an adaptive trial that aims at pre-selecting a certain number of treatments or vaccine strategies for further investigations in clinical trials, practical constraints (the cost and feasibility of the trials) impose a constraint on the maximal number of interesting arms that can be identified. This motivates the introduction of a new setting where the agent is asked to identify at most  $k$  Pareto optimal arms. Interestingly the sampling rule that we propose for this setting can be used to solve (some generalizations of) other relaxations considered in the literature.
Related work The work most closely related to ours is that of Auer et al. [4], who propose a relaxation, which we refer to as  $\varepsilon_{1}$ -PSI: their algorithm returns a set  $\widehat{S}$  that contains w.h.p. all the Pareto optimal arms and possibly some sub-optimal arms, which when increased by  $\varepsilon_{1}$  coordinate-wise become Pareto optimal. For arms that have sub-Gaussian marginals, they provide an instance-dependent sample complexity bound scaling with some notion of sub-optimality gap for each arm. The work of Zaluaga et al. [28, 27] studies a structured variant of fixed-confidence PSI in which the means are regular functions of arms' descriptors. They use Gaussian process modeling and obtain worse-case sample complexity bounds. In particular [27] considers the identification of an  $\varepsilon$ -cover of the Pareto set, which is a representative subset of the  $(\varepsilon)$ -Pareto set that will be related to our  $(\varepsilon_{1}, \varepsilon_{2})$ -PSI criterion. The algorithms of [4] and those of [28, 27] in the unstructured setting have the same flavor: they sample uniformly from a set of active arms and remove arms that have been found sub-optimal (or not representative). Auer et al. [4] further adds an acceptance mechanism to stop sampling some of the arms that have been found (nearly-)optimal and are guaranteed not to dominate an arm of the active set. In this paper, we propose instead a more adaptive exploration strategy, which departs from such accept/reject mechanisms and is suited for different types of relaxation, including our novel  $k$ -relaxation.
Adaptive Pareto Exploration (APE) leverages confidence intervals on the differences of arms' coordinates in order to identify a single arm to explore, in the spirit of the LUCB [15] or UGapEc [9] algorithms for Top- $m$  identification in (one-dimensional) bandit models. These algorithms have been found out to be preferable in practice to their competitors based on uniform sampling and eliminations [19], an observation that will carry over to APE. Besides the multi-dimensional observations, we emphasize that a major challenge of the PSI problem with respect to e.g. Top  $m$  identification is that the number of arms to identify is not known in advance. Moreover, when relaxations are considered, there are multiple correct answers. In the one-dimensional settings, finding optimal algorithms in the presence of multiple correct answers is notoriously hard as discussed by the authors of [5], and their lower-bound based approach becomes impractical in our multi-dimensional setting. Finally, we remark that the  $k$ -relaxation can be viewed as an extension of the problem of identifying any  $k$ -sized subset out of the best  $m$  arms in a standard bandit [25].
<sup>1</sup>PAL relies on confidence intervals that follow from Gaussian process regression, but can also be instantiated with simpler un-structured confidence intervals as those used in our work and in Auer's
Beyond Pareto set identification, other interesting multi-objective bandit identification problems have been studied in the literature. For example [6] propose an algorithm to identify some particular arms in the Pareto set through a scalarization technique [23]. The idea is to turn the multi-objective pure-exploration problem into a single-objective one (unique optimal arm) by using a real-valued preference function which is only maximized by Pareto optimal arms (see e.g [23] for some examples of these functions). In practice, a family of those functions can be used to identify many arms of the Pareto set but it is not always possible to identify the entire Pareto set using this technique (see e.g [7] for weighted sum with a family of weights vectors). In a different direction, the authors of [16] introduce the feasible arm identification problem, in which the goal is to identify the set of arms whose mean vectors belong to a known polyhedron  $P \subset \mathbb{R}^D$ . In a follow up work [17], they propose a fixed-confidence algorithm for finding feasible arms that further maximize a given weighted sum of the objectives. In clinical trials, this could be used to find treatments maximizing efficacy (or a weighted sum of different efficacy indicators), under the constraint that the toxicity remains below a threshold. However, in the presence of multiple indicators of biological efficacy, choosing the weights may be difficult, and an examination of the Pareto set could be more suitable. Finally, some papers consider extensions of the Pareto optimality condition. The authors of [1] tackle the identification of the set of non-dominated arms of any partial order defined by an  $\mathbb{R}^D$  polyhedral ordering cone (the usual Pareto dominance corresponds to using the cone defined by the positive orthant  $\mathbb{R}_+^D$ ), and they provide worst-case sample complexity in the PAC setting. The work of [3] studies the identification of the set of non-dominated elements in a partially ordered set under the dueling bandit setting, in which the observations consist in pairwise comparison between arms.
Outline and contributions First, we formalize in Section 2 different relaxations of the PSI problem:  $\varepsilon_{1}$ -PSI, as introduced by [4],  $\varepsilon_{1}, \varepsilon_{2}$ -PSI, of which a particular case was studied by [27] and  $\varepsilon_{1}$ -PSI- $k$ , a novel relaxation that takes as input an upper bound  $k$  on the maximum number of  $\varepsilon_{1}$ -optimal arms that can be returned. Then, we introduce in Section 3 Adaptive Pareto Exploration, a simple, adaptive sampling rule which can simultaneously tackle all three relaxations, when coupled with an appropriate stopping rule that we define for each of them. In Section 4, we prove high-probability upper bounds on the sample complexity of APE under different stopping rules. For  $\varepsilon_{1}$ -PSI, our bound slightly improves upon the state-of-the-art. Our strongest result is the bound for  $\varepsilon_{1}$ -PSI- $k$ , which leads to a new notion of sub-optimality gap, quantifying the reduction in sample complexity that is obtained. Then, Section 5 presents the result of a numerical study on synthetic datasets, one of them being inspired by a Covid-19 vaccine clinical trial. It showcases the good empirical performance of APE compared to existing algorithms, and illustrates the impact of the different relaxations.
2 Problem Setting
In this section, we introduce the Pareto Set Identification (PSI) problem and its relaxations. Fix  $K, D \in \mathbb{N}^{\star}$ . Let  $\nu_{1}, \ldots, \nu_{K}$  be distributions over  $\mathbb{R}^{D}$  with means  $\pmb{\mu}_{1}, \dots, \pmb{\mu}_{K} \in \mathbb{R}^{D}$ . Let  $\mathbb{A} := [K] := \{1, \dots, K\}$  denote the set of arms. Let  $\nu := (\nu_{1}, \dots, \nu_{K})$  and  $\mathcal{X} := (\pmb{\mu}_{1}, \dots, \pmb{\mu}_{K})$ . We use boldfaced symbols for  $\mathbb{R}^{D}$  elements. Letting  $\mathbf{X} \in \mathbb{R}^{D}$ ,  $u \in \mathbb{R}$ , for any  $d \in \{1, \dots, D\}$ ,  $X^{d}$  denotes the  $d$ -th coordinate of  $\mathbf{X}$  and  $\mathbf{X} + u := (X^{1} + u, \dots, X^{D} + u)$ . In the sequel, we will assume that  $\nu_{1}, \dots, \nu_{K}$  have 1-subgaussian marginals
Definition 1. Given two arms  $i, j \in \mathbb{A}$ ,  $i$  is weakly (Pareto) dominated by  $j$  (denoted by  $\pmb{\mu}_i \preceq \pmb{\mu}_j$ ) if for any  $d \in \{1, \dots, D\}$ ,  $\mu_i^d \leq \mu_j^d$ . The arm  $i$  is (Pareto) dominated by  $j$  ( $\pmb{\mu}_i \preceq \pmb{\mu}_j$  or  $i \preceq j$ ) if  $i$  is weakly dominated by  $j$  and there exists  $d \in \{1, \dots, D\}$  such that  $\mu_i^d < \mu_j^d$ . The arm  $i$  is strictly (Pareto) dominated by  $j$  ( $\pmb{\mu}_i \prec \pmb{\mu}_j$  or  $i \prec j$ ) if for any  $d \in \{1, \dots, D\}$ ,  $\mu_i^d < \mu_j^d$ .
For  $\varepsilon \in \mathbb{R}_{+}^{D}$ , the  $\varepsilon$ -Pareto set  $S_{\varepsilon}^{\star}(\mathcal{X})$  is the set of  $\varepsilon$ -Pareto optimal arms, that is:
$$
\mathcal {S} _ {\varepsilon} ^ {\star} (\mathcal {X}) := \{i \in \mathbb {A} \mathrm {s . t} \not \exists j \in \mathbb {A}: \boldsymbol {\mu} _ {i} + \varepsilon \prec \boldsymbol {\mu} _ {j} \}.
$$
In particular,  $S_0^\star(\mathcal{X})$  is called the Pareto set and we will simply write  $S^\star(\mathcal{X})$  to denote  $S_0^\star(\mathcal{X})$ . When it is clear from the context, we write  $S^\star$  (or  $S_\varepsilon^\star$ ) to denote  $S^\star(\mathcal{X})$  (or  $S_\varepsilon^\star(\mathcal{X})$ ). By abuse of notation we write  $S_\varepsilon^\star$  when  $\varepsilon \in \mathbb{R}^+$  to denote  $S_\varepsilon^\star$ , with  $\varepsilon := (\varepsilon, \ldots, \varepsilon)$ .
In each round  $t = 1,2,\ldots$ , the agent chooses an arm  $A_{t}$  and observes an independent draw  $\mathbf{X}_t\sim \nu_{A_t}$  with  $\mathbb{E}(\mathbf{X}_{A_t}) = \pmb{\mu}_{A_t}$ . We denote by  $\mathbb{P}_{\nu}$  the law of the stochastic process  $(\mathbf{X}_t)_{t\geq 1}$  and by  $\mathbb{E}_{\nu}$ , the
2A random variable  $X$  is  $\sigma$ -subgaussian if for any  $\lambda \in \mathbb{R}$ ,  $\mathbb{E}(\exp(\lambda(X - \mathbb{E}(X))) \leq \exp(\frac{\lambda^2 \sigma^2}{2})$ .
expectation under  $\mathbb{P}_{\nu}$ . Let  $\mathcal{F}_t \coloneqq \sigma(A_1, \mathbf{X}_1, \ldots, A_t, \mathbf{X}_t)$  the  $\sigma$ -algebra representing the history of the process. An algorithm for PSI consists in: i) a sampling rule which determines which arm to sample at time  $t$  based on history up to time  $t - 1$ , ii) a stopping rule  $\tau$  which is a stopping time w.r.t the filtration  $(\mathcal{F}_t)_{t \geq 1}$  and iii) a recommendation rule which is a  $\mathcal{F}_{\tau}$ -measurable random set  $\hat{S}_{\tau}$  representing the guess of the learner. The goal of the learner is to make a correct guess with high probability, using as few samples  $\tau$  as possible. Before formalizing this, we introduce the different notion of correctness considered in this work, depending on parameters  $\varepsilon_1 \geq, \varepsilon_2 \geq 0$  and  $k \in [K]$ . Our first criterion is the one considered by [4].
Definition 2.  $\widehat{S} \subset \mathbb{A}$  is correct for  $\varepsilon_{1}$ -PSI if  $S^{\star} \subset \widehat{S} \subset S_{\varepsilon_{1}}^{\star}$ .
To introduce our second criterion, we need the following definition.
Definition 3. Let  $\varepsilon_1, \varepsilon_2 \geq 0$ . A subset  $S \subset \mathbb{A}$  is an  $(\varepsilon_1, \varepsilon_2)$ -cover of the Pareto set if:  $S \subset S_{\varepsilon_1}^\star$  and for any  $i \notin S$  either  $i \notin S^\star$  or  $\exists j \in S$  such that  $\pmb{\mu}_i \prec \pmb{\mu}_j + \varepsilon_2$ .
The  $\varepsilon$ -accurate set of [27] is a particular case of  $(\varepsilon_1, \varepsilon_2)$ -cover for which  $\varepsilon_1 = \varepsilon_2 = \varepsilon$ . Allowing  $\varepsilon_1 \neq \varepsilon_2$  generalizes the notion of  $\varepsilon$ -correct set and can be useful, e.g., in scenarios when we want to identify the exact Pareto set (setting  $\varepsilon_1 = 0$ ) but allow some optimal arms to be discarded if they are too close (parameterized by  $\varepsilon_2$ ) to another optimal arm already returned. We note however that the sparse cover of [4] is an  $(\varepsilon, \varepsilon)$ -cover with and additional condition that the arms in the returned set should not be too close to each other. Identifying a sparse cover from samples requires in particular to identify  $S_{\varepsilon_1}^{\star}$  hence it can not be seen as a relaxation of  $\varepsilon_1$ -PSI.
Definition 4.  $\widehat{S} \subset \mathbb{A}$  is correct for  $(\varepsilon_1, \varepsilon_2)$ -PSI if it is an  $(\varepsilon_1, \varepsilon_2)$ -cover of the Pareto set.
Definition 5.  $\widehat{S} \subset \mathbb{A}$  is correct for  $\varepsilon_1$ -PSI- $k$  if either i)  $|\widehat{S}| = k$  and  $\widehat{S} \subset S_{\varepsilon_1}^\star$  or ii)  $|\widehat{S}| < k$  and  $S^\star \subset \widehat{S} \subset S_{\varepsilon_1}^\star$  holds.
Given a specified objective  $(\varepsilon_{1}$ -PSI,  $(\varepsilon_{1},\varepsilon_{2})$ -PSI or  $\varepsilon_{1}$ -PSI- $k$ ), and a target risk parameter  $\delta \in (0,1)$ , the goal of the agent is to build a  $\delta$ -correct algorithm, that is to guarantee that with probability larger than  $1 - \delta$ , her guess  $\widehat{S}_{\tau}$  is correct for the given objective, while minimizing the number of samples  $\tau$  needed to make the guess, called the sample complexity.
We now introduce two important quantities to characterize the (Pareto) optimality or sub-optimality of the arms. For any two arms  $i,j$ , we let
$$
\mathrm {m} (i, j) := \min  _ {1 \leq d \leq D} \left(\mu_ {j} ^ {d} - \mu_ {i} ^ {d}\right), \text {a n d} \mathrm {M} (i, j) := \max  _ {1 \leq d \leq D} \left(\mu_ {i} ^ {d} - \mu_ {j} ^ {d}\right),
$$
which have the following interpretation. If  $i \preceq j$ ,  $\mathrm{m}(i,j)$  is the minimal quantity  $\alpha \geq 0$  that should be added component-wise to  $\pmb{\mu}_i$  so that  $\pmb{\mu}_i + \pmb{\alpha} \neq \pmb{\mu}_j$ ,  $\pmb{\alpha} := (\alpha, \dots, \alpha)$ . Moreover,  $\mathrm{m}(i,j) > 0$  if and only if  $i \prec j$ . Then, for any arms  $i,j$ , if  $i \not\prec j$ ,  $\mathrm{M}(i,j)$  is the minimum quantity  $\alpha'$  such  $\pmb{\mu}_i \leq \pmb{\mu}_j + \pmb{\alpha}', \pmb{\alpha}' := (\alpha', \dots, \alpha')$ . We remark that  $\mathrm{M}(i,j) < 0$  if and only if  $i \prec j$ . Our algorithms, presented in the next section, rely on confidence intervals on these quantities.
3 Adaptive Pareto Exploration
We describe in this section our sampling rule, Adaptive Pareto Exploration, and present three stopping and recommendation rules to which it can be combined to solve each of the proposed relaxation. Let  $T_{k}(t) \coloneqq \sum_{s=1}^{t-1} \mathbb{1}(A_{s} = k)$  be the number of times arm  $k$  has been pulled up to round  $t$  and  $\widehat{\pmb{\mu}}_{k}(t) \coloneqq T_{k}(t)^{-1} \sum_{s=1}^{T_{k}(t)} \mathbf{X}_{k,s}$  the empirical mean of this arm at time  $t$ , where  $\mathbf{X}_{k,s}$  denotes the  $s$ -th observation drawn i.i.d from  $\nu_{k}$ . For any arms  $i,j \in \mathbb{A}$ , we let
$$
\mathrm {m} (i, j, t) := \min  _ {d} (\widehat {\mu} _ {j} ^ {d} (t) - \widehat {\mu} _ {i} ^ {d} (t)) \quad \text {a n d} \quad \mathrm {M} (i, j, t) := \max  _ {d} (\widehat {\mu} _ {i} ^ {d} (t) - \mu_ {j} ^ {d} (t)).
$$
The empirical Pareto set is defined as
$$
\begin{array}{l} S (t) \quad := \quad \{i \in \mathbb {A}: \not \exists j \in \mathbb {A}: \widehat {\boldsymbol {\mu}} _ {i} (t) \prec \widehat {\boldsymbol {\mu}} _ {j} (t) \}, \\ = \left\{i \in \mathbb {A}: \forall j \in \mathbb {A} \backslash \{i \}, \mathrm {M} (i, j, t) > 0 \right\}. \\ \end{array}
$$
3.1 Generic algorithm(s)
Adaptive Pareto Exploration relies on a lower/upper confidence bound approach, similar to single-objective BAI algorithms like UGapEc [9], LUCB [15] and LUCB++ [26]. These three algorithms identify in each round two contentious arms:  $b_{t}$ : a current guess for the optimal arm (defined as the empirical best arm or the arm with the smallest upper bound on its sub-optimality gap),  $c_{t}$ : a contender of this arm; the arm which is the most likely to outperform  $b_{t}$  (in all three algorithms, it is the arm with the largest upper confidence bound in  $[K]\backslash \{b_t\}$ ). Then, either both arms are pulled (LUCB, LUCB++) or the least explored among  $b_{t}$  and  $c_{t}$  is pulled (UGapEc). The originality of our sampling rule lies in how to appropriately define  $b_{t}$  and  $c_{t}$  for the multi-objective setting. To define those, we suppose that there exists confidence intervals  $[L_{i,j}^{d}(t,\delta),U_{i,j}^{d}(t,\delta)]$  on the difference of expected values for each pair of arms  $(i,j)$  and each objective  $d\in D$ , such that introducing
$$
\mathcal {E} _ {t} := \bigcap_ {i = 1} ^ {K} \bigcap_ {j \neq i} \bigcap_ {d = 1} ^ {D} \left\{L _ {i, j} ^ {d} (t, \delta) \leq \mu_ {i} ^ {d} - \mu_ {j} ^ {d} \leq U _ {i, j} ^ {d} (t, \delta) \right\} \text {a n d} \mathcal {E} = \bigcap_ {t = 1} ^ {\infty} \mathcal {E} _ {t}, \tag {1}
$$
we have  $\mathbb{P}(\mathcal{E})\geq 1 - \delta$  . Concrete choices of these confidence intervals will be discussed in Section 3.2. To ease the notation, we drop the dependency in  $\delta$  in the confidence intervals and further define
$$
\mathrm {M} ^ {-} (i, j, t) := \max  _ {d} L _ {i, j} ^ {d} (t) \quad \text {a n d} \quad \mathrm {M} ^ {+} (i, j, t) := \max  _ {d} U _ {i, j} ^ {d} (t) \tag {2}
$$
$$
\mathrm {m} ^ {-} (i, j, t) := - \mathrm {M} ^ {+} (i, j, t) \quad \text {a n d} \quad \mathrm {m} ^ {+} (i, j, t) := - \mathrm {M} ^ {-} (i, j, t). \tag {3}
$$
Lemma 1. For any round  $t \geq 1$ , if  $\mathcal{E}_t$  holds, then for any  $i,j \in \mathbb{A}$ ,  $\mathrm{M}^{-}(i,j,t) \leq \mathrm{M}(i,j) \leq \mathrm{M}^{+}(i,j,t)$  and  $\mathrm{m}^{-}(i,j,t) \leq \mathrm{m}(i,j) \leq \mathrm{m}^{+}(i,j,t)$ .
Noting that  $S_{\varepsilon_1}^{\star} = \{i\in \mathbb{A}:\forall j\neq i,\mathrm{M}(i,j) + \varepsilon_1 > 0\}$ , we define the following set of arms that are likely to be  $\varepsilon_{1}$ -Pareto optimal:
$$
\mathrm {O P T} ^ {\varepsilon_ {1}} (t) := \{i \in \mathbb {A}: \forall j \in \mathbb {A} \backslash \{i \}, \mathrm {M} ^ {-} (i, j, t) + \varepsilon_ {1} > 0 \}.
$$
Sampling rule In round  $t$ , Adaptive Pareto Exploration samples  $a_{t}$ , the least pulled arm among two candidate arms  $b_{t}$  and  $c_{t}$  given by
$$
b_{t}:= \operatorname *{argmax}_{i\in \mathbb{A}\setminus \mathrm{OPT}^{\varepsilon_{1}}(t)}\min_{j\neq i}\mathrm{M}^{+}(i,j,t),
$$
$$
c_{t}:= \operatorname *{argmin}_{j\neq b_{t}}\mathrm{M}^{-}(b_{t},j,t)
$$