SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The main novelty of Lemma 1 is that the right-hand side scales with  $\lambda^{1/2}$ , which might be chosen smaller than 1. This improvement is due to the fact that multitask posterior variances are smaller than  $(N + b) / (N + bN) \leq 1$ . The right-hand side also depends on the multitask information gain  $\gamma_T^{\mathrm{mt}}(b)$ , which is nontrivial to compute or upper bound. In the next proposition, we provide practical upper bounds of  $\gamma_T^{\mathrm{mt}}(b)$ , in terms of the kernel parameter  $b$  and the single-task information gain  $\gamma_T^{\mathrm{st}}$ .
Proposition 1. Let  $\lambda \leq 1$ ,  $N \geq 2$ , and  $T_{i} \geq 1$  for all  $i \in [N]$ . Then, for any  $b \geq 0$ , we have
$$
\gamma_ {T} ^ {\mathrm {m t}} (b) \leq N \gamma_ {T} ^ {\mathrm {s t}} - \frac {N b}{8 (1 + b)}, \quad a n d \quad \gamma_ {T} ^ {\mathrm {m t}} (b) \leq \gamma_ {T} ^ {\mathrm {s t}} + \frac {T}{2 \lambda (1 + b)}.
$$
We can now combine Theorem 1, Lemma 1, and Proposition 1 to obtain our main result: a bound on the multitask regret of MT-UCB run with the confidence width  $\beta_t^{\mathrm{new}}$  from Theorem 1 and a specific  $\lambda$ .
Theorem 2. Assume that  $B \geq 1$ , and that MT-UCB is run with  $\beta_{t} = \beta_{t}^{\mathrm{new}}$  from Theorem 1, and  $\lambda = (N + b) / (N + bN)$ . Let  $b = N / \epsilon^{2}$  if  $T \leq N$ ,  $b = 1 / \epsilon^{2}$  if  $T \geq N$  and  $\epsilon \leq N^{-1/4}T^{-1/2}$ , and  $b = 0$  otherwise. Let  $R^{\mathrm{st}}(T) = B\sqrt{T\gamma_T^{\mathrm{st}}} + \sqrt{T\gamma_T^{\mathrm{st}}(\gamma_T^{\mathrm{st}} + \ln(1/\delta))}$  be the single task regret bound achieved by IGP-UCB (up to constant factors). Then, there exists a universal constant  $C$  such that with probability  $1 - 2\delta$  we have (up to log  $N$  factors)
$$
\begin{array}{l} R ^ {\mathrm {m t}} (T) \leq C \min  \left\{\sqrt {N} R ^ {\mathrm {s t}} (T), R ^ {\mathrm {s t}} (T) + \epsilon B T ^ {3 / 2} \left(\sqrt {\gamma_ {T} ^ {\mathrm {s t}} + \ln (1 / \delta)} + \epsilon \sqrt {T}\right), \right. \\ \left. R ^ {\mathrm {s t}} (T) + \epsilon B T \sqrt {N} \left(\sqrt {\gamma_ {T} ^ {\mathrm {s t}} + \ln (1 / \delta)} + \epsilon \sqrt {N T}\right) \right\}. \\ \end{array}
$$
The regret bound of Theorem 2 is the minimum between three bounds, obtained exploiting the three different regimes of the confidence width  $\beta_{t}^{\mathrm{new}}$  derived in Theorem 1 (see Figure 2). The first bound is obtained using  $\beta_{t}^{\mathrm{new}} \leq \beta_{t}^{\mathrm{small-b}}$ , and shows that our approach cannot be worse than independent learning. Indeed, it can be checked that, when facing  $N$  tasks, the regret of running  $N$  independent instances of IGP-UCB can be bounded by  $\sqrt{N}$  times the single-task regret bound of IGP-UCB, that we denoted by  $R^{\mathrm{st}}(T)$ . Note however that our analysis slightly differs, insofar as we leverage the multitask information gain, while the independent analysis uses Jensen's inequality to aggregate the individual bounds, see Appendix B for details. Note finally that we are able to recover this bound as  $\beta_{t}^{\mathrm{small-b}}$  is tight at  $b = 0$ , unlike  $\beta_{t}^{\mathrm{naive}}$ . The second bound uses  $\beta_{t}^{\mathrm{new}} \leq \beta_{t}^{\mathrm{large-b}}$  and consists of two terms: the single task regret bound and an additional term that scales with the task deviation  $\epsilon$ . When the latter is small, i.e., when tasks are similar, the dominant term is  $R^{\mathrm{st}}(T)$ , as if only one task were solved. The third bound is similar, but obtained using  $\beta_{t}^{\mathrm{new}} \leq \beta_{t}^{\mathrm{naive}}$  and is useful when  $T \geq N$ . In
contrast with the independent bound, which does not exploit the task structure, the last two bounds show that multitask learning is always beneficial when the horizon  $T$  (and thus the additional  $\epsilon$ -related term) is small. As expected, this is particularly true when the number of tasks  $N$  is large: while the independent bound increases, the second bound does not depend on  $N$ . On the other hand, one can note that the condition on  $\epsilon$  to improve over independent becomes more constraining as the horizon  $T$  increases. This suggests that the benefit of multitask may vanish with the number of available points per task, an observation which is well-known by practitioners, see e.g. [21]. As far as we know, this work is the first one to provide theoretical evidence of such a phenomenon in online MT learning.
We conclude this section by comparing Theorem 2 to existing results. As already mentioned in the above discussion, independent IGP-UCB is a particular case of MT-UCB, such that we cannot be worse than the independent approach. We incidentally recover its regret bound as the first bound in the minimum of Theorem 2. Regarding GoB.Lin, since it is also a specific instance of MT-UCB (for  $\beta_{t} = \beta_{t}^{\mathrm{naive}}$  and  $\lambda = 1$ ), Lemma 1 allows to recover its regret bound [6, Theorem 1].
Corollary 1 (Regret of GoB.Lin [6]). For any  $b$ , the multitask regret of GoB.Lin using parameter  $b$  satisfies with probability  $1 - \delta$
$$
R ^ {\mathrm {m t}} (T) \leq 4 \beta_ {T} ^ {\mathrm {n a i v e}} (b) \sqrt {T \gamma_ {T} ^ {\mathrm {m t}} (b)} \leq 6 \left(B \sqrt {N \left(1 + b \epsilon^ {2}\right)} + \sqrt {\gamma_ {T} ^ {\mathrm {m t}} (b) + \ln \left(1 / \delta\right)}\right) \sqrt {T \gamma_ {T} ^ {\mathrm {m t}} (b)}. \tag {8}
$$
If tasks are similar, i.e., when  $\epsilon \ll 1$ , bound (8) suggests to choose  $b > 0$ ; this does not impact too much the first term, but makes  $\gamma_T^{\mathrm{mt}}(b)$  smaller. However, we recall that the above bound instantiated with  $b = 0$  does not recover the independent bound. It is instead  $\sqrt{N}$  bigger, since  $\beta_t^{\mathrm{naive}}$  is not tight at  $b = 0$ . Hence, the Gob. Lin analysis is not sufficient to show that multitask learning improves over independent learning. Our refined analysis, which uses instead  $\beta_t^{\mathrm{new}}$ , closes this gap.
3.2 Adapting to unknown task similarity
In this section, we consider the case where parameter  $\epsilon$  (i.e., a bound on the task deviation from the average, see (5)) is a-priori unknown. Despite this challenge, we show that the regret bound of Theorem 2 can be approximately attained using an adaptive procedure, AdaMT-UCB (Algorithm 3), relegated to Appendix B.4 due to space limitations. The proposed approach is inspired by the model selection scheme of [22, Section 7] with a few important modifications that we will outline at the end of this section. AdaMT-UCB considers a plausible set of parameters  $\mathcal{E} = \{e_1,\dots ,e_{|\mathcal{E}|}\} \subset (0,2]$  and, for each  $e\in \mathcal{E}$ , initializes an instance of the MT-UCB algorithm with parameters set according to Theorem 2 assuming  $\epsilon = e$ . We denote such an instance as MT-UCB(e). Moreover, we use the notation  $\mathrm{ucb}_t^e$  to denote the upper confidence bounds constructed by MT-UCB(e). We assume the existence of some  $e\in \mathcal{E}$  such that  $e\geq \epsilon$ , so that at least one of the learners is well-specified (i.e., its confidence bounds contain  $f^{\mathrm{mt}}$  with high probability). Our goal is to incur a regret which grows as the regret of the learner with the smallest  $e$  such that  $e\geq \epsilon$ , since the smaller the  $e$  the smallest the regret bound (see Theorem 2), as long as  $e$  is a valid upper bound for  $\epsilon$ . Let us identify with  $e^{\star}$  such learner.
At each round  $t$ , AdaMT-UCB uses learner  $e_t = \min \mathcal{E}$ , and plays the action  $x_t$  suggested by it, i.e., the maximizer of  $\mathrm{ucb}_t^{e_t}(i_t, \cdot)$ . Then, all MT-UCB( $e$ ) learners are updated based on the observed reward. In the meantime, a misspecification test is carried out to check whether learner  $e_t$  is well-specified. Such a test compares the obtained cumulative reward, a lower confidence estimate on such reward according to the other learners, and the believed regret of learner  $e_t$ . As long as the test does not trigger, the regret of learner  $e_t$  is controlled by the believed one. Instead, if the test triggers, learner  $e_t$  can be considered misspecified with high probability. As a result, it gets removed from  $\mathcal{E}$  and a new epoch starts with the new set  $\mathcal{E}$ . Let  $\overline{R_{\star}^{\mathrm{mt}}}(T)$  denote the regret bound (Theorem 2) of learner  $e^{\star}$  had it been chosen from round 0. We can state the following.
Theorem 3. Assume that there exists  $e \in \mathcal{E}$  such that  $e \geq \epsilon$ , and let  $M$  be the number of learners  $e \in \mathcal{E}$  such that  $e < \epsilon$  (i.e., the number of misspecified learners in  $\mathcal{E}$ ). The regret of AdaMT-UCB satisfies with high probability  $R^{\mathrm{mt}}(T) = \mathcal{O}\big(\sqrt{M + 1} \cdot \overline{R_{\star}^{\mathrm{mt}}(T)}\big)$ .
Clearly, the number  $M$  of misspecified learners is not known in advance but is always less than  $|\mathcal{E}|$ . Note that when  $\epsilon = 0$ , we have  $M = 0$  and we recover the single task regret bound. Moreover, given  $\rho \leq 1$ , we show in Appendix B.4 that one can attain a multiplicative accuracy  $\rho$  over  $\epsilon$ , assuming that  $\epsilon \geq \epsilon_{\min} > 0$ , through an exponential grid with  $M$  being polylogarithmic in  $1 / \rho$  and  $1 / \epsilon_{\min}$ .
Relation with the approach of [22]. Compared to [22, Section 7]—where the goal is to adapt to an unknown features' dimension—the set of learners considered in AdaMT-UCB share the same dimension  $d$ . This allows us to exploit the following two novelties with respect to [22]: (1) all learners are updated from the data gathered from learner  $i_t$  (Line 6 in Algorithm 3), and (2) the lower confidence bounds  $L^e$  in the misspecification test (Line 8) are all computed using action  $x_t$  (i.e., the action recommended by learner  $i_t$ ), as opposed to using the actions recommended by each learner  $e$ . Both these points are only applicable to our setting, leading to a simpler regret analysis.
4 Multitask Active Learning
The goal of the online learning setup of Section 3 is to optimize the tasks sequentially revealed by nature. In some situations (e.g., in [20] or the drug discovery problem considered in Section 5), however, we care about the performance of multiple tasks simultaneously, to eventually learn the best strategy for each one of them. Moreover, we ought to do so with minimal interactions  $T$ , i.e., minimizing the queries of the function  $f^{\mathrm{mt}}$ . We capture this by the following active learning protocol.
Learning protocol and regret. At each round  $t$ , the learner: chooses a strategy  $\{x_{t}^{i}, i \in [N]\}$  for each task, chooses which task  $i_{t} \in [N]$  to query, and observes the noisy feedback  $y_{t} = f^{\mathrm{mt}}(i_{t}, x_{t}) + \xi_{t}$ . The learner's goal is to minimize the active learning regret:
$$
R _ {\mathrm {A L}} ^ {\mathrm {m t}} (T) = \sum_ {t = 1} ^ {T} \frac {1}{N} \sum_ {i = 1} ^ {N} \max _ {x \in \mathcal {X}} f ^ {\mathrm {m t}} (i, x) - \sum_ {t = 1} ^ {T} \frac {1}{N} \sum_ {i = 1} ^ {N} f ^ {\mathrm {m t}} (i, x _ {t} ^ {i}).
$$
Compared to the online learning regret of Equation (6), the learner's performance at each round is here measured by the average reward coming from each task (as opposed to just the task presented by nature). Moreover, compared to online learning, the learner faces the additional challenge of choosing—at each round—from which task information should be gathered. Intuitively, more difficult (or informative) tasks should be queried more often to ensure  $R_{\mathrm{AL}}^{\mathrm{mt}}(T)$  grows sublinearly. To the best of our knowledge, the above protocol and regret notion are novel in the multitask literature.
In Algorithm 2 we present MT-AL, an efficient strategy that ensures sublinear active learning regret. Like in MT-UCB, MT-AL constructs confidence intervals around  $f^{\mathrm{mt}}$  and, at each round, select strategy  $x_{t}^{i} = \arg \max_{x\in \mathcal{X}}\mathrm{ucb}_{t - 1}(i,x)$  for each task  $i\in [N]$ . When it comes to selecting which task to query, MT-AL selects  $i_t\in$  arg max  $i_{i\in [N]}\beta_{t - 1}^{i}\sigma_{t - 1}(i,x_{t}^{i})$ , i.e., the task for which the believed optimizer  $x_{t}^{i}$  is subject to max
Algorithm 2 MT-AL
$$
\begin{array}{l} x _ {t} ^ {i} = \arg \max  _ {x \in \mathcal {X}} \mathbf {u c b} _ {t - 1} (i, x),   \forall i \in [ N ] \\ i _ {t} = \arg \max  _ {i \in [ N ]} \beta_ {t - 1} ^ {i} \sigma_ {t - 1} (i, x _ {t} ^ {i}) \end{array}
$$
Observe:  $y_{t} = f^{\mathrm{mt}}(i_{t},x_{t}^{i_{t}}) + \xi_{t}$
Update  $\mathbf{ucb}_t(\cdot ,\cdot)$  and  $\sigma_t(\cdot ,\cdot)$  based on observations.
imal uncertainty (we use generic task-dependent widths  $\beta_t^i$  for completeness). This rule, also known as uncertainty sampling in the literature [23], intuitively makes sure the learner can control the regrets for the tasks not queried and leads to the following theorem.
Theorem 4. Suppose that for all tasks  $i$ , point  $x$ , and time  $t$ , we have that  $f^{\mathrm{mt}}(i,x) \in [\mu_t(i,x) \pm \beta_t^i \cdot \sigma_t(i,x)]$ . Then, the MT-AL algorithm ensures the active learning regret is bounded by
$$
R _ {\mathrm {A L}} ^ {\mathrm {m t}} \leq 2 \sum_ {t = 1} ^ {T} \beta_ {t} ^ {i _ {t}} \sigma_ {t} \left(i _ {t}, x _ {t} ^ {i _ {t}}\right),
$$
where  $\{i_t\}$  is the sequence of queried tasks and  $\{x_{t}^{i_{t}}\}$  the strategies selected for each of them.
The above bound only relies on MT-AL utilizing valid intervals around  $f^{\mathrm{mt}}$  and thus applies more broadly than our agnostic MT regression, e.g., when such intervals are constructed using a known multitask kernel  $k\big((i,x),(i',x')\big)$ . However, Theorem 4 shows the active learning regret heavily depends on the constructed intervals, similar to online learning. In MT-AL, these are additionally utilized for deciding which task to query at each round. When specialized to our agnostic MT kernel and improved confidence, we obtain the following.
Corollary 2. Let MT-AL utilize the  $MT$  regression estimates of Eq. (3)-(4) with parameters set according to Theorem 2. Moreover, let  $\overline{R^{\mathrm{mt}}}(T)$  be the bound on the online learning regret obtained in Theorem 2. Then, with high probability, we have  $R_{\mathrm{AL}}^{\mathrm{mt}}(T) \leq \overline{R^{\mathrm{mt}}}(T)$ .
(a) Online learning
(b) Active learning
Figure 3: Online and active learning regrets on synthetic and drug discovery MHC-I data, respectively. When utilizing the improved confidence intervals, MT-UCB and MT-AL outperform the other baselines.
Thus, MT-AL ensures the active learning regret is always bounded by its online learning counterpart. Moreover, the same considerations as in Theorem 2 apply also here, regarding the benefit of multitask learning over independent single-task regression for instance.
The goal of our experiments is to evaluate the effectiveness of the studied MT regression, and in particular of the improved confidence intervals obtained in Section 2, both in online learning and active learning setups. We utilize the following synthetic and real-world data<sup>2</sup>.
Synthetic data: We generate tasks of the form  $f_{i} = (1 - \delta) \cdot \bar{f} + \delta \cdot f_{\mathrm{dev}}^{i}, i \in [N]$ , where  $\bar{f}, f_{\mathrm{dev}}^{i}$  are random unit vectors representing a common model and individual deviations, respectively. Moreover, actions consist of  $10^4$  vectors  $x \in \mathbb{R}^d$  from the sphere of radius 10. Observation noise is unit normal.
Drug discovery MHC-I data [27]: The goal is to discover the peptides with maximal binding affinity to each Major Histocompatibility Complex class-I (MHC-I) allele. The dataset from [27] contains the standardized binding affinities  $\mathrm{(IC_{50}}$  values) of different peptide candidates to the MHC-I alleles (tasks). For each allele, the dataset contains  $\sim 1000$  peptides represented as  $x\in \mathbb{R}^{45}$  feature vectors. For our experiments, we utilize the 5 alleles A-{0201, 0202, 0203, 2301, 2402}, since they were shown in [27] to share binding similarity. Note that such a problem falls into our multitask active learning setup, since we would like to retrieve the best peptide for each allele minimizing the number of interactions (i.e., lab experiments). Nevertheless, we also consider its online learning analog where we care about finding the best peptides for each revealed allele.
Online learning. At each round  $t$ , a random task  $i_t \in [N]$  is observed and point  $x_t$  is selected according to the following baselines: (1) Independent, which runs  $N$  independent IGP-UCB [8] algorithms (corresponding to MT-UCB with  $b = 0$ ), (2) Single, which treats all tasks to be the same and runs a unique single-task IGP-UCB (corresponding to MT-UCB with  $b = +\infty$ ), (3) MT-UCB which utilizes an appropriate parameter  $0 < b < \infty$  as well as a bound on the tasks similarity  $\epsilon$  (for synthetic data this can be exactly computed, while for MHC-I data we use  $\epsilon = 0.3$ ) and utilizes the naive (i.e., Gob.Lin) or improved confidence bounds, and (4) AdaMT-UCB which is run with the same  $b$  but uses the set of plausible deviations  $\mathcal{E} = \{ .1, .2, \dots, 1 \}$  instead of knowing the true  $\epsilon$ . For choosing  $b$ , we sweep over possible values and select the best-performing one, keeping it fixed for all the baselines.
Active learning. We follow the multitask active learning setup of Section 4. All baselines utilize confidence intervals from the agnostic MT regression of Section 2, where  $\epsilon$  and  $b$  are chosen as for online learning. Moreover, they all utilize the improved confidence intervals, unless otherwise specified. We compare: (1) Unif. which chooses the task  $i_t$  to be queried uniformly at random (but still selects  $x_t^i \in \arg \max_x \mathrm{ucb}_t^i(i, x)$ ) and employs the naive or the improved confidence intervals, the offline contextual Bayesian optimization baselines (2) MTS [7] and (3) AE-LSVI [18], and (4) MT-AL which utilizes the naive or the improved confidence intervals.
$^{2}$ code available at: https://github.com/sessap/multitask-noregret.
We report the cumulative regret (online and active learning, respectively) of the considered baselines in Figure 3, averaged over 5 runs. For the synthetic data, we report results for  $d = 4$ ,  $N = 5$ ,  $\delta = 0.4$ , but provide a full set of experiments for different parameters in Appendix D. In Appendix D we also report the frequencies of each task being queried in our active learning experiments. In Figure 3 (a), both MT-UCB and AdaMT-UCB lead to superior performance compared to the Independent and Single baselines, demonstrating the benefits of MT regression. In addition, the improved confidence intervals significantly outperform the naive ones. Moreover, we observe AdaMT-UCB achieves comparable (sometimes even better, see Appendix D) performance to MT-UCB. Indeed, instead of using a conservative choice of  $\epsilon$ , the misspecification test (Line 8 of Algorithm 3) of AdaMT-UCB allows to use a smaller  $\epsilon$  and only increase it when there is evidence that the constructed intervals do not contain the true tasks. In active learning (Figure 3 (b)), we observe MT-AL has a significant advantage over the uniform sampling baselines and MTS, while performing comparably to AE-LSVI (both methods are similar as discussed in Appendix C.3). Moreover, its regret is bounded by the online learning regret of MT-UCB, conforming with Theorem 4. Importantly, the improved confidence intervals play a crucial role also here and enable a drastic performance improvement compared to the naive ones.
6 Future Directions
We believe this paper opens up several future research directions. The derived confidence intervals, as well as our analysis of the multitask information gain, heavily exploit the structure of the task Gram matrix  $K_{\mathrm{task}}(b)$ , see Equation (2). However, it remains unclear whether these can be extended to more general kernels. According to the graph perspective of [13],  $K_{\mathrm{task}}(b)$  can be seen as  $K_{\mathrm{task}}(b) = I_N + L(b)$ , where  $L(b) \in \mathbb{R}^{N \times N}$  is the Laplacian matrix of a clique graph with vertices  $[N]$  and edge weight  $b$ . Hence, it would be interesting to extend our results to different graph structures. Furthermore, we believe the proposed multitask confidence intervals hold potential for various related domains., e.g., to assess uncertainty in safety-critical systems [3], or to balance exploration-exploitation in multitask reinforcement learning [26], spam filtering [15], or personalized health [14]. In such applications, the introduced notion of active learning regret can serve as a measure of the overall sample efficiency.
This work was partially supported by ELSA (European Lighthouse on Secure and Safe AI) funded by the European Union under grant agreement No. 101070617. PL and NCB gratefully acknowledge the financial support from the MUR PRIN grant 2022EKNE5K (Learning in Markets and Society), funded by the NextGenerationEU program within the PNRR scheme (M4C2, investment 1.1), the FAIR (Future Artificial Intelligence Research) project, funded by the NextGenerationEU program within the PNRR-PE-AI scheme (M4C2, investment 1.3, line on Artificial Intelligence), and the EU Horizon CL4-2022-HUMAN-02 research and innovation action under grant agreement 101120237, project ELIAS (European Lighthouse of AI for Sustainability).
[1] Abbasi-Yadkori, Y. (2012). Online learning for linearly parametrized control problems. PhD thesis, Edmonton, Alberta, Canada.
[2] Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems (NeurIPS), 24.
[3] Camilleri, R., Wagenmaker, A., Morgenstern, J. H., Jain, L., and Jamieson, K. G. (2022). Active learning with safety constraints. Advances in Neural Information Processing Systems (NeurIPS).
[4] Caruana, R. (1997). Multitask learning. Machine learning, 28:41-75.
[5] Cavallanti, G., Cesa-Bianchi, N., and Gentile, C. (2010). Linear algorithms for online multitask classification. Journal of Machine Learning Research, 11:2901-2934.
[6] Cesa-Bianchi, N., Gentile, C., and Zappella, G. (2013). A gang of bandits. Advances in Neural Information Processing Systems (NeurIPS), 26.
[26] Vithayathil Varghese, N. and Mahmoud, Q. H. (2020). A survey of multi-task deep reinforcement learning. *Electronics*, 9(9):1363.
[27] Widmer, C., Toussaint, N. C., Altun, Y., and Ratsch, G. (2010). Inferring latent task structure for multitask learning by multiple kernel learning. BMC bioinformatics, 11(8):1-8.