SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Omid Sadeghi Maryam Fazel
{omids,mfazel@uw.edu}
We study a generalization of the online binary prediction with expert advice framework where at each round, the learner is allowed to pick  $m \geq 1$  experts from a pool of  $K$  experts and the overall utility is a modular or submodular function of the chosen experts. We focus on the setting in which experts act strategically and aim to maximize their influence on the algorithm's predictions by potentially misreporting their beliefs about the events. Among others, this setting finds applications in forecasting competitions where the learner seeks not only to make predictions by aggregating different forecasters but also to rank them according to their relative performance. Our goal is to design algorithms that satisfy the following two requirements: 1) Incentive-compatible: Incentivize the experts to report their beliefs truthfully, and 2) No-regret: Achieve sublinear regret with respect to the true beliefs of the best-fixed set of  $m$  experts in hindsight. Prior works have studied this framework when  $m = 1$  and provided incentive-compatible no-regret algorithms for the problem. We first show that a simple reduction of our problem to the  $m = 1$  setting is neither efficient nor effective. Then, we provide algorithms that utilize the specific structure of the utility functions to achieve the two desired goals.
Learning from a constant flow of information is one of the most prominent challenges in machine learning. In particular, online learning requires the learner to iteratively make decisions and at the time of making each decision, the outcome associated with it is unknown to the learner. The experts problem is perhaps the most well-known problem in online learning [1, 2, 3, 4]. In this problem, the learner aims to make predictions about a sequence of  $T$  binary events. To do so, the learner has access to the advice of  $K$  experts who each have internal beliefs about the likelihood of each event. At each round  $t \in [T]$ , the learner has to choose one among the advice of  $K$  experts and upon making her choice, the  $t$ -th binary event is realized and a loss bounded between zero and one is revealed. The goal of the learner is to have no regret, i.e., to perform as well as the best-fixed expert in hindsight. In many applications, however, the experts are strategic and wish to be selected by the learner as often as possible. To this end, they may strategically misreport their beliefs about the events. For instance, FiveThirtyEight<sup>1</sup> aggregates different pollsters according to their past performance to make a single prediction for elections and sports matches. To do so, FiveThirtyEight maintains publicly available pollster ratings<sup>2</sup>. A low rating can be harmful to the pollster's credibility and adversely impact their revenue opportunities in the future. Therefore, instead of maximizing their expected performance by reporting their predictions truthfully, the pollsters may decide to take risks and report more extreme beliefs to climb higher on the leaderboard. Therefore, it is important to design algorithms that not only achieve no-regret but also motivate the experts to report their true beliefs (incentive-compatible).
<sup>1</sup>https://fivethirtyeight.com/
$^{2}$ https://projects.fivethirtyeight.com/pollster-ratings/
Otherwise, the quality of the learner's predictions may be harmed.
As we have mentioned in Section 1.1, all the previous works on this topic have focused on the standard experts problem where the goal is to choose a single expert among the  $K$  experts. In the offline setting, this is equivalent to a forecasting competition in which only the single highest-ranked forecaster wins and receives prizes. However, in many applications, a set of top-performing forecasters are awarded perks and benefits. For instance, in the Good Judgement Project, a recent geopolitical forecasting tournament, the top  $2\%$  of forecasters were given the "superforecaster" status and received benefits such as paid conference travel and employment opportunities [5]. Similarly, in the latest edition of Kaggle's annual machine learning competition to predict the match outcomes of the NCAA March Madness college basketball tournament (called "March Machine Learning Mania 2023"), the top 8 forecasters on the leaderboard received monetary prizes.
Variants of the  $m$ -experts problem have been previously studied in [6, 7, 8, 9], however, all of these works focused only on providing no-regret algorithms for the problem and the incentive compatibility considerations were not taken into account. To the best of our knowledge, this is the first work that focuses on the strategic  $m$ -experts problem where the experts may misreport their true beliefs to increase their chances of being chosen by the learner.
For the setting with modular utilities, perhaps the simplest approach to learning well compared to the best-fixed set of  $m$  experts while maintaining incentive compatibility is to run the incentive-compatible WSU algorithm of [10] for the standard experts problem (the setting with  $m = 1$ ) over the set of  $\binom{K}{m}$  meta-experts where each meta-expert corresponds to one of the sets of size  $m$ . This approach has two major drawbacks: 1) There are exponentially many meta-experts and maintaining weights per each meta-expert and running the WSU algorithm is computationally expensive, and 2) The dependence of the regret bound on  $m$  is sub-optimal. Therefore, for our setting, it is preferable to design algorithms that are tailored for the  $m$ -experts problem.
Prior works have studied the experts problem under incentive compatibility considerations for two feedback models: In the full information setting, the learner observes the reported prediction of all experts at each round. In the partial information setting, however, the learner is restricted to choosing a single expert at each round and does not observe the prediction of other experts. [11] considered algorithms that maintain weights over the experts and choose experts according to these weights. They assumed that experts' incentives are only affected by the unnormalized weights of the algorithm over the experts. However, since the probability of an expert being chosen by the learner equals her normalized weight, the aforementioned assumption might not be suitable. Later on, [10] made the assumption that at each round  $t \in [T]$ , incentives are tied to the expert's normalized weight (i.e., the probability of being chosen at round  $t + 1$ ) and studied this problem under both feedback models. They proposed the WSU and WSU-UX algorithms for the full information and partial information settings respectively where both algorithms are incentive-compatible and they obtained  $\mathcal{O}(\sqrt{T\ln K})$  and  $\mathcal{O}(T^{2/3}(K\ln K)^{1/3})$  regret bounds for the algorithms. Then, [12] considered non-myopic strategic experts where the goal of each expert is to maximize a conic combination of the probabilities of being chosen in all subsequent rounds (not just the very next round). They showed that the well-known Follow the Regularized Leader (FTRL) algorithm with the negative entropy regularizer obtains a regret bound of  $\mathcal{O}(\sqrt{T\ln K})$  while being  $\Theta(\frac{1}{\sqrt{T}})$  approximately incentive-compatible, i.e., it is a strictly dominated strategy for any expert to make reports  $\Theta(\frac{1}{\sqrt{T}})$  distant from their true beliefs.
For the non-strategic  $m$ -experts problem with modular utilities, [7] proposed the Component Hedge (CH) algorithm and obtained a regret bound of  $\sqrt{2m\ell^{*}\ln(\frac{K}{m})} + m\ln (\frac{K}{m})$  where  $\ell^{*}$  is the cumulative loss of the best-chosen set in hindsight. They also gave a matching lower bound for this problem. [8] studied the FTPL algorithm with Gaussian noise distribution and provided an  $\mathcal{O}(m\sqrt{T\ln(\frac{K}{m})})$  regret bound for this setting. For the non-strategic  $m$ -experts problem with submodular utility functions, [13] proposed the online distorted greedy algorithm (with dual averaging algorithm as the algorithm  $A_{i}$  for  $i = 1,\dots ,m$ ) whose regret bound is  $\mathcal{O}(\sqrt{mT\ln(\frac{K}{m})})$ . More recently, [9] studied the  $m$ -experts problem under various choices of the utility function (sum-reward, max-reward, pairwise-reward and monotone reward). In particular, for the setting with modular utilities (sum-reward), they proposed
<sup>3</sup>https://www.kaggle.com/competitions/march-machine-learning-mania-2023/
an algorithm that matches the optimal regret bound of the CH algorithm while being computationally more efficient.
In this paper, we focus on a generalization of the experts problem (called "m-experts problem") where at each round, instead of picking a single expert, we are allowed to pick  $m \geq 1$  experts and our utility is either a modular or submodular function of the chosen experts. In particular, at round  $t \in [T]$  and for a set of experts  $S_{t} \subseteq [K]$ , the utility function is defined as  $f_{t}(S_{t}) = \frac{|S_{t}|}{m} - \frac{1}{m}\sum_{i \in S_{t}}\ell_{i,t}$  and  $f_{t}(S_{t}) = 1 - \prod_{i \in S_{t}}\ell_{i,t}$  in the modular and submodular cases respectively where  $\ell_{i,t} \in [0,1]$  is the loss of expert  $i$  at round  $t$ . The goal is to design algorithms that perform as well as the best-fixed set of  $m$  experts in hindsight (no-regret) and incentivize the experts to report their beliefs about the events truthfully (incentive-compatible).
Towards this goal, we build upon the study of the Follow the Perturbed Leader (FTPL) algorithm for the  $m$ -experts problem with modular utility functions by [8] and derive a sufficient condition for the perturbation distribution to guarantee approximate incentive compatibility. Furthermore, we show how this condition is related to the commonly used bounded hazard rate assumption for noise distribution. In particular, we show that while FTPL with Gaussian perturbations is not incentive-compatible, choosing Laplace or hyperbolic noise distribution guarantees approximate incentive compatibility.
Moreover, inspired by Algorithm 1 of [13] for online monotone submodular maximization subject to a matroid constraint, we first introduce a simpler algorithm (called the "online distorted greedy algorithm") for the special case of cardinality constraints. This algorithm utilizes  $m$  incentive-compatible algorithms for the standard experts problem (i.e.,  $m = 1$  setting) and outputs their combined predictions. We provide  $(1 - \frac{c}{e})$ -regret bounds for the algorithm where  $c \in [0,1]$  is the average curvature of the submodular utility functions. Therefore, applying the algorithm to the setting where the utility functions are modular (i.e.,  $c = 0$ ), the approximation ratio is 1. For submodular utility functions, the algorithm achieves the optimal  $1 - \frac{c}{e}$  approximation ratio.
Finally, we validate our theoretical results through experiments on data gathered from a forecasting competition run by FiveThirtyEight in which forecasters make predictions about the match outcomes of the recent 2022-2023 National Football League (NFL).
Notation. The set  $\{1,2,\ldots ,n\}$  is denoted by  $[n]$ . For vectors  $x_{t},y\in \mathbb{R}^{n}$ ,  $x_{i,t}$  and  $y_{i}$  denote their  $i$ -th entry respectively. Similarly, for a matrix  $A\in \mathbb{R}^{n\times n}$ , we use  $A_{i,j}$  to indicate the entry in the matrix's  $i$ -th row and  $j$ -th column. The inner product of two vectors  $x,y\in \mathbb{R}^n$  is denoted by either  $\langle x,y\rangle$  or  $x^{T}y$ . For a set function  $f$ , we use  $f(j|A)$  to denote  $f(A\cup \{j\}) - f(A)$ .
A set function  $f$  defined over the ground set  $V$  is monotone if for all  $A \subseteq B \subseteq V$ ,  $f(A) \leq f(B)$  holds.  $f$  is called submodular if for all  $A \subseteq B \subset V$  and  $j \notin B$ ,  $f(j|A) \geq f(j|B)$  holds. In other words, the marginal gain of adding the item  $j$  decreases as the base set gets larger. This property is known as the diminishing returns property. If equality holds in the above inequality for all  $A$ ,  $B$ , and  $j$ , the function is called modular. As mentioned in Section 1.2, at round  $t \in [T]$  and for a set of experts  $S \subseteq [K]$ , the submodular utility function is defined as  $f_{t}(S_{t}) = 1 - \prod_{i \in S_{t}} \ell_{i,t}$  where  $\ell_{i,t} \in [0,1]$  is the loss of expert  $i$  at round  $t$ . To show that this function is submodular, note that we have:
$$
f (A \cup \{j \}) - f (A) = (1 - \ell_ {j, t}) \prod_ {i \in A} \ell_ {i, t} \geq (1 - \ell_ {j, t}) \prod_ {i \in B} \ell_ {i, t} = f (B \cup \{j \}) - f (B),
$$
where the inequality follows from  $\ell_{i,t}\in [0,1]$  for  $i\in B\setminus A$
For a normalized monotone submodular set function  $f$  (i.e.,  $f(\emptyset) = 0$ ), the curvature  $c_{f}$  is defined as [14]:
$$
c _ {f} = 1 - \min  _ {j \in V} \frac {f (j | V \setminus \{j \})}{f (\{j \})}.
$$
It is easy to see that  $c_{f} \in [0,1]$  always holds.  $c_{f} \leq 1$  is due to monotonicity of  $f$  and  $c_{f} \geq 0$  follows from  $f$  being submodular. Curvature characterizes how submodular the function is. If  $c_{f} = 0$ , the function is modular, and larger values of  $c_{f}$  correspond to the function exhibiting a stronger diminishing returns structure.
3  $m$ -experts problem
We introduce the  $m$ -experts problem in this section. In this problem, there are  $K$  experts available and each expert makes probabilistic predictions about a sequence of  $T$  binary outcomes. At round  $t \in [T]$ , expert  $i \in [K]$  has a private belief  $b_{i,t} \in [0,1]$  about the outcome  $r_t \in \{0,1\}$ , where  $r_t$  and  $\{b_{i,t}\}_{i=1}^K$  are chosen arbitrarily and potentially adversarially. Expert  $i$  reports  $p_{i,t} \in [0,1]$  as her prediction to the learner. Then, the learner chooses a set  $S_t$  containing  $m$  of the experts. Upon committing to this action, the outcome  $r_t$  is revealed, and expert  $i$  incurs a loss of  $\ell_{i,t} = \ell(b_{i,t}, r_t)$  where  $\ell : [0,1] \times \{0,1\} \to [0,1]$  is a bounded loss function. In this paper, we focus on the quadratic loss function defined as  $\ell(b,r) = (b - r)^2$ . The utility of the learner at round  $t$  is one of the following:
- Modular utility function:  $f_{t}(S_{t}) = \frac{|S_{t}|}{m} - \frac{1}{m}\sum_{i\in S_{t}}\ell_{i,t} = \frac{|S_{t}|}{m} - \frac{1}{m}\sum_{i\in S_{t}}\ell (b_{i,t},r_{t})$ .
- Submodular utility function:  $f_{t}(S_{t}) = 1 - \prod_{i\in S_{t}}\ell_{i,t} = 1 - \prod_{i\in S_{t}}\ell (b_{i,t},r_{t})$ .
It is easy to see that  $f_{t}$  is monotone in both cases and  $\tilde{f}_t(S_t)\in [0,1]$  holds. Note that the utility at each round is defined with respect to the true beliefs of the chosen experts rather than their reported beliefs.
The goal of the learner is twofold:
1) Minimize the  $\alpha$ -regret defined as  $\alpha - R_T = \mathbb{E}\left[\alpha \max_{S \subseteq [K]:|S| = m} \sum_{t=1}^{T} f_t(S) - \sum_{t=1}^{T} f_t(S_t)\right]$ , where the expectation is taken with respect to the potential randomness of the algorithm. For the modular utility function,  $\alpha = 1$  and for the submodular setting, we set  $\alpha = 1 - \frac{c_f}{e}$  (where  $f = \sum_{t=1}^{T} f_t$ ) which is the optimal approximation ratio for any algorithm making polynomially many queries to the objective function.
2) Incentivize experts to report their private beliefs truthfully. To be precise, at each round  $t \in [T]$ , each expert  $i \in [K]$  acts strategically to maximize their probability of being chosen at round  $t + 1$  and the learner's algorithm is called incentive-compatible if expert  $i$  maximizes this probability by reporting  $p_{i,t} = b_{i,t}$ . To be precise, we define the incentive compatibility property below.
Definition 1. An online learning algorithm is incentive-compatible if for every  $t \in [T]$ , every expert  $i \in [K]$  with belief  $b_{i,t}$ , every report  $p_{i,t}$ , reports of other experts  $p_{-i,t}$ , every history of reports  $\{p_{j,s}\}_{j \in [K], s < t}$ , and outcomes  $\{r_s\}_{s < t}$ , we have:
$$
\begin{array}{l} \mathbb {E} _ {r _ {t} \sim \operatorname {B e r n} (b _ {i, t})} \left[ \pi_ {i, t + 1} \mid b _ {i, t}, p _ {- i, t}, \left\{p _ {j, s} \right\} _ {j \in [ K ], s <   t}, \left\{r _ {s} \right\} _ {s <   t} \right] \\ \geq \mathbb {E} _ {r _ {t} \sim \operatorname {B e r n} (b _ {i, t})} \left[ \pi_ {i, t + 1} \mid p _ {i, t}, p _ {- i, t}, \left\{p _ {j, s} \right\} _ {j \in [ K ], s <   t}, \left\{r _ {s} \right\} _ {s <   t} \right], \\ \end{array}
$$
where  $\text{Bern}(b)$  denotes a Bernoulli distribution with probability of success  $b$  and  $\pi_{i,t+1}$  is the probability of expert  $i$  being chosen at round  $t + 1$ .
In other words, an online learning algorithm is incentive-compatible if it is in the best interest of the experts to report their private beliefs truthfully to maximize the probability of being chosen for the next round.
As mentioned earlier, we focus on the quadratic loss function in this paper. The quadratic loss function is an instance of proper loss functions [15], i.e., the following holds:
$$
\mathbb {E} _ {r \sim \operatorname {B e r n} (b)} [ \ell (p, r) ] \geq \mathbb {E} _ {r \sim \operatorname {B e r n} (b)} [ \ell (b, r) ] \forall p \neq b,
$$
i.e., each expert minimizes her expected loss (according to their true belief  $b$ ) by reporting truthfully.
3.1 Motivating applications
Several interesting motivating applications could be cast into our framework. We mention two classes of such applications below.
- Forecasting competitions: In this problem, there are a set of  $K$  forecasters who aim to predict the outcome of sports games or elections (between two candidates). At each round  $t \in [T]$ , information on the past performance of the two opposing teams or candidates is revealed and forecasters provide a probabilistic prediction (as a value between  $[0,1]$ ) about which team or candidate will win. The learner can choose up to  $m$  forecasters at each round and her utility is simply the average of the utilities of chosen experts.
- Online paging problem with advice [16]: There is a library  $\{1, \dots, N\}$  of  $N$  distinct files. A cache with limited storage capacity can store at most  $m$  files at any time. At each round  $t \in [T]$ , a user arrives and requests one file. The learner has access to a pool of  $K$  experts where each expert  $i \in [K]$  observes the user history and makes a probabilistic prediction  $p_{i,t} \in [0,1]^N$  for the next file
request (where  $1^T p_{i,t} = 1$ ). For instance,  $p_{i,t} = e_j$  if expert  $i$  predicts the file  $j \in [N]$  where  $e_j$  is the  $j$ -th standard basis vector. Also,  $r_t = e_j$  if the  $j$ -th file is requested at round  $t \in [T]$ . The learner can choose  $m$  of these experts at each round and put their predictions in the cache. The learner's prediction for round  $t$  is correct if and only if one of the  $m$  chosen experts has correctly predicted the file. Thus, the loss of expert  $i$  can be formulated as  $\ell_{i,t} = \| p_{i,t} - r_t\|_2^2$  and the utility at round  $t$  could be written as  $f_t(S_t) = 2 - \prod_{i \in S_t} \ell_{i,t}$  which is exactly our submodular utility function. Note that this is a slight generalization of our framework where instead of binary outcomes, we consider nonbinary (categorical) outcomes. All our results could be easily extended to this setting as well.
3.2 Naive approach
The WSU algorithm of [10] for the standard experts problem is derived by drawing a connection between online learning and wagering mechanisms. The framework of one-shot wagering mechanisms was introduced by [17] and is as follows: There are  $K$  experts and each expert  $i \in [K]$  holds a belief  $b_i \in [0,1]$  about the likelihood of an event. Expert  $i$  reports a probability  $p_i \in [0,1]$  and a wager  $\omega_i \geq 0$ . A wagering mechanism  $\Gamma$  is a mapping from the reports  $p = (p_1, \ldots, p_K)$ , wagers  $\omega = (\omega_1, \ldots, \omega_K)$  and the realization  $r$  of the binary event to the payments  $\Gamma_i(p, \omega, r)$  to expert  $i$ . It is assumed that  $\Gamma_i(p, \omega, r) \geq 0 \forall i \in [K]$ , i.e., no expert loses more than her wager. A wagering mechanism is called budget-balanced if  $\sum_{i=1}^{K} \Gamma_i(p, \omega, r) = \sum_{i=1}^{K} \omega_i$ . [17, 18] introduced a class of incentive-compatible budget-balanced wagering mechanisms called the Weighted Score Wagering Mechanisms (WSWMs) which is defined as follows: For a fixed proper loss function  $\ell$  bounded in  $[0,1]$ , the payment to expert  $i$  is
$$
\Gamma_ {i} (p, \omega , r) = \omega_ {i} \big (1 - \ell (p _ {i}, r) + \sum_ {j = 1} ^ {K} \omega_ {j} \ell (p _ {j}, r) \big).
$$
The proposed algorithm in [10] is called Weighted-Score Update (WSU) and the update rule for the weights of the experts  $\{\pi_t\}_{t=1}^T$  is the following:
$$
\pi_ {i, t + 1} = \eta \Gamma_ {i} \left(p _ {t}, \pi_ {t}, r _ {t}\right) + (1 - \eta) \pi_ {i, t},
$$
where  $\pi_{i,1} = \frac{1}{K}\forall i\in [K]$ . In other words, the normalized weights of the experts at round  $t$  are interpreted as the wager of the corresponding expert, and the normalized weights at round  $t + 1$  are derived using a convex combination of the weights at the previous round and the payments in WSwM. Note that since WSwM is budget-balanced, the derived weights at each round automatically sum to one and there is no need to normalize the weights (which might break the incentive compatibility). Also, considering the incentive compatibility of WSwM, the WSU algorithm is incentive-compatible as well.
The update rule of WSU could be rewritten as follows:
$$
\pi_ {i, t + 1} = \eta \pi_ {i, t} \big (1 - \ell_ {i, t} + \sum_ {j = 1} ^ {K} \pi_ {j, t} \ell_ {j, t} \big) + (1 - \eta) \pi_ {i, t} = \pi_ {i, t} (1 - \eta L _ {i, t}),
$$
where  $L_{i,t} = \ell_{i,t} - \sum_{j=1}^{K} \pi_{j,t} \ell_{j,t}$ . Therefore, the WSU update rule is similar to that of the Multiplicative Weights Update (MWU) algorithm [19] with the relative loss  $L_{i,t}$  instead of  $\ell_{i,t}$  in the formula.
A simple approach to solving the  $m$ -experts problem with modular utilities  $(\ell_{S,t} = \frac{1}{m}\sum_{j\in S}\ell_{j,t})$  is to define an "expert" for each of the possible  $\binom{K}{m}$  sets of size  $m$  and apply the incentive-compatible WSU algorithm of [10] for the standard experts problem to this setting. Note that we still define incentive compatibility with respect to individual experts (instead of the  $\binom{K}{m}$  meta-experts). To be precise, we define  $\pi_{i,t} = \sum_{S:|S| = m,i\in S}\pi_{S,t}$ . We can show the following:
$$
\pi_ {i, t + 1} = \sum_ {S: | S | = m, i \in S} \pi_ {S, t + 1} = \pi_ {i, t} (1 - \frac {\eta}{m} L _ {i, t}) - \frac {\eta}{m} \sum_ {s \neq i} (\sum_ {S: | S | = m, \{i, s \} \subseteq S} \pi_ {S, t}) \ell_ {s, t}.
$$
Given that  $\pi_{i,t + 1}$  is linear in  $L_{i,t}$ ,  $L_{i,t} = \ell_{i,t} - \sum_{j = 1}^{K}\pi_{j,t}\ell_{j,t}$  is linear in  $\ell_{i,t}$ , and the loss function is proper, we can conclude that incentive compatibility holds in this setting as well.
We summarize the result of this approach in the theorem below.
Theorem 1. If we apply the WSU algorithm of [10] to a standard experts problem with  $\binom{K}{m}$  experts corresponding to each  $S$  with  $|S| = m$ , and set  $\eta = \sqrt{\frac{m \ln(\frac{Ke}{m})}{T}}$ , the algorithm is incentive-compatible and its regret is bounded as follows:
$$
\mathbb {E} [ 1 - R _ {T} ] \leq \mathcal {O} (\sqrt {m T \ln (\frac {K}{m})}).
$$
This approach has two major drawbacks:
1) Computational complexity of maintaining weights for each  $\binom{K}{m}$  feasible sets. In particular, we have to do exponentially many queries to the objective function at each round to update these weights. 2) The regret bound has a  $\sqrt{m}$  dependence on the number of experts  $m$  that is suboptimal.