SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Milton Abramowitz, Irene A Stegun, and Robert H Romer. Handbook of mathematical functions with formulas, graphs, and mathematical tables, 1988.
[2] Ibrahim Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[3] Tal Alon, Magdalen Dobson, Ariel D. Procaccia, Inbal Talgam-Cohen, and Jamie Tucker-Foltz. Multiagent evaluation mechanisms. In AAAI 2020, pages 1774-1781, 2020.
[4] Nivasini Ananthakrishnan, Stephen Bates, Michael I Jordan, and Nika Haghtalab. Delegating data collection in decentralized machine learning. arXiv preprint arXiv:2309.01837, 2023.
[5] Pablo D Azar and Silvio Micali. Computational principal-agent problems. Theoretical Economics, 13(2):553-578, 2018.
[6] Moshe Babaioff, Michal Feldman, and Noam Nisan. Combinatorial agency. In Proceedings of the 7th ACM Conference on Electronic Commerce, pages 18-28, 2006.
[7] Moshe Babaioff, Michal Feldman, Noam Nisan, and Eyal Winter. Combinatorial agency. Journal of Economic Theory, 147(3):999-1034, 2012.
[8] Stephen Bates, Michael I Jordan, Michael Sklar, and Jake A Soloff. Principal-agent hypothesis testing. arXiv preprint arXiv:2205.06812, 2022.
[9] Devansh Bisla, Apoorva Nandini Saridena, and Anna Choromanska. A theoretical-empirical approach to estimating sample complexity of dnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3270–3280, 2021.
[10] Patrick Bolton and Mathias Dewatripont. Contract theory. MIT press, 2004.
[11] Olivier Bousquet, Steve Hanneke, Shay Moran, Ramon Van Handel, and Amir Yehudayoff. A theory of universal learning. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 532-541, 2021.
[12] Olivier J Bousquet, Amit Daniely, Haim Kaplan, Yishay Mansour, Shay Moran, and Uri Stemmer. Monotone learning. In Conference on Learning Theory, pages 842-866. PMLR, 2022.
[13] Michael L. Bynum, Gabriel A. Hackebeil, William E. Hart, Carl D. Laird, Bethany L. Nicholson, John D. Siirola, Jean-Paul Watson, and David L. Woodruff. *Pyomo-optimization modeling in python*, volume 67. Springer Science & Business Media, third edition, 2021.
[14] Yang Cai, Constantinos Daskalakis, and Christos H. Papadimitriou. Optimum statistical estimation with strategic data sources. In COLT 2015, pages 280-296, 2015.
[15] Gabriel Carroll. Robustness and linear contracts. American Economic Review, 105(2):536-563, 2015.
[16] Junjie Chen, Minming Li, and Haifeng Xu. Selling data to a machine learner: Pricing via costly signaling. In International Conference on Machine Learning, pages 3336-3359. PMLR, 2022.
[17] Alon Cohen, Argyrios Deligkas, and Moran Koren. Learning approximately optimal contracts. In SAGT 2022, pages 331-346, 2022.
[18] Paul Dütting, Tim Roughgarden, and Inbal Talgam-Cohen. Simple versus optimal contracts. In EC 2019, pages 369-387, 2019.
[38] Sébastien Lahaie. How to take the dual of a linear program. Columbia University, New York, 2008.
[39] Yann LeCun. The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/, 1998.
[40] Erich Leo Lehmann, Joseph P Romano, and George Casella. Testing statistical hypotheses, volume 3. Springer, 1986.
[41] Rafid Mahmood, James Lucas, David Acuna, Daiqing Li, Jonah Philion, Jose M Alvarez, Zhiding Yu, Sanja Fidler, and Marc T Law. How much more data do I need? Estimating requirements for downstream tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 275-284, 2022.
[42] David Martimort and Jean-Jacques Laffont. The theory of incentives: the principal-agent model. Princeton University Press, 2009.
[43] Felix Mohr, Tom J Viering, Marco Loog, and Jan N van Rijn. Lcdb 1.0: An extensive learning curves database for classification tasks. Machine Learning and Knowledge Discovery in Databases, ECMLPKDD. p. accepted. Lecture Notes in Computer Science, Springer, 2022.
[44] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
[45] Phillippe Rigollet and Jan-Christian Hütter. High dimensional statistics. Lecture notes for course 18S997, 813(814):46, 2015.
[46] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=ryenvpEKDr.
[47] Royal Swedish Academy of Sciences. Scientific background on the 2016 Nobel Prize in Economic Sciences, 2016.
[48] Bernard Salanie. The Economics of Contracts: A Primer. MIT press, 2017.
[49] Tom Viering and Marco Loog. The shape of learning curves: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[50] Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Veselin Stoyanov. Training trajectories of language models across scales. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13711-13738. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.ACL-LONG.767.
[51] Xinyi Zhao, Weixin Liang, and James Zou. Data budgeting for machine learning. arXiv preprint arXiv:2210.00987, 2022.
[52] Banghua Zhu, Stephen Bates, Zhuoran Yang, Yixin Wang, Jiantao Jiao, and Michael I. Jordan. The sample complexity of online contract design. In Proceedings of the 24th ACM Conference on Economics and Computation, EC 2023, London, United Kingdom, July 9-12, 2023, page 1188. ACM, 2023. doi: 10.1145/3580507.3597673.
A Broader implications
In this paper, we set out to formalize and study the task of delegating classification through the lens of contract design. Given that our work is largely motivated by the increasingly common practice of outsourcing learning tasks to specialized service providers, we believe our algorithm, analysis, and empirical observations carry meaningful implications for practitioners and decision-makers alike. At the same time, it is important to remember that our work—as others considering economic aspects of learning—studies delegation in a simplified setting and under certain assumptions. As such, and since devising and agreeing to legally-binding contracts can have concrete implications on real-world outcomes, care should be taken when applying ideas or conclusions that derive from our work in practice.
For example, consider that our formalism relies on the assumption that the learning agent is all-knowing and rational. Yet in reality, agents must act under partial information, face irreducible (and often unquantifiable) uncertainty, and—being human—are subject to common behavioral biases. It is unclear a-priori if and how our statements and conclusions carry over to this setting. As another example, notice that our formalism considers a one-shot setting where a single contract between a single principal-agent pair is instantiated once. But in reality, competition and long-term reputation may play a significant role in determining the agent's incentive structure, and consequently, her behavior. In such cases, naively applying our framework without careful inspection of the appropriate incentives on both ends can result in sub-optimal contracts, possibly to the detriment of all involved parties. Nonetheless, given that our work aims to take an initial step towards establishing delegated learning, we view its extension to non-rational agents and temporal and competitive settings as intriguing future work.
One message that our work conveys is that in delegation, simplicity, in the form of threshold contracts, has merit. This draws connections to other works that similarly argue for simplicity as an important and useful property of effective delegation mechanisms. Our work shows that simple contracts are, under reasonable conditions, computationally feasible and theoretically optimal. Economically, threshold contracts are practically appealing since they are easy to understand, communicate, and regulate. Given that contracts are in essence social constructs, we believe these properties are key for establishing threshold contracts as effective building blocks for machine learning markets.
B Min-budget contract design - deferred proofs
Notation. To align with traditional contract design notation, in this section, we denote the action space by  $\mathcal{A} = [n] = \{1,\dots ,n\}$ , and the outcome space by  $\Omega = \{0,\ldots ,m\}$ . We denote by  $\Delta (\mathcal{X})$  the set of distributions over a given set  $\mathcal{X}$ . For contract design problems, we denote the outcome probabilities by  $F_{i,j}$ , such that  $F_{i}\in \Delta (\Omega)$  is the outcome distribution associated with action  $i$ , and also denote the cost of each action by  $c_{i}$ . Given  $x\in \mathbb{R}$ , we denote  $x^{+} = \mathrm{ReLU}(x) = \max \left\{0,x\right\}$ . We denote the indicator function by  $\mathbb{1}[\cdot ]$ , the total variation distance between distributions  $P,Q$  by  $\| P - Q\|_{\mathrm{TV}}$  (see Definition 4), and the survival function of  $P\in \Delta (\Omega)$  by  $\mathbb{S}_P(\cdot)$  (see Definition 14).
A note on individual rationality. In the contract design literature, a contract is said to be incentive compatible (IC) with respect to some action  $a^*$  if it satisfies  $u_{a^*}(t) \geq u_a(t)$  for all actions  $a \in \mathcal{A}$ . In the delegated classification setting, this corresponds to the constraint  $n(t) = n^*$  in Eq. (4). As an additional constraint, contracts in which the agent's expected utility is always non-negative ( $u_{n^*}(t) \geq 0$ ) are said to be individually rational (IR). In the case of delegated classification, it is natural to assume that there always exists a valid action that the agent can take at zero cost — for example, returning a dummy classifier that always abstains from prediction (thus having zero accuracy), or a classifier which makes a prediction at random. As such, contracts in the delegated classification setting can be assumed to be individually rational without loss of generality, as any action  $n$  that is chosen by the agent has utility which is weakly larger than the utility of the zero-cost action, which is always non-negative. Moreover, even in cases where a zero-loss action does not exist, we show that individual rationality (IR) can be attained in a straightforward manner, by adding the minimal cost  $c_1$  to each entry of an incentive compatible (IC) contract ( $t_j \mapsto t_j + c_1$ ):
Claim 1. Given an IC contract  $t$  with  $c_{1} > 0$ , the contract  $t + c_{1}$  (add  $c_{1}$  to every coordinate of the contract) is both IC and IR.
Proof. An IC contract implementing action  $a^*$  satisfies  $u_{a^*}(t) \geq u_a(t)$ , for all actions  $a \in \mathcal{A}$ . In particular, this inequality holds in relation to the least-costly action, denoted by  $1 \in \mathcal{A}$ . Plugging the definition of the agent's expected utility (Eq. (2)), we obtain  $u_{a^*}(t) \geq \mathbb{E}_{j \sim f_1}[t] - c_1$ . Thus, under
the mapping  $t_j^{\prime} = t_j + c_1$ , it holds that:
$$
u _ {a ^ {*}} (t ^ {\prime}) = u _ {a ^ {*}} (t + c _ {1}) \geq \mathbb {E} _ {j \sim f _ {1}} [ t + c _ {1} ] - c _ {1} = \mathbb {E} _ {j \sim f _ {1}} [ t ] \geq 0
$$
and therefore the contract  $t' = t + c_1$  is individually rational and still implements action  $a^*$ .
B.1 Relation between budget-optimal and min-budget contracts
Proof of Proposition 1. Given budget  $B > 0$ , denote by  $t_{\mathrm{BO}}$  the budget-optimal contract, and denote the action it implements by  $n^*$ . By definition,  $t_{\mathrm{BO}}$  is a feasible solution to the min-budget contract design problem implementing action  $n^*$ . Denote by  $t_{\mathrm{MB}}$  the corresponding optimal solution to the min-budget problem implementing action  $n^*$  (Eq. (4)).  $t_{\mathrm{MB}}$  implements the same action  $n^*$  by definition, and satisfies  $\| t_{\mathrm{MB}} \|_{\infty} \leq \| t_{\mathrm{BO}} \|_{\infty} \leq B$  due to the optimization objective. Hence,  $t_{\mathrm{MB}}$  is a budget-optimal contract for the given budget  $B$ , which is also min-budget.
Iterative min-budget. To find the budget-optimal contract using iterative applications of Eq. (4), we observe that any budget-limited contract  $t: \Omega \to [0, B]$  has bounded expected pay  $\mathbb{E}_{f_n}[t] \leq B$  for any distribution  $f_n$ . Hence, actions  $n'$  with cost  $c_{n'} > B$  cannot be implemented using budget  $B$ , as the agent's utility  $u_{n'} = \mathbb{E}_{f_{n'}}[t] - c_{n'} < 0$  will be smaller than the utility of the zero-cost action  $u_0 = \mathbb{E}_{f_n}[t] - 0 \geq 0$ . Define the reduced action set:
$$
\mathcal {A} ^ {\prime} = \{n \in \mathcal {A} \mid c _ {n} \leq B \wedge n \text {i s i m p l e m e n t a b l e} \}
$$
$\mathcal{A}'$  is finite when  $\mathcal{A}$  is finite, or when the data cost is unbounded and the learning curve is monotone. To find  $n^*$  within this space, go over all  $n \in \mathcal{A}'$ , calculate the minimal budget  $B_n$  required for implementation, and take  $\mathrm{argmin}_{n \in \mathcal{A}'} B_n$ . This is possible within a finite number of steps.
B.2 The min-budget linear program and equivalent forms
The min-budget contract (Eq. (4)) implementing action  $i \in \mathcal{A}$  is given by the MIN-BUDGET linear program:
$$
\min_{\substack{t\in \mathbb{R}_{\geq 0}^{\lfloor \Omega \rfloor},B\in \mathbb{R}_{\geq 0}}}B
$$
$$
\forall j \in \Omega : t _ {j} \leq B \tag {6}
$$
$$
\forall i ^ {\prime} \neq i: \sum_ {j \in \Omega} F _ {i ^ {\prime}, j} t _ {j} - c _ {i ^ {\prime}} \leq \sum_ {j \in \Omega} F _ {i, j} t _ {j} - c _ {i} \tag {IC}
$$
The dual of the min-budget LP is given by:
Claim 2. The dual linear program of Eq. (6) is given by:
$$
\max_{\lambda \in \mathbb{R}^{n - 1}_{\geq 0},\mu \in \mathbb{R}^{| \Omega |}_{\geq 0}}\sum_{i^{\prime}\neq i}\left(c_{i} - c_{i^{\prime}}\right)\lambda_{i^{\prime}}
$$
$$
\forall j \in \Omega : \sum_ {i ^ {\prime} \neq i} \left(F _ {i, j} - F _ {i ^ {\prime}, j}\right) \lambda_ {i ^ {\prime}} \leq \mu_ {j} \tag {7}
$$
$$
\sum_ {j \in \Omega} \mu_ {j} \leq 1
$$
Proof. We take the dual by translating the optimization problem into canonical form. The canonical form we target:
$$
\min  _ {x \geq 0} c ^ {T} x \quad \text {s . t .} \quad C x \leq d
$$
and its dual, as given by Lahaie [38], is:
$$
\max  _ {y \geq 0} - d ^ {T} y \quad \text {s . t .} \quad C ^ {T} y \geq - c
$$
To translate Eq. (6), we note that in our case the components  $c, C, d$  are given by:
$$
c ^ {T} = ( \begin{array}{c c c c} 0 & \ldots & 0 & 1 \end{array} ) \in \mathbb {R} ^ {| \Omega | + 1}
$$
$$
C = \left( \begin{array}{c c c c c c} F _ {1, 0} - F _ {i, 0} & \dots & & & F _ {1, | \Omega |} - F _ {i, | \Omega |} & 0 \\ \vdots & \ddots & & & \vdots & \vdots \\ & & F _ {i ^ {\prime}, j} - F _ {i, j} & & \\ & & & \ddots & \\ F _ {n, 0} - F _ {i, 0} & & & & F _ {n, m} - F _ {i, m} & 0 \\ \hline & & & & & - 1 \\ & & I _ {m + 1} & & & \vdots \\ & & & & & - 1 \end{array} \right) \in \mathbb {R} ^ {(n - 1 + | \Omega |) \times (| \Omega | + 1)}
$$
$$
d ^ {T} = \left( \begin{array}{c c c c c c} c _ {1} - c _ {i} & \ldots & c _ {n} - c _ {i} & | & 0 & \ldots & 0) \in \mathbb {R} ^ {n - 1 + | \Omega |} \end{array} \right.
$$
To simplify formulation, we denote the dual optimization variable  $y$  as follows:
$$
y ^ {T} = \left( \begin{array}{c c c c c c c c} \lambda_ {1} & \ldots & \lambda_ {i - 1} & \lambda_ {i + 1} & \ldots \lambda_ {n} & \mid & \mu_ {0} & \ldots & \mu_ {m} \end{array} \right) \in \mathbb {R} ^ {n - 1 + | \Omega |}
$$
Under this formulation, the dual's objective is:
$$
- d ^ {T} y = \sum_ {i \neq i ^ {\prime}} \left(c _ {i} - c _ {i ^ {\prime}}\right) \lambda_ {i ^ {\prime}} \tag {8}
$$
The constraints given by the first  $m$  rows of  $C^T$  are:
$$
\forall j \in \Omega : \sum_ {i ^ {\prime} \neq i} \left(F _ {i ^ {\prime}, j} - F _ {i, j}\right) + \mu_ {j} \geq 0
$$
and equivalently:
$$
\forall j \in \Omega : \sum_ {i ^ {\prime} \neq i} \left(F _ {i, j} - F _ {i ^ {\prime}, j}\right) \leq \mu_ {j} \tag {9}
$$
The constraint corresponding to the last row of  $C^T$  is given by:
$$
\sum_ {j \in \Omega} - \mu_ {m} \geq - 1
$$
$$
\sum_ {j \in \Omega} \mu_ {m} \leq 1 \tag {10}
$$
Combining equations (8, 9, 10) yields the linear program given by Eq. (7).
To map between contracts and hypothesis tests, we introduce the following variable transformation:
Definition 3 (Statistical representation of contracts). For a given contract  $t: \Omega \to \mathbb{R}_{\geq 0}$ , denote  $B = \max_{j} t_{j}$ . The statistical representation of  $t$  is given by:
$$
t _ {j} = \phi_ {j} \beta^ {- 1}
$$
where  $\phi_j\in [0,1]$  and  $\beta = B^{-1}$ .
Note that the transformation  $(t,B)\mapsto (\phi ,\beta)$  is non-linear, and well-defined for all  $B > 0$  . Under this variable transformation, the MIN-BUDGET transforms to an equivalent linear program:
Lemma 2. For a feasible design problem, the min-budget linear program (Eq. (6)) is equivalent to:
$$
\max_{\substack{\phi \in [0,1]^{\left|\Omega \right|},\beta \in \mathbb{R}_{\geq 0}}}\beta
$$
s.t. (11)
$$
\forall i ^ {\prime} \neq i: \sum_ {j \in \Omega} F _ {i, j} (1 - \phi_ {j}) + \sum_ {j \in \Omega} F _ {i ^ {\prime}, j} \phi_ {j} \leq 1 - (c _ {i} - c _ {i ^ {\prime}}) \beta
$$
Proof. Given Eq. (6), define  $\phi \in [0,1]^{\left|\Omega\right|}$ $\beta \geq 0$  according to Definition 3:
$$
B = \beta^ {- 1} \tag {12}
$$
$$
t _ {j} = \phi_ {j} B = \phi_ {j} \beta^ {- 1}
$$
Under the transformation defined by Eq. (12), the (BUDGET) constraint in Eq. (6) transforms as follows:
$$
t _ {j} \leq B \Leftrightarrow \phi_ {j} \beta^ {- 1} \leq \beta^ {- 1} \tag {13}
$$
$$
\Leftrightarrow \phi_ {j} \leq 1
$$
and the (IC) constraint in Eq. (6) transforms as:
$$
\begin{array}{l} \sum_ {j} F _ {i, j} t _ {j} - c _ {i} \geq \sum_ {j} F _ {i ^ {\prime}, j} t _ {j} - c _ {i ^ {\prime}} \Leftrightarrow \sum_ {j} \left(F _ {i, j} - F _ {i ^ {\prime}, j}\right) \phi_ {j} \beta^ {- 1} \geq c _ {i} - c _ {i ^ {\prime}} \\ \Leftrightarrow \sum_ {j} F _ {i, j} \phi_ {j} - \sum_ {j} F _ {i ^ {\prime}, j} \phi_ {j} \geq \left(c _ {i} - c _ {i ^ {\prime}}\right) \beta \\ \Leftrightarrow 1 - \sum_ {j} F _ {i, j} \left(1 - \phi_ {j}\right) - \sum_ {j} F _ {i ^ {\prime}, j} \phi_ {j} \geq \left(c _ {i} - c _ {i ^ {\prime}}\right) \beta \tag {14} \\ \Leftrightarrow \sum_ {j} F _ {i, j} (1 - \phi_ {j}) + \sum_ {j} F _ {i ^ {\prime}, j} \phi_ {j} \leq 1 - \left(c _ {i} - c _ {i ^ {\prime}}\right) \beta \\ \end{array}
$$
where the first equivalence is by Eq. (12), and the third equivalence is valid as  $\sum_{j}F_{i^{\prime},j} = 1$  for all  $i^{\prime}$ . Finally, the objective of Eq. (6) transforms as:
$$
\min  B \Leftrightarrow \max  \beta \tag {15}
$$
Combining equations (13, 14, 15) yields the linear program in Eq. (11).
B.3 Relation to min-pay contract design
Min-pay contract design aims to design a contract which minimizes the expected pay under the implemented action  $n^{*}$ :
$$
t ^ {*} = \operatorname {a r g m i n} _ {t} \mathbb {E} _ {j \sim f _ {n ^ {*}}} [ t _ {j} ] \quad \text {s . t .} \quad n (t) = n ^ {*} \tag {16}
$$
In contrast to the min-budget contract (Eq. (4)), the  $\| t\|_{\infty}$  objective measuring maximal pay is replaced with the  $\mathbb{E}_{j\sim f_{n^*}}[t_j]$  objective measuring expected pay. Eq. (16) is equivalent to the MIN-PAY linear program:
$$
\min_{t\in \mathbb{R}_{\geq 0}^{|\Omega}}\sum_{j\in \Omega}F_{i,j}t_{j}
$$
s.t. (17)
$$
\forall i ^ {\prime} \neq i: \sum_ {j \in \Omega} F _ {i ^ {\prime}, j} t _ {j} - c _ {i ^ {\prime}} \leq \sum_ {j \in \Omega} F _ {i, j} t _ {j} - c _ {i} \quad (\mathrm {I C})
$$
B.3.1 Equivalence of implementability
The implementatability of min-pay contracts is characterized in Dütting et al. [19]. We cite the main result for completeness:
Proposition 3 (Min-pay implementability; [19], A.2). An action  $i \in \mathcal{A}$  is implementable (up to tie-breaking) if and only if there is no convex combination of the other actions that results in the same distribution  $f_{i} = \sum_{i^{\prime} \neq i} \alpha_{i^{\prime}} f_{i^{\prime}}$ , but lower cost  $c_{i} > \sum_{i^{\prime} \neq i} \alpha_{i^{\prime}} c_{i^{\prime}}$ .
Figure 5: Qualitative comparison of min-pay and min-budget contracts under MLRP. (Left) The contract design setting, representing two possible actions with binomial outcome distributions  $(p_1 = 0.5, p_2 = 0.8, m = 15)$ . Action 1 has zero cost  $c_1 = 0$ , and action 2 has unit cost  $c_2 = 1$ . (Right) The resulting min-pay (red) and min-budget (purple) contracts, obtained by solving Eq. (17) and Eq. (6), respectively. The min-pay awards payment only for the highest outcome, in alignment with Proposition 4; the min-budget is a threshold contract, in alignment with claim 12.
We leverage to Proposition 3 to characterize the implementability of min-budget contracts. This is possible due to the following connection:
Claim 3 (Implementability equivalence). A contract  $t$  is feasible solution of MIN-BUDGET if and only if it is a feasible solution of MIN-PAY.
Proof. A contract  $t$  which satisfies MIN-BUDGET (Eq. (6)) satisfies the (IC) constraint, and thus also satisfies MIN-PAY (Eq. (17)). Conversely, a contract  $t$  which satisfies MIN-PAY satisfies the (IC) constraint. Set  $B = \max_{j} t_{j}$  and obtain a feasible solution to MIN-BUDGET.
B.3.2 Functional form of min-pay contracts under MLRP
The optimal min-pay contracts under the MLRP assumption (Definition 1) are characterized in [19]:
Proposition 4 (Optimal min-pay contract under MLRP; [19], Lemma 7). Consider a contract design setting for which MLRP holds. If the highest-cost action  $n$  is implementable, then there is a min-pay contract has a single nonzero-payment, which is rewarded for the highest outcome  $m$ .
See Fig. 5 for a qualitative comparison of min-pay and min-budget contracts.
Min-pay contracts in delegated classification settings with MLRP. While simple, the min-pay contract given by Proposition 4 would be impractical in many realistic scenarios of delegated classification. In a delegated classification setting, the highest outcome corresponds to  $100\%$  validation set accuracy (i.e all validation set samples classified correctly). As  $m$  grows, the highest outcome becomes exponentially less likely, and the min-pay contract awards increasingly high payment with increasingly low probability (See Fig. 6). In such settings, even a slight degree of risk-aversion is likely to affect the agent's decisions: From the agent's perspective, the probability of receiving any payment from a min-pay contract may become small to a degree where even a slight degree of agent risk-aversion will manifest itself in the decision-making process. From the principal's perspective, even though min-pay contracts guarantee small payment in expectation, the amount payment in the case of a rare event would quickly become unfeasible.
B.4 Min-budget contracts with two actions
In this section, we explore min-budget settings with two actions, and prove Theorem 2. When  $n = 2$ , assume without loss of generality that the contract implements action  $i = 2$ , and denote  $c = c_2 - c_1 > 0$ .
We recall the definition of total variation distance:
Definition 4 (Total variation distance). Given two distributions  $P, Q \in \Delta(\Omega)$ , the total variation distance between  $P$  and  $Q$  is:
$$
\| P - Q \| _ {\mathrm {T V}} = \frac {1}{2} \| P - Q \| _ {1}
$$