SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Conversely, assume that  $t'$  is an optimal min-budget contract (without minimum guaranteed payout), satisfying the MIN-BUDGET linear program in Eq. (17) with budget  $B'$ . Apply the inverse transformation  $t = t' + \delta$  and  $B = B' + \delta$ , and the equivalence relations (44, 43, 45) in the inverse direction to obtain that  $t, B$  is an optimal solution to Eq. (41).
C Experimental details
C.1 Data
LCDB. We base our main experimental environment on the LCDB dataset [43], which includes a large collection of stochastic learning curves for multiple learning algorithms and classification benchmark datasets. For each learning method and benchmark dataset, the database includes held-out accuracy measurements, obtained for exponentially-increasing training set sizes  $n \in \{2^4 = 16, \dots, \text{round}(2^{k/2}), \dots, 2^{15} = 32, 768\}$ , with multiple repetitions per  $n$  obtained by cross-validation. For all learning curves we consider in our analysis, and for each  $n$ , the number of
repetitions provided in the dataset is in the range  $R \in \{5, \dots, 125\}$ , where the specific number of repetitions in LCDB depends on the algorithm and benchmark dataset (see the dataset documentation for additional details). For each trained classifier, each accuracy point on the learning curve is estimated using 5,000 held-out samples. Formally, and using our notation, for each learning algorithm Alg (e.g. MLP), dataset  $D$  (e.g. MNIST), and training set size  $n$ , LCDB provides a set of accuracy estimates  $\{a_{n}^{1,\mathrm{Alg},D}, \dots, a_{n}^{R,\mathrm{Alg},D}\}$ , such that each  $a_{n}$  is distributed according to  $a_{n} \sim \mathrm{acc}_{D}(h_{n})$  where  $h_{n}$  is a classifier trained using training set  $S \sim D^{n}$  and learning algorithm Alg (see Section 2 for definition of  $\mathrm{acc}_D(h_n)$ ).
Benchmark dataset and algorithms. In our main analysis, we focus on the popular MNIST dataset [39, OpenML 554], and on multilayer perceptrons (MLP) and gradient-boosted decision trees (GBDT) as representative classifiers. For all classifiers, results are obtained for the respective default scikit-learn [44] implementations (e.g. MLPClassifier and GradientBoostingClassifier for MLP and GBDT, respectively).
Delegated learning settings from empirical data. For a given validation set size  $m$  where  $m = |V|$ , we instantiate a contract design task  $(\mathcal{A}, c, \Omega, F)$  as follows: (i) the set of actions with the set of training set sizes provided by LCDB  $(\mathcal{A} = \{2^4, 2^{4.5}, \dots, 2^{15}\})$ ; (ii) action costs are set to fixed per-unit cost, i.e.,  $c_n = n$ ; and (iii) the distribution  $F$  over outcomes  $\Omega$  is associated with a binomial mixture distribution, resulting from applying bootstrap sampling to empirical error measurements:
$$
f _ {n} = \frac {1}{R} \sum_ {r = 1} ^ {R} \operatorname {B i n o m i a l} (m, a _ {n} ^ {r, \mathrm {A l g}, D})
$$
where  $a_{n}$  are the accuracy estimates defined above. Fig. 2 (Left) shows an example of such a setting, obtained by applying the above procedure to data describing learning curves for the MLP algorithm trained on the MNIST benchmark dataset.
C.2 Implementation details
Code. We implement our code in Python. Our code relies on Pyomo [13, 27] and GLPK for solving linear and mixed-integer programs.
Code is available at: https://github.com/edensaig/delegated-classification.
Hardware. All experiments were run on a single laptop, with 16GB of RAM, M1 Pro processor, and with no GPU support.
Runtime. A single run consisting the entire pipeline takes roughly 5 minutes. The main bottleneck is running the LP solvers, taking roughly  $70\%$  of runtime to compute.
C.2.1 Contract design solvers
To find the optimal contracts, we implement and compare several different solvers:
- LP solver: To find min-budget contracts given outcome distributions  $\{f_n\}$  and costs  $\{c_n\}$ , we solve the MIN-BUDGET LP (Eq. (6)) directly using Pyomo and GLPK. Given budget  $B$ , budget-optimal contracts are obtained by iterating through the actions set  $\mathcal{A}$ , invoking the min-budget solver on each target action  $n \in \mathcal{A}$ , enforcing incentive compatibility against all actions  $n'$  which satisfy  $\alpha_{n'} < \alpha_n$ , and taking the action yielding the maximal expected accuracy within budget (see Appendix B.1). In our code, the LP solver is implemented within the MinBudgetContract class. Typical running time for a single problem instance:  $10^{-1}\mathrm{s}$ .
- SBA solver: Implementation of the single binding action algorithm presented in Section 3.4. The local solver is implemented within the MinBudgetSingleBindingConstraintContract class. Typical running time for a single problem instance:  $10^{-4}\mathrm{s}$ .
- Hybrid solver: A meta-solver implementing the 'try SBA first' computational approach. The solver starts by invoking the SBA solver, and applies the LP solver if the former fails. In our code, the LP solver is implemented within the MinBudgetHybridContract class. Running time depends on whether SBA is applicable.
- MIP solver: To find optimal all-or-nothing contracts, we solve the MIN-BUDGET LP in its statistical formulation (Eq. (11)) while restricting  $\phi_j \in \{0, 1\}$ . This turns the LP into a mixed-integer program, which can also be solved using GLPK. In our code, the IP solver is implemented within the MinBudgetStatisticalContract class. Typical running time for a single problem instance:  $10^{-1}$ s.
- Min-pay LP solver: To compare between min-budget and min-pay contracts (Appendix B.3), min-pay contracts were obtained by solving the MIN-PAY LP (Eq. (17)) using Pyomo and GLPK, similar to the full min-budget LP solver. In our code, the min-pay is implemented within the MinPayContract class. Running time is similar to the other LP-based methods.
- Full enumeration solver: To find all-or-nothing contracts for low-dimensional problems and avoid numerical instabilities, we implement a solver which performs full enumeration of all-or-nothing contracts. By the statistical variable transformation in Definition 3, the (IC) constraint in the MIN-BUDGET LP translates to  $\sum_{j\in \Omega}\left(F_{i,j} - F_{i',j}\right)\phi_j\leq (c_i - c_{i'})\beta$ , where  $i$  is the target action,  $i'\in \mathcal{A}\setminus \{i\}$ , and the objective is minimize  $\beta$  (see Eq. (11)). Thus, when  $\phi \in \{0,1\}^{\Omega}$  is fixed, the optimal  $\beta$  is given by:
$$
\beta^ {*} (\phi) = \min _ {i ^ {\prime} \in \mathcal {A} \backslash \{i \}} \frac {\sum_ {j \in \Omega} \left(F _ {i , j} - F _ {i ^ {\prime} , j}\right) \phi_ {j}}{c _ {i} - c _ {i ^ {\prime}}}
$$
and optimizing  $\beta^{*}$  over  $\phi \in \{0,1\}^{\Omega}$  yields an optimal all-or-nothing contract. The full enumeration solver is implemented within the FullEnumerationContract class. For for enumeration of all-or-nothing contracts, this solver is mostly applicable for small problem instances ( $m < 20$ ) due to its exponential complexity. In contrast, for threshold contracts the number of possible  $\phi$  configurations is linear in  $m$ , and therefore enumeration is more efficient.
- Fixed-budget solver: To find budget-optimal threshold contracts given a fixed budget  $B$ , we implement a simple solver which iterates through all possible threshold contracts  $t_{j_0}(j) = B\mathbb{1}[j \geq j_0]$  for all  $j_0 \in \{0, \dots, m\}$ . The solver then simulates the agent's rational choice (Eq. (2)) and selects the value of  $j_0$  which incentivizes the best action. In case of ties between possible values of  $j_0$ , they are broken in favor of larger values, as this was shown to lead to greater numerical stability. The fixed-budget solver is implemented within the FixedBudgetThresholdContract class.
C.3 Empirical prevalence estimation
Since our theoretical analysis relies on MLRP assumptions, we would like to understand whether the results are applicable to real-world learning curves. Towards this, we run an empirical prevalence evaluation on the MNIST dataset. For each learning algorithm Alg, and training set size  $n$ , we construct a delegated learning setting with  $m = 10$  and collect the following statistics:
- Structural properties:
- Is MLRP? (\% MLRP): Check whether all pairs  $n_1, n_2 \leq n$  such that  $c_{n_1} < c_{n_2}$  satisfy the monotone likelihood ratio property (see Definition 1).
Computational properties:
- SBA successful? (% SBA): Check whether the SBA algorithm was successful on the given instance.
- Functional form of optimal contract:
- Is monotone? ( $\%$  M): Check whether the resulting min-budget contract satisfies  $t_j \geq t_{j-1}$  for all  $j \in [m]$ .
- Is all or nothing? (% AoN): Check whether the resulting min-budget contract is an all-or-nothing contract (i.e. whether there exists  $j_0 \in \Omega, B > 0$  such that  $t_j = 0$  for all  $j < j_0$  and  $t_j = B$  otherwise).
- Is threshold? (\% T): Check whether the resulting min-budget contract is a threshold contract (i.e. whether there exists  $j_0 \in \Omega, B > 0$  such that  $t_j = 0$  for all  $j < j_0$  and  $t_j = B$  otherwise).
- Excess cost:
- Min-budget: Optimal objective  $B_{\mathrm{LP}}$  of MIN-BUDGET LP without additional restrictions, implemented using the full LP solver.
- All-or-nothing budget: Optimal objective of MIN-BUDGET LP, when the resulting contract is restricted to all-or-nothing form  $t_j \in \{0, B_{\mathrm{AoN}}\}$ . Implemented using the full enumeration solver.
- Threshold budget: Optimal objective of MIN-BUDGET LP, when the resulting contract is restricted to threshold form  $t_j = B_{\mathrm{Thr}} \mathbb{1}[j \geq j_0]$ . Implemented using the full enumeration solver.
Table 2: Empirical robustness estimation on the MNIST dataset. Each row corresponds to a learning algorithm, and columns are specified in Appendix C.3. Results are averaged across all implementable actions.
Structure	Compute	Functional form of opt. contract	Excess cost
% MLRP	% SBA	% M	% AoN	% T	AoN	T
MLP	100%	94.4%	100%	94.4%	94.4%	0.04%	0.04%
GBDT	100%	76.5%	100%	82.4%	82.4%	0.71%	0.71%
Logistic	93.8%	81.2%	93.8%	93.8%	93.8%	0.00%	0.00%
Perceptron	68.8%	75%	93.8%	93.8%	93.8%	0.00%	0.00%
Linear SVM	71.4%	78.6%	85.7%	85.7%	85.7%	0.95%	1.45%
Poly SVM	100%	94.4%	100%	94.4%	94.4%	0.24%	0.24%
RBF SVM	100%	94.4%	100%	94.4%	94.4%	0.04%	0.04%
KNN	100%	100%	100%	100%	100%	0.00%	0.00%
Overall	92.6%	87.4%	97%	92.6%	92.6%	0.22%	0.26%
- The excess cost columns in Table 2 indicate the relative excess cost incurred by restricting the min-budget optimization space to simple contracts  $\left(\frac{B_{\{\mathrm{AoN,Thr}\}} - B_{\mathrm{LP}}}{B_{\mathrm{LP}}}\right)$ . As all-or-nothing contracts are a superset of threshold contracts, we expect this excess cost to be smaller than the one associated with threshold contracts. The averaging in table 2 is performed on problem instances where both all-or-nothing and threshold contracts are feasible.
Averaging over all implementable actions, we obtain the results in Table 2. The results show that threshold contracts are relatively common in this dataset (more than  $85\%$ ), and that excess cost of simple contracts is relatively low (around  $1\%$ ), indicating that threshold contracts may provide good approximation to the optimal min-budget contracts in some cases. All simple optimal contracts in our dataset had a threshold functional form, and some simple optimal contracts were not recovered by SBA, indicating that a tighter characterization of simple contracts may be possible.
Fig. 10 provides qualitative interpretation of the analysis for a selected subset of learners. While the survival functions are not perfectly concave (and thus don't satisfy Concave-MLRP), the SBA algorithm successfully terminates in three cases (MLP, Logistic, KNN). The binding action is  $a_{N - 1}$  for two of the classifiers (MLP, KNN), similar to the condition implied by Theorem 4. Note that SBA also terminated successfully on Logistic Regression data, despite the learning curve having a distinctive non-concave shape. In the case of GBDT, the survival function is not concave, and the optimal contract does not assume a threshold form.
Figure 10: Qualitative comparison of optimal contract computation on LCDB MNIST data ( $n^* = 16384$ ,  $m = 25$ ). Each row represents a learning algorithm. (Left column) Expected accuracy curves  $\alpha_{n}$ . (Center column) Survival function at transition  $s_i$ , whenever SBA is successful, the binding actions are marked with triangles. (Right column) Optimal contract  $t_j^*$ .