SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A brief summary of the applicative background in AI for justice is the following. The learner wants to maximize the total number of appearances to court for people of concern. To achieve this goal, the learner is able to provide, or not, some transportation assistance: rideshare assistance (the highest level of help), or a transit voucher (a more modest level of help). There are of course budget limits on these assistance means, and the learner also wants to control how (un)fair the assistance policy is, in terms of subgroups of the population, while maximizing the total number of appearances. Some subgroups take a better advantage of assistance to appear in court, thus, without the fairness control, all assistance would go to these groups. The fairness costs described in Section 2.1 force the learner to perform some tradeoff between spending all assistance on most reactive subgroups and spending it equally among subgroups.
Outline of this appendix. We first recall the experimental setting of Chohlas-Wood et al. [2021] in particular, how contexts, rewards, and costs are generated in their simulations (we cannot replicate their study on real data, which is not accessible). We then specify the strategies we implemented and how we tuned them—this includes describing the estimation procedure discussed in Section 2.2. We finally report the performance observed, in terms of (average) rewards and costs.
G.1 The experimental setting of Chohlas-Wood et al. [2021]
We follow strictly the experimental setting of Chohlas-Wood et al. [2021] as provided in the public repository https://github.com/stanford-policylab/learning-to-be-fair. This experimental setting, as reverse-engineered from the code, deals with purely simulated data and seems actually inconsistent with the description made in Chohlas-Wood et al. [2021, Section 5.4 and Appendix E], which would anyway rely on proprietary data that could not be made public.
Context generation. Each individual is described by four variables  $\pmb{x}$ : age, proximity, poverty, and group, which are to be read in  $\pmb{x}$  in its components, referred to as  $x_{\mathrm{age}}$ ,  $x_{\mathrm{prox}}$ ,  $x_{\mathrm{pov}}$  and  $\operatorname{gr}(x) = x_{\mathrm{group}}$ . The first three variables are assumed to be normalized and are simulated independently, according to uniform distributions on [0, 1]. Groups are also simulated independently of these three variables: two groups are assumed, with respective probabilities of  $1/2$ . This defines the context distribution  $\nu$ .
As we describe now, assistance has stronger impact on group 0 than in group 1.
Cost generation. We recall that three actions are available: offering some rideshare assistance (action  $a_{\mathrm{ride}}$ ), providing a transportation voucher (action  $a_{\mathrm{voucher}}$ ), or providing no help (which is a control situation: action  $a_{\mathrm{control}}$ ). The associated spending costs are deterministic and do not depend on  $x$ , and there are two separate budgets for rideshares and vouchers. More precisely, at round  $t$  upon taking action  $a_t$ , the following deterministic spending costs are suffered:
$$
c _ {\text {r i d e}} \left(\boldsymbol {x} _ {t}, a _ {t}\right) = \mathbb {1} _ {\left\{a _ {t} = a _ {\text {r i d e}} \right\}} \quad \text {a n d} \quad c _ {\text {v o u c h e r}} \left(\boldsymbol {x} _ {t}, a _ {t}\right) = \mathbb {1} _ {\left\{a _ {t} = a _ {\text {v o u c h e r}} \right\}},
$$
where the corresponding average budgets are  $B_{\mathrm{ride}} = 0.05$  and  $B_{\mathrm{voucher}} = 0.20$ .
We measure the extent of unfair allocation of spending among groups with the following eight fairness costs: a first series of four fairness costs is given by
$$
2 \mathbb {1} _ {\left\{a _ {t} = a _ {\text {r i d e}} \right\}} \mathbb {1} _ {\left\{\mathrm {g r} \left(\boldsymbol {x} _ {t}\right) = 0 \right\}} - \mathbb {1} _ {\left\{a _ {t} = a _ {\text {r i d e}} \right\}}, \quad 2 \mathbb {1} _ {\left\{a _ {t} = a _ {\text {r i d e}} \right\}} \mathbb {1} _ {\left\{\mathrm {g r} \left(\boldsymbol {x} _ {t}\right) = 1 \right\}} - \mathbb {1} _ {\left\{a _ {t} = a _ {\text {r i d e}} \right\}},
$$
$$
2 \mathbb {1} _ {\left\{a _ {t} = a _ {\text {v o u c h e r}} \right\}} \mathbb {1} _ {\left\{\operatorname {g r} \left(\boldsymbol {x} _ {t}\right) = 0 \right\}} - \mathbb {1} _ {\left\{a _ {t} = a _ {\text {v o u c h e r}} \right\}}, \quad 2 \mathbb {1} _ {\left\{a _ {t} = a _ {\text {v o u c h e r}} \right\}} \mathbb {1} _ {\left\{\operatorname {g r} \left(\boldsymbol {x} _ {t}\right) = 1 \right\}} - \mathbb {1} _ {\left\{a _ {t} = a _ {\text {v o u c h e r}} \right\}},
$$
and the second series is given by the opposite values of these costs. We denote by  $\tau$  the corresponding average cost constraints  $\tau$ , and set  $\tau = 10^{-7}$  or  $\tau = 0.025$  in our experiments.
All in all, the global (spending and fairness) vector costs  $c_{t}$  takes deterministic values in  $\mathbb{R}^{10}$ . The vector cost function  $c$  is fully known to the learner, and no estimation is needed.
Reward generation. The rewards are binary:  $r_t = 1$  if the  $t$ -individual appeared in court, and  $r_t = 0$  otherwise. That is, the expected reward equals the probability of appearance. A logistic regression model is assumed: denoting by  $\Phi(u) = 1 / (1 + \mathrm{e}^u)$ , we assume
$$
r (\boldsymbol {x}, a _ {\text {c o n t r o l}}) = \Phi (- x _ {\text {a g e}}),
$$
$$
r (\boldsymbol {x}, a _ {\text {v o u c h e r}}) = \Phi (- x _ {\text {a g e}} + 2 x _ {\text {p r o x}}) \mathbb {1} _ {\{\operatorname {g r} (\boldsymbol {x}) = 0 \}} + \Phi (- x _ {\text {a g e}} + x _ {\text {p r o x}}) \mathbb {1} _ {\{\operatorname {g r} (\boldsymbol {x}) = 1 \}},
$$
$$
r (\boldsymbol {x}, a _ {\text {r i d e}}) = \Phi (- x _ {\text {a g e}} + 4 x _ {\text {p o v}}) \mathbb {1} _ {\left\{\operatorname {g r} (\boldsymbol {x}) = 0 \right\}} + \Phi (- x _ {\text {a g e}} + 2 x _ {\text {p o v}}) \mathbb {1} _ {\left\{\operatorname {g r} (\boldsymbol {x}) = 1 \right\}}.
$$
We may write these six equalities in a compact format as:
$$
r (\boldsymbol {x}, a) = \Phi (\varphi (\boldsymbol {x}, a) ^ {\mathrm {T}} \mu_ {\star})
$$
$$
\text {w h e r e} \quad \varphi (\boldsymbol {x}, a) = \left[ \begin{array}{l} x _ {\text {a g e}} \\ x _ {\text {p r o x}}   \mathbb {1} _ {\{a = a _ {\text {v o u c h e r}} \}} \\ x _ {\text {p r o x}}   \mathbb {1} _ {\{a = a _ {\text {v o u c h e r}} \}}   \mathbb {1} _ {\{\operatorname {g r} (\boldsymbol {x}) = 0 \}} \\ x _ {\text {p o v}}   \mathbb {1} _ {\{a = a _ {\text {r i d e}} \}} \\ x _ {\text {p o v}}   \mathbb {1} _ {\{a = a _ {\text {r i d e}} \}}   \mathbb {1} _ {\{\operatorname {g r} (\boldsymbol {x}) = 0 \}} \end{array} \right] \quad \text {a n d} \quad \mu_ {\star} = \left[ \begin{array}{c} - 1 \\ 1 \\ 1 \\ 2 \\ 2 \end{array} \right].
$$
The learner ignores the coefficients  $\mu_{\star}$  of this structure and only knows that expected rewards are of the form  $\Phi (\varphi (\pmb {x},a)^{\mathrm{T}}\pmb {\theta})$  for some parameter  $\pmb {\theta}\in \mathbb{R}^5$ . The learner will estimate the reward function  $r$  by estimating  $\mu_{\star}$ .
Reward estimation. We deal with a logistic model and follow the methodology described in Li and Stoltz [2022]; see Modeling 2 in Section 2.2. In particular, the parameters  $\mu_{\star}$  are estimated, after each round  $t \geqslant 1$ , thanks to the maximum likelihood estimator
$$
\widehat {\boldsymbol {\mu}} _ {t} \in \underset {\boldsymbol {\mu} \in \mathbb {R} ^ {5}} {\arg \max } \sum_ {s = 1} ^ {t} r _ {s} \Phi (\boldsymbol {\varphi} (a _ {s}, \boldsymbol {x} _ {s}) ^ {\mathrm {T}} \boldsymbol {\mu}) + (1 - r _ {s}) \ln \left(1 - \Phi (\boldsymbol {\varphi} (a _ {s}, \boldsymbol {x} _ {s}) ^ {\mathrm {T}} \boldsymbol {\mu})\right) - \frac {\lambda^ {\text {l o g i s t i c}}}{2} \| \boldsymbol {\mu} \| ^ {2},
$$
where  $\lambda^{\mathrm{logistic}}\geqslant 0$  is a regularization factor. We define as  $\widehat{r}_t(\pmb {x},a) = \Phi \bigl (\varphi (\pmb {x},a)^\mathrm{T}\widehat{\pmb{\mu}}_t\bigr)$  the estimated expected rewards, and the associated uniform estimation errors (see Assumption 2) are of the form
$$
\varepsilon_ {t} (\boldsymbol {x}, a, \delta) = C _ {\delta} \left(1 + \ln t\right) \sqrt {\varphi (a , \boldsymbol {x}) ^ {\mathrm {T}} V _ {t} ^ {- 1} \varphi (a , \boldsymbol {x})}
$$
$$
\text {w h e r e} \quad V _ {t} = \sum_ {s = 1} ^ {t} \varphi \left(a _ {s}, \boldsymbol {x} _ {s}\right) \varphi \left(a _ {s}, \boldsymbol {x} _ {s}\right) ^ {\mathrm {T}} + \lambda^ {\text {l o g i s t i c}} \mathrm {I} _ {5},
$$
where  $\mathrm{I}_5$  is the  $5\times 5$  identity matrix.
In our simulations, we tested a range of values and picked (in hindsight) the well-performing values  $C_{\delta} = 0.025$  and  $\lambda^{\mathrm{logistic}} = 0$ .
G.2 Strategies implemented in this numerical study
We run all these strategies not with the total-cost constraints
$$
\boldsymbol {B} = (0. 0 5, 0. 2, \tau , \tau , \tau , \tau), \quad \text {w h e r e} \quad \tau \in \{1 0 ^ {- 7}, 0. 0 2 5 \},
$$
but take a margin  $b = 0.005$  and use  $B' = B - (b, b, 0, 0, 0, 0)$  instead. This is a slightly different way of taking some margin on the average cost constraint: we do so because we are not aiming for a strict respect of the fairness constraints but rather want to report the level of violation on it. Again, we tried a range of values for  $b$  (between 0.001 to 0.01) and this value of 0.005 led to a good balance between (lack of) total-cost constraint violations and rewards.
Performance of optimal static policies. We use  $\mathrm{OPT}(r, c, B)$  as the benchmark in the definition of regret; our methodology also reveals that  $\mathrm{OPT}(r, c, B')$  is another benchmark, see the discussion in Section 4. We report both values on our graphs. To compute them, we proceed as follows, e.g., for  $B$ . As computing directly the minimum stated in (4) is difficult, even when fully knowing the distribution  $\nu$ , we compute 100 estimates
$$
\mathrm {O P T} ^ {(j)} (r, \boldsymbol {c}, \boldsymbol {B}), \qquad \text {w h i c h w e a v e r a g e o u t i n t o} \qquad \widehat {\mathrm {O P T}} (r, \boldsymbol {c}, \boldsymbol {B}) = \frac {1}{1 0 0} \sum_ {j = 1} ^ {1 0 0} \mathrm {O P T} ^ {(j)} (r, \boldsymbol {c}, \boldsymbol {B}).
$$
For each  $j$ , we sample  $S = 10,000$  contexts from the distribution  $\nu$ , and denote by  $\widehat{\nu}_S^{(j)}$  the associated empirical distribution; we then solve numerically the problem (4) with  $\nu$  replaced by this empirical distribution:
$$
\mathrm {O P T} ^ {(j)} (r, \boldsymbol {c}, \boldsymbol {B}) = \min  _ {\boldsymbol {\lambda} \geqslant 0} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {S} ^ {(j)}} \left[ \max  _ {a \in \mathcal {A}} \left\{r (\boldsymbol {X}, a) - \left\langle \boldsymbol {c} (\boldsymbol {X}, a) - \boldsymbol {B}, \boldsymbol {\lambda} \right\rangle \right\} \right].
$$
Mixed policy knowing  $\lambda_{B'}^{\star}$  but estimating  $r$ . For the sake of illustration, we report the performance of a policy that would have oracle knowledge of  $\lambda_{B'}^{\star}$ , which is a finite-dimensional parameter that summarizes  $\nu$ , but would ignore  $r$ , i.e., the underlying logistic model. That is, this mixed policy would pick, at each round,
$$
\max _ {a \in \mathcal {A}} \left\{\widehat {r} _ {\delta , t - 1} ^ {\mathrm {u c b}} (\boldsymbol {x} _ {t}, a) - \left\langle \widehat {\boldsymbol {c}} _ {\delta , t - 1} ^ {\mathrm {l c b}} (\boldsymbol {x} _ {t}, a) - \boldsymbol {B} ^ {\prime}, \boldsymbol {\lambda} _ {\boldsymbol {B} ^ {\prime}} ^ {\star} \right\rangle \right\},
$$
(and would omit the  $\lambda$  update in Box B).
To compute (an approximation of)  $\lambda_{B'}^*$ , we proceed 100 times as described below to compute estimates
$$
\lambda_ {B ^ {\prime}} ^ {\star , (j)}, \qquad \text {w h i c h w e a v e r a g e o u t i n t o} \qquad \widehat {\lambda} _ {B ^ {\prime}} ^ {\star} = \frac {1}{1 0 0} \sum_ {j = 1} ^ {1 0 0} \lambda_ {B ^ {\prime}} ^ {\star , (j)},
$$
where we noted that the numerical values obtained for the  $\lambda_{B'}^{\star,(j)}$  are rather similar. With the notation above, for each  $j$ , we sample  $S = 10,000$  contexts from the distribution  $\nu$ , and solve
$$
\boldsymbol {\lambda} _ {\boldsymbol {B} ^ {\prime}} ^ {\star , (j)} \in \underset {\boldsymbol {\lambda} \geqslant \boldsymbol {0}} {\arg \min } \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {S} ^ {(j)}} \left[ \max  _ {a \in \mathcal {A}} \Big \{r (\boldsymbol {X}, a) - \big <   \boldsymbol {c} (\boldsymbol {X}, a) - \boldsymbol {B}, \boldsymbol {\lambda} \big > \Big \} \right].
$$
(These estimations are independent from the estimations used to compute the OPT values, i.e., we use different seeds.)
$\mathbf{PGD}\gamma$ . We refer to the Box B strategy as PGD  $\gamma$ , for which we report the performance for values  $\gamma \in \{0.01, 0.02, 0.04, 0.05, 0.1\}$ .
We also implemented the Box C strategy, with an alternative value of  $M_{T,\delta,k}$ , given that the one exhibited based on the theoretical analysis was too conservative, so that no regime break occurred. We resort to a value of the same order of magnitude:
$$
M _ {T, \delta , k} ^ {\prime} = c d \sqrt {T \ln (T (k + 2))},
$$
for a numerical constant  $c$  that we set to 0.01 in our simulations. We call this strategy PGD Adaptive in Figure 1 and Table 1.
G.3 Outcomes of the simulations
We take  $T = 10,000$  individuals (instead of  $T = 1,000$  as in the code by Chohlas-Wood et al., 2021) and set initial 50 rounds as a warm start for strategies (mostly because of the logistic estimation). We were limited by the computational power (see Appendix G.4) and could only perform  $N = 100$  simulations for each (instance of each) strategy. We report averages (strong lines in the graphs) as well as  $\pm 2$  times standard errors (shaded areas in the graphs or values in parentheses in the table).
Graphs. In the first line of graphs in Figure 1, we report the average rewards (over the  $N$  runs) of the strategy under scrutiny as a function of the sample size. More precisely, with obvious notation, we plot
$$
t \mapsto \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\frac {1}{t} \sum_ {s = 1} ^ {t} r _ {s} ^ {(n)}\right).
$$
We report the values of  $\mathrm{OPT}(r, c, B)$  and  $\mathrm{OPT}(r, c, B')$  as dashed horizontal lines.
In the second and third lines of graphs in Figure 1, we report the average costs suffered; again, with obvious notation, we plot
$$
t \mapsto \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\frac {1}{t} \sum_ {s = 1} ^ {t} \mathbb {1} _ {\{a _ {s} ^ {(n)} = a _ {\text {r i d e}} \}}\right) \quad \text {a n d} \quad t \mapsto \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\frac {1}{t} \sum_ {s = 1} ^ {t} \mathbb {1} _ {\{a _ {s} ^ {(n)} = a _ {\text {v o u c h e r}} \}}\right).
$$
We include the average budget constraints  $B_{\mathrm{ride}} = 0.05$  and  $B_{\mathrm{voucher}} = 0.20$  as dashed horizontal lines.
Finally, the fourth line of the graphs in Figure 1 reports the fairness costs; we average their absolute values and draw, again with obvious notation,
$$
t \mapsto \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\frac {1}{4} \sum_ {a \in \{a _ {\text {r i d e}}, a _ {\text {v o u c h e r}} \}} \sum_ {g \in \{0, 1 \}} \left| \frac {1}{t} \sum_ {s = 1} ^ {t} \left(2 \mathbb {1} _ {\{a _ {s} ^ {(n)} = a \}} \mathbb {1} _ {\{\operatorname {g r} (\boldsymbol {x} _ {s} ^ {(n)}) = g \}} - \mathbb {1} _ {\{a _ {s} ^ {(n)} = a \}}\right) \right|\right).
$$
We include the fairness tolerance  $\tau$  as a dashed horizontal line.
Table. We also report the performance of the strategies at the final round  $T$  in following table,with  $\pm  2$  times standard errors in parentheses. We note that the performance of the mixed policy is poor,in particular in terms of fairness costs, which is why we omitted it on the graphs,to keep them readable.
Comments. When  $\gamma$  is well set, i.e., large enough (see the bound of Lemma 1), the PGD strategies control spending costs thanks to targeting  $B^{\prime}$  instead of  $B$ . We observe that for  $\gamma = 0.01$ , the PGD strategy does not control the rideshare costs, but for all larger values of  $\gamma$ , the associated strategies control the three costs considered. The average rewards achieved are coherent with the average spendings: the smaller the average spendings, the smaller the average rewards. There is some lag: the strategies tuned with  $\gamma$  parameters in  $\{0.04, 0.05, 0.1\}$  could use costly actions more. We also observe that fairness costs remain under the target limits.
The mixed policy does not success in controlling the costs, in particular, the fairness costs. While the optimal dual variables  $\lambda_{B'}^{\star}$  summarize the distribution  $\nu$ , it seems that  $\lambda_{B'}^{\star}$  has to be used with care: only with the exact values  $r$  and  $c$ , and not with estimates. On the contrary, the PGD strategies of Box B are more stable, as the dual variables are learned based also on the estimated reward and cost functions.
Figure 1: Performance of the PGD strategies.
Fairness Tolerance  $= 1\mathrm{e} - 7$
Fairness Tolerance  $= 0.025$
Table 1: Performance of the strategies as computed at the final round  $T$  .
Average rewards	Rideshare costs	Voucher costs	Fairness costs
Fairness tolerance τ = 10-7
OPT(r, c, B)	0.4688 (0.0002)
OPT(r, c, B')	0.4648 (0.0002)
PGD γ = 0.01	0.4651 (0.0002)	0.0519 (<0.0001)	0.1984 (<0.0001)	0.0006 (<0.0001)
PGD γ = 0.02	0.4613 (0.0002)	0.0492 (<0.0001)	0.1967 (0.0004)	0.0004 (<0.0001)
PGD γ = 0.04	0.4571 (0.0002)	0.0479 (<0.0001)	0.1962 (0.0002)	0.0004 (<0.0001)
PGD γ = 0.05	0.4554 (0.0002)	0.0476 (<0.0001)	0.1961 (0.0002)	0.0003 (<0.0001)
PGD γ = 0.1	0.4502 (0.0002)	0.0471 (<0.0001)	0.196 (0.0002)	0.0003 (<0.0001)
PGD Adaptive	0.4581 (0.0002)	0.0498 (0.0002)	0.1971 (0.0002)	0.0005 (<0.0001)
Mixed Policy	0.4402 (0.0056)	0.0499 (0.0058)	0.1056 (0.017)	0.0411 (0.0052)
Fairness tolerance τ = 0.025
OPT(r, c, B)	0.4731 (0.0002)
OPT(r, c, B')	0.4691 (0.0002)
PGD γ = 0.01	0.4698 (0.0002)	0.0518 (0.0002)	0.1983 (<0.0001)	0.0246 (0.0002)
PGD γ = 0.02	0.4663 (0.0002)	0.0492 (<0.0001)	0.1966 (0.0006)	0.0242 (0.0002)
PGD γ = 0.04	0.4621 (0.0004)	0.0478 (<0.0001)	0.1958 (0.001)	0.0223 (0.0004)
PGD γ = 0.05	0.4604 (0.0004)	0.0476 (<0.0001)	0.1955 (0.0014)	0.0208 (0.0004)
PGD γ = 0.1	0.4538 (0.0002)	0.0471 (<0.0001)	0.1958 (0.0004)	0.0128 (0.0004)
PGD Adaptive	0.4634 (0.0002)	0.0499 (0.0002)	0.1972 (0.0002)	0.0228 (0.0002)
Mixed Policy	0.4466 (0.0054)	0.0566 (0.0052)	0.1053 (0.0164)	0.0473 (0.0054)
Finally, we note that in our experiments, the regimes in PGD Adaptive strategy typically covered range from  $k = 0$  (corresponding to  $\gamma_0 = 1 / \sqrt{T} = 0.01$ ) to  $k = 2$  (corresponding to  $\gamma_2 = 2^2 / \sqrt{T} = 0.04$ ). The PGD Adaptive strategy performs well and is (only) outperformed by the PDG strategy with a fixed  $\gamma = 0.02$  (of course difficult to pick in advance). In particular, the PGD Adaptive strategy controls costs, and does so by switching to larger step sizes when needed.
G.4 Computation time and environment
As requested by the NeurIPS checklist, we provide details on the computation time and environment. Our experiments were ran on the following hardware environment: no GPU was required, CPU is 3.3 GHz 8 Cores with total of 16 threads, and RAM is 32 GB 4800 MHz DDR5. We ran 100 simulations with 10 different seeds on parallel each time. In the setting and for the data described above, the average time spend on each algorithm for a single run was inferior to 10 minutes.