SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} f _ {t} \left(p _ {T _ {0}} ^ {*}\right) - f _ {t} \left(p ^ {*}\right) \\ = T \cdot \left[ T \cdot V _ {0} - T \cdot V _ {\mathrm {N}} \right] \\ = T \cdot \left[ \left(S (1) ^ {2} \left\{\frac {1}{p _ {T _ {0}} ^ {*}} - 1 \right\} + S (0) ^ {2} \left\{\frac {1}{1 - p _ {T _ {0}} ^ {*}} - 1 \right\}\right) - \left(S (1) ^ {2} \left\{\frac {1}{p ^ {*}} - 1 \right\} + S (0) ^ {2} \left\{\frac {1}{1 - p ^ {*}} - 1 \right\}\right) \right] \\ = T \cdot \left[ S (1) ^ {2} \left(\frac {1}{p _ {T _ {0}} ^ {*}} - \frac {1}{p ^ {*}}\right) + S (0) ^ {2} \left(\frac {1}{1 - p _ {T _ {0}} ^ {*}} - \frac {1}{1 - p ^ {*}}\right) \right] \\ \end{array}
$$
$$
= T \cdot \left[ S (1) ^ {2} \Big (\frac {S (0)}{S (1)} - \frac {S _ {T _ {0}} (0)}{S _ {T _ {0}} (1)} \Big) + S (0) ^ {2} \Big (\frac {S (1)}{S (0)} - \frac {S _ {T _ {0}} (1)}{S _ {T _ {0}} (0)} \Big) \right],
$$
where the last equality follows by definition of the probabilities. By Assumption, we have that this bracketed term is constant so that the third term is linear in  $T$ .
Putting these together, we have that the Neyman regret is lower bounded as
$$
\mathcal {R} _ {T} \geqslant \Omega (T) - \mathcal {O} _ {p} (T ^ {1 - \epsilon / 2}) - \mathcal {O} _ {p} (T ^ {1 / 2}) = \Omega_ {p} (T).
$$
E.2 Analysis of Designs for Outcome Regret (Proposition 6.2)
In this section, we prove Proposition 6.2, which establishes that outcome regret and Neyman regret cannot be simultaneously minimized in general. We restate a more formal version of the proposition here. In order for a simpler proof, we make restrictions that the units have constant treatment effect and that each of the individual outcomes are more strictly bounded. We conjecture that the trade-off between Neyman and outcome regret will hold under weaker conditions.
Proposition 6.2*. Let  $\mathcal{A}$  be an adaptive treatment algorithm achieving sublinear outcome regret, i.e. there exists  $q\in (0,1)$  such that  $\mathbb{E}[\mathcal{R}_T^{outcme}] \leqslant O(T^q)$  for all outcome sequences satisfying Assumption 1. Consider an outcome sequence satisfying Assumption 1 with constants  $C\geq c > 0$  and the additional conditions:
-  $\max_{1\leqslant t\leqslant T}y_t(0)^2\leqslant C^2$
- For all  $t \in [T]$ ,  $y_t(1) - y_t(0) = \tau$  and  $\tau > c'$  for a constant  $c' > 0$ .
Then,  $\mathcal{A}$  suffers super-linear Neyman regret on this outcome sequence:  $\mathbb{E}[\mathcal{R}_T] \geqslant \Omega (T^{2 - q})$ .
Proof. To begin, we re-express the outcome regret in terms of the expected treatment probabilities played by algorithm  $\mathcal{A}$ . Observe that the expected outcome regret may be written as
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {R} _ {T} ^ {\text {o u t c o m e}} \right] = \mathbb {E} \left[ \max  _ {k \in \{0, 1 \}} \sum_ {t = 1} ^ {T} y _ {t} (k) - \sum_ {t = 1} ^ {T} Y _ {t} \right] \quad (\text {d e f o f r e g r e t}) \\ = \sum_ {t = 1} ^ {T} y _ {t} (1) - \sum_ {t = 1} ^ {T} \mathbb {E} [ Y _ {t} ] \quad (\tau > 0) \\ = \sum_ {t = 1} ^ {T} y _ {t} (1) - \sum_ {t = 1} ^ {T} \mathbb {E} \left[ y _ {t} (1) \mathbf {1} \left[ Z _ {t} = 1 \right] + y _ {t} (0) \mathbf {1} \left[ Z _ {t} = 0 \right] \right] \\ = \sum_ {t = 1} ^ {T} y _ {t} (1) - \sum_ {t = 1} ^ {T} y _ {t} (1) \mathbb {E} [ p _ {t} ] + y _ {t} (0) \cdot \left(1 - \mathbb {E} [ p _ {t} ]\right) \\ = \sum_ {t = 1} ^ {T} \left(y _ {t} (1) - y _ {t} (0)\right) \cdot \left(1 - \mathbb {E} [ p _ {t} ]\right) \\ = \tau \cdot \sum_ {t = 1} ^ {T} \left(1 - \mathbb {E} \left[ p _ {t} \right]\right), \\ \end{array}
$$
where the last equality follows as  $y_{t}(1) - y_{t}(0) = \tau$  for all  $t\in [T]$  by assumption. Because the outcome sequence satisfies Assumption 1, the expected outcome regret is at most  $\mathbb{E}[\mathcal{R}_T^{\mathrm{outcome}}]\leqslant \beta \cdot T^q$  for some constant  $\beta$ . By the above, this implies that the expectation of the sum of probabilities  $1 - p_t$  must be small,
$$
\sum_ {t = 1} ^ {T} \left(1 - \mathbb {E} [ p _ {t} ]\right) \leqslant \frac {\beta}{\tau} T ^ {q}.
$$
Next, we show that  $\mathcal{A}$  must incur a large cost with respect to the functions  $f_{t}$  in the definition of Neyman regret. To do this, we will use a weighted version of the AM-HM inequality which states that for  $x_{1}\ldots x_{T} > 0$  and  $w_{1}\ldots w_{n}\geqslant 0$ , we have that
$$
\frac {\sum_ {t = 1} ^ {T} w _ {t} x _ {t}}{\sum_ {t = 1} ^ {T} w _ {t}} \geqslant \frac {\sum_ {t = 1} ^ {T} w _ {t}}{\sum_ {t = 1} ^ {T} \frac {w _ {t}}{x _ {t}}} .
$$
The usual AM-HM inequality is recovered when  $w_{t} = 1 / T$ . We now bound the expected Neyman loss, observing that
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} f _ {t} \left(p _ {t}\right) \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \frac {y _ {t} (1) ^ {2}}{p _ {t}} + \frac {y _ {t} (0) ^ {2}}{1 - p _ {t}} \right] \quad (\text {d e f o f} f _ {t}) \\ \geqslant \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \frac {y _ {t} (0) ^ {2}}{1 - p _ {t}} \right] \quad (\text {n o n - n e g a t i v i t y}) \\ = \sum_ {t = 1} ^ {T} y _ {t} (0) ^ {2} \cdot \mathbb {E} \left[ \frac {1}{1 - p _ {t}} \right] \quad (\text {l i n e a r i t y o f} \mathbb {E} [ \cdot ]) \\ \geqslant \sum_ {t = 1} ^ {T} y _ {t} (0) ^ {2} \cdot \frac {1}{\mathbb {E} \left[ 1 - p _ {t} \right]} \quad (\text {J e s s e n ' s i n e q u a l i t y}) \\ \geqslant \frac {\left(\sum_ {t = 1} ^ {T} y _ {t} (0) ^ {2}\right) ^ {2}}{\sum_ {t = 1} ^ {T} y _ {t} (0) ^ {2} \cdot \mathbb {E} \left[ 1 - p _ {t} \right]} \quad (\text {w e i g h t e d A M - H M}) \\ \geqslant T ^ {2} \frac {\left(\frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} (0) ^ {2}\right) ^ {2}}{\max  _ {1 \leqslant t \leqslant T} y _ {t} (0) ^ {2}} \cdot \frac {1}{\sum_ {t = 1} ^ {T} \mathbb {E} [ 1 - p _ {t} ]} \\ = T ^ {2} \frac {S (0) ^ {2}}{\max  _ {1 \leqslant t \leqslant T} y _ {t} (0) ^ {2}} \cdot \frac {1}{\sum_ {t = 1} ^ {T} \mathbb {E} [ 1 - p _ {t} ]} \\ \geqslant T ^ {2} \frac {c ^ {2}}{C ^ {2}} \cdot \frac {\tau}{\beta} T ^ {- q} \\ \geqslant \frac {c ^ {2} c ^ {\prime}}{C ^ {2} \beta} T ^ {2 - q}, \\ \end{array}
$$
where the last two inequalities follow from moment bounds in Assumption 1 together with the assumptions on the outcome sequence stated in the Theorem.
Next, we show that the optimal Neyman design incurs a much smaller cost. In particular,
$$
\begin{array}{l} \min  _ {p \in [ 0, 1 ]} \sum_ {t = 1} ^ {T} f _ {t} (p) \leqslant \sum_ {t = 1} ^ {T} f _ {t} (1 / 2) \\ = \sum_ {t = 1} ^ {T} \frac {y _ {t} (1) ^ {2}}{1 / 2} + \frac {y _ {t} (0) ^ {2}}{1 / 2} \\ = 2 T \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} (1) ^ {2} + \frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} (0) ^ {2} \right] \\ = 2 T \left(S (1) ^ {2} + S (0) ^ {2}\right) \\ \leqslant 4 C ^ {2} T. \\ \end{array}
$$
Together, these facts establish that the Neyman regret for  $\mathcal{A}$  is lower bounded as
$$
\mathbb {E} \left[ \mathcal {R} _ {T} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} f _ {t} \left(p _ {t}\right) - \min  _ {p \in [ 0, 1 ]} \sum_ {t = 1} ^ {T} f _ {t} (p) \right] \geqslant \frac {c ^ {2} c ^ {\prime}}{C ^ {2} \beta} T ^ {2 - q} - 4 C ^ {2} T \geqslant \Omega \left(T ^ {2 - q}\right).
$$
F Ethical Considerations
There are—at least—two objectives when constructing an adaptive treatment allocation.
- Minimizing Cumulative Regret: give the "best" treatment to as many people as possible.
- Minimizing Variance of the Effect Estimate: estimate the effect of the treatment to as high precision as possible.
As we show in Proposition 6.2, these two objective are fundamentally incompatible: an adaptive design which aims to estimate the effect to high precision must assign treatment which has worse outcomes. Likewise, a design which seeks to maximize the utility of assigned treatments will not be able to reliably estimate causal effects to high precision. Which one of these is more ethically desirable depends on the purpose and the context of the experiment. For guidance on this ethical question, we turn to The Belmont Report.
In 1979, the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research released the "Belmont Report", which has been one of the foundational texts for ethical guidance in research conducted with human subjects [for the Protection of Human Subjects of Biomedical and Research, 1978]. One of the three basic ethical principles laid out in the report is "benevolence" which is understood as an obligation the researcher has for the research to improve the general well-being of society, including those people in the experiment. The report writes:
The Hippocratic maxim 'do no harm' has long been a fundamental principle of medical ethics. Claude Bernard extended it to the realm of research, saying that one should not injure one person regardless of the benefits that might come to others. However, even avoiding harm requires learning what is harmful; and, in the process of obtaining this information, persons may be exposed to risk of harm. Further, the Hippocratic Oath requires physicians to benefit their patients 'according to their best judgment.' Learning what will in fact benefit may require exposing persons to risk. The problem posed by these imperatives is to decide when it is justifiable to seek certain benefits despite the risks involved, and when the benefits should be foregone because of the risks.
From this perspective, it may be ethically advisable to use a variance minimizing adaptive design because it allows the researcher to learn the effect while subjecting fewer human subjects to the experimental treatments. In other words, a variance minimizing design allows researchers to learn what is harmful and what is beneficial while subjecting fewer human subjects to possible harm. An adaptive treatment plan which minimizes cumulative regret will ensure that minimal harm is done to subjects in the experiment, but will offer less certainty about the extent of the benefit or harm of the treatments. Such an approach will lead to less informative generalizable knowledge of treatment effects, possibly defeating the goal of the research study.
That being said, it is not our goal to suggest that one adaptive allocation plan is most ethical in all circumstances. Indeed, these ethical questions have no systematic answers which are generally applicable. It is the burden of the researchers to carefully "decide when it is justifiable to seek certain benefits despite the risks involved, and when the benefits should be foregone because of the risks." Our goal in this work is merely to provide improved statistical methodology which affords the researchers more choices when addressing these ethical questions.