SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this case, the exponents  $\alpha_{1}$  and  $\alpha_{2}$  denote the convergence rates of the estimated behavior policy and value function, respectively. Note that the first term is the leading term, while the last two terms represent the estimation errors of the nuisance function estimators and converge at a rate much faster than  $(n - 2m_0)^{-1}$ . The error bound above is minimized when  $m_0$  is zero. In such a scenario, the bound is asymptotically equal to  $n^{-1}\mathrm{EB}_1(\pi^{b*})$ , which is equivalent to the MSE of an oracle ATE estimator. Specifically, the oracle estimator is a version of the DR estimator with correctly specified value function and behavior policy, and is constructed based on data collected over  $n$  days from the optimal design  $\pi^{b*}$ . Thus, the proposed estimator is asymptotically optimal.
Next, we show how to construct the confidence interval of the ATE estimator. A key observation is that, under certain regularity conditions, the proposed estimator is asymptotically normal. More specifically, similar to Kallus and Uehara (2020), we have
$$
\sqrt {n - 2 m _ {0}} \left(\widehat {\mathrm {A T E}} _ {1} - \mathrm {A T E}\right) \xrightarrow {d} N \left(0, \mathrm {E B} _ {1} \left(\pi^ {b *}\right)\right).
$$
This motivates us to consider the following Wald-type confidence interval
$$
[ \widehat {\mathrm {A T E}} _ {1} - \Phi^ {- 1} (1 - \alpha / 2) \sqrt {\mathrm {E B} _ {1} (\pi^ {b *}) / (n - 2 m _ {0})}, \widehat {\mathrm {A T E}} _ {1} + \Phi^ {- 1} (1 - \alpha / 2) \sqrt {\mathrm {E B} _ {1} (\pi^ {b *}) / (n - 2 m _ {0})} ],
$$
where  $\Phi^{-1}$  is the inverse cumulative distribution function of a standard normal random variable. It then suffices to estimate the asymptotic variance  $\mathrm{EB}_1(\pi^{b*})$  to construct asymptotically valid confidence intervals. Notice that  $\widehat{\mathrm{ATE}}_1$  can be represented as an average of martingale differences  $\widehat{\mathrm{ATE}}_1 = \sum_{i=2m_0+1}^n \psi_i^1 / (n - 2m_0)$  where
$$
\psi_ {i} ^ {1} = \sum_ {a = 0} ^ {1} \frac {(- 1) ^ {a + 1}}{T} \Big [ \widehat {V} _ {1, i - 1} ^ {a} (O _ {1} ^ {(i)}) + \frac {\mathbb {I} (A _ {1} ^ {(i)} = a)}{\widehat {\pi} _ {1 , i - 1} ^ {b *} (a | O _ {1} ^ {(i)})} [ \sum_ {t} R _ {t} ^ {(i)} - \widehat {V} _ {1, i - 1} ^ {a} (O _ {1} ^ {(i)}) ] \Big ].
$$
We propose using the sample variance of  $\{\psi_i^1\}_i$  to estimate  $\mathrm{EB}_1(\pi^{b*})$ . Similar to Theorem 15 of Kallus and Uehara (2022), we can establish the consistency of the sampling variance estimator.
4 Design, Implementation and Evaluation in TMDPs and MDPs
We proceed to study the optimal design and the subsequent ATE estimation in TMDPs.
Design. According to Theorem 2 of Kallus and Uehara (2020), the efficiency bound for the ATE estimator equals
$$
\mathbf {E B} _ {2} \left(\pi^ {b}\right) = \frac {1}{T ^ {2}} \sum_ {t = 1} ^ {T} \sum_ {a \in \{0, 1 \}} \mathbb {E} ^ {\pi^ {b}} \left[ \frac {\mathbb {I} \left(A _ {t} = a\right) p _ {t} ^ {a} \left(O _ {t}\right)}{p _ {t} ^ {b} \left(O _ {t} , a\right)} \sigma_ {t} \left(O _ {t}, a\right) \right] ^ {2} + \frac {1}{T ^ {2}} \operatorname {V a r} \left[ V _ {1} ^ {1} \left(O _ {1}\right) - V _ {1} ^ {0} \left(O _ {1}\right) \right], \tag {5}
$$
where  $p_t^1(\bullet)$  ( $p_t^0(\bullet)$ ) denotes the probability mass/density function of  $O_t$  under the new policy (control),  $p_t^b(\bullet, \bullet)$  denotes the probability mass/density function of the observation-action pair ( $O_t, A_t$ ) at time  $t$  under the behavior policy, and  $\sigma_t^2(O_t, a)$  is the conditional variance of the temporal difference error  $R_t + V_{t+1}^a(O_{t+1}) - V_t^a(O_t)$  given the current observation  $O_t$  and that  $A_t = a$ .
The efficiency bound in (5) bears a striking resemblance to that in (1). The sole difference lies in the replacement of the sequential IS ratio in (1) with the ratio of the marginal observation-action pair
under the Markov assumption. However, in contrast to NMDPs, the marginal distribution function  $p_t^b$  in TMDPs cannot be represented in a closed-form as a function of  $\pi^b$ . This intricate dependence of  $p_t^b$  on  $\pi^b$  makes it exceptionally challenging to identify the optimal  $\pi^{b*}$  that minimizes (5). To elaborate, let  $\Pi^b$  represent the class of behavior policies that randomly assign the initial action, and set  $\Pi^b = \{\pi^b : \pi_2^b(A_1|H_2) = \pi_3^b(A_1|H_3) = \dots = \pi_T^b(A_1|H_T) = 1\}$ . In Theorem 1, we show that the optimal behavior policy  $\pi^{b*}$  belongs to  $\Pi^b$  in NMDPs. However, this is not the case in TMDPs without additional assumptions.
Proposition 1 (Informal Statement). There exists a TMDP such that  $\pi^{b*} \notin \Pi^b$ .
Identifying the exact optimal design remains challenging. Motivated by the simplicity of the policy class  $\Pi^b$ , we shift our focus to finding the optimal in-class behavior policy within  $\Pi^b$  that minimizes (5). We restrict the treatment assignment policies to  $\Pi^b$  for two primary reasons. First, the optimal design for NMDP resides in class  $\Pi^b$ , leveraging this result allows us to anticipate that in-class behavior policies from  $\Pi^b$  will perform well in TMDPs/MDPs, which are subclasses of NMDP; Second, frequently alternating treatments can introduce significant carryover bias in policy evaluation (Hu and Wager, 2022). By focusing on  $\Pi^b$ , we avoid distributional shifts between the behavior policy and the target policy, effectively mitigating carryover bias. To further simplify the analysis, we consider scenarios where the number of decision stages  $T$  approaches infinity. The following theorem provides the form of an optimal  $\pi^{b*} \in \Pi^b$  that asymptotically minimizes (5) under a mild  $\beta$ -mixing condition. Additionally, it shows that  $\pi^{b*}$  is optimal among all history-dependent policies under certain constancy conditions.
Theorem 3. Suppose the  $\beta$ -mixing condition holds such that  $\lim_{t\to \infty}\mathbb{E}\sup_{a,o}|p_t^a (o|O_1) - p_t^a (o)|\to$  0 where  $p_t^a (\bullet |O_1)$  denotes the probability mass function of  $O_{t}$  given  $O_{1}$  following the action  $a$ . Then an asymptotically optimal in-class behavior policy  $\pi^{b*}$  satisfies (i) for any  $a\in \{0,1\}$ ,
$$
\pi_ {1} ^ {b *} (a | O _ {1}) = \pi_ {1} ^ {b *} (a) = \frac {\sigma_ {a *}}{\sigma_ {1 *} + \sigma_ {0 *}}, \text {w h e r e} \sigma_ {a *} ^ {2} = \sum_ {t = 1} ^ {T} \mathbb {E} ^ {a} \left[ \sigma_ {t} ^ {2} \left(O _ {t}, 1\right) \right]; \tag {6}
$$
(ii)  $\pi_2^{b*}(A_1|H_2) = \pi_3^{b*}(A_1|H_3) = \dots = \pi_T^{b*}(A_1|H_T) = 1$  almost surely. In other words, for  $\pi^{b*}$  that satisfies (i) and (ii), we have  $\lim_{T} T[EB_2(\pi^{b*}) - EB_2(\pi^b)] \leq 0$  for any  $\pi^b \in \Pi^b$ . Additionally, suppose both  $\sigma_t^2(o,1)$  and  $\sigma_t^2(o,0)$  are constant as functions of  $o$  and  $t$ . Then  $\lim_{T} T[EB_2(\pi^{b*}) - EB_2(\pi^b)] \leq 0$  for any  $\pi^b$ .
The  $\beta$ -mixing condition essentially requires the Markov chain to be ergodic. Similar conditions have been imposed in the literature (Bhandari et al., 2018; Zou et al., 2019; Luckett et al., 2020; Kallus and Uehara, 2022; Li et al., 2022; Shi et al., 2022, 2023). It is also weaker than the independence assumption that requires the transition tuples to be independent over time and has been frequently imposed in the literature. The constancy condition is automatically satisfied when the temporal difference error is independent of the current observation and the time step. Different from the optimal design in NMDPs, the initial policy in (6) is independent of the initial observation, thanks to the mixing condition.
Implementation. We summarize the procedure in Algorithm 2.
Evaluation. Similar to Section 3, we can apply value-based, IS-based, or doubly robust estimator to learn the ATE from the collected data. Consider the double reinforcement learning estimator (DRL, Kallus and Uehara, 2020) as an example. A key observation is that the marginal observation-action probability distribution function  $p_t^b (O_t,A_t)$  under  $\pi^{b*}$  equals  $\pi_1^* (A_t)p_t^{A_t}(O_t)$ . As such, the resulting marginal ratio  $\mathbb{I}(A_t = a)p_t^a (O_t) / p_t^b (O_t,a)$  is equal to  $\mathbb{I}(A_t = a) / \pi_1^* (A_t)$ , or equivalently  $\mathbb{I}(A_1 = a) / \pi_1^* (A_1)$ , independent of  $O_{t}$ . Consequently, one can show the resulting DRL is reduced to
$$
\widehat {\mathrm {A T E}} _ {2} = \sum_ {a = 0} ^ {1} \frac {(- 1) ^ {a + 1}}{T (n - 2 m _ {0})} \sum_ {i = 2 m _ {0} + 1} ^ {n} \left\{\widehat {V} _ {1, i - 1} ^ {a} (O _ {1} ^ {(i)}) + \frac {\mathbb {I} (A _ {1} ^ {(i)} = a)}{\widehat {\pi} _ {1 , i - 1} ^ {b *} (a)} [ \sum_ {t} R _ {t} ^ {(i)} - \widehat {V} _ {1, i - 1} ^ {a} (O _ {1} ^ {(i)}) ] \right\},
$$
where  $\widehat{V}_{1,i}^{a}$  and  $\widehat{\pi}_1^{b*}(a)$  denotes the estimated value function and estimated initial allocation probability using data from the first  $i$ th days; see Step 3 of Algorithm 2 for the estimation procedure. Similar to  $\widehat{\mathrm{ATE}}_1$  in (4),  $\widehat{\mathrm{ATE}}_2$  can be updated in an online manner without storing historical data. We provide an upper bound for the MSE of  $\widehat{\mathrm{ATE}}_2$  below.
Algorithm 2 Treatment allocation algorithm for TMDPs
1: Run each global policy for  $m_0$  days and obtain  $\{O_t^{(i)}, A_t^{(i)}, R_t^{(i)}\}_{i=1}^{2m_0}$ .
2: while  $2m_0 < m \leq n$  do
3: Obtain  $\widehat{V}_{t,m-1}^{a}$  by regressing  $\{\sum_{j=t}^{T} R_{t}^{(i)} : i < m, A_{1}^{(i)} = a\}$  on  $\{O_{t}^{(i)} : i < m, A_{1}^{(a)} = a\}$ .
4: For  $a\in \{0,1\}$  , set
$$
\widehat {\sigma} _ {a *} ^ {2} = \frac {\sum_ {i <   m} \sum_ {t = 1} ^ {T} [ R _ {t} ^ {(i)} + \widehat {V} _ {t + 1 , i} ^ {a} (O _ {t + 1} ^ {(i)}) - \widehat {V} _ {t , i} ^ {a} (O _ {t} ^ {(i)}) ] ^ {2} \mathbb {I} (A _ {1} ^ {(i)} = a)}{\sum_ {i <   m} \mathbb {I} (A _ {1} ^ {(i)} = a)}.
$$
5: Assign  $A_1^{(m)}$  according to  $\widehat{\pi}_{1,m-1}^{b*}(a) = \widehat{\sigma}_{a*} / (\widehat{\sigma}_{1*} + \widehat{\sigma}_{0*})$ .
6: Set  $A_2^{(m)} = \dots = A_T^{(m)} = A_1^{(m)}$ .
7: end while
Theorem 4. Suppose that  $\min_i\widehat{\pi}_{1,i}^{b*}\geq \epsilon$  and  $V_{1,i}^{a}\leq TR_{\max}$  for some constants  $\epsilon >0$  and  $R_{\mathrm{max}} < \infty$ ,  $\max_{a,i}\mathbb{E}|1 / \widehat{\pi}_{1,i}^{b*}(a) - 1 / \pi_1^{b*}(a)|_2^2\leq Ci^{-2\alpha_1}$  and  $\max_{a,i}T^{-2}\mathbb{E}|\widehat{V}_{1,i}^{a}(O_{1}) - V_{1,i}^{a}(O_{1})|_{2}^{2}\leq Ci^{-2\alpha_{2}}$  for some constants  $C < \infty, 0 < \alpha_{1}, \alpha_{2} < 1/2$ . Then we have
$$
\mathbb {E} (\widehat {A T E} _ {2} - A T E) ^ {2} \leq \frac {E B _ {2} (\pi^ {b *})}{n - 2 m _ {0}} + O (C \epsilon^ {- 1} (n - 2 m _ {0}) ^ {- 1 - 2 \alpha_ {2}}) + O (\sqrt {C} R _ {\max} ^ {2} (n - m _ {0}) ^ {- 1 - \alpha_ {1}}).
$$
Similar to Theorem 2, the MSE of the proposed ATE estimator relies on  $m_0$ , the estimated behavior policy and value function. The first term is asymptotically equal to  $n^{-1}\mathrm{EB}_2(\pi^{b*})$  when  $m_0$  is much smaller than  $n$ . This suggests that the proposed estimator is asymptotically optimal in TDMPs.
Analogous to the procedures in Section 3, we can similarly establish the asymptotic normality of  $\widehat{\mathrm{ATE}}_2$ , i.e.,  $\sqrt{n - 2m_0} (\widehat{\mathrm{ATE}}_2 - \mathrm{ATE}) \xrightarrow{d} N(0,\mathrm{EB}_2(\pi^{b*}))$ . The corresponding  $1 - \alpha$  confidence interval can be constructed by
$$
[ \widehat {\mathrm {A T E}} _ {2} - \Phi^ {- 1} (1 - \alpha / 2) \sqrt {\mathrm {E B} _ {2} (\pi^ {b *}) / (n - 2 m _ {0})}, \widehat {\mathrm {A T E}} _ {2} + \Phi^ {- 1} (1 - \alpha / 2) \sqrt {\mathrm {E B} _ {2} (\pi^ {b *}) / (n - 2 m _ {0})} ],
$$
where the unknown asymptotic variance  $\mathrm{EB}_2(\pi^{b*})$  can be similarly estimated via the sampling variance estimator.
MDPs. Finally, we consider MDPs. To save space, we briefly introduce our proposal here and relegate the technical details to the supplementary article. The proposed design is built upon the efficiency bound developed by Liao et al. (2022) in the average reward infinite horizon setting. To simplify the analysis, similar to our proposal in TMDPs, we focus on in-class optimal designs and consider an asymptotic regime where  $T \to \infty$ . The following theorem summarizes the properties of the proposed behavior policy  $\tilde{\pi}^{b*} \in \Pi^b$ .
Theorem 5 (Information Statement). Suppose the  $\beta$ -mixing condition holds. Then  $\tilde{\pi}^{b*}$  asymptotically minimizes the efficiency bound among all  $\pi^b \in \Pi^b$ . In addition, under a constancy condition, it is asymptotically optimal among all policies.
To learn the ATE, we construct an online doubly robust estimator in the supplementary article based on the estimated relative value functions (Puterman, 2014) and the designed behavior policy.
This section presents four experiments designed to evaluate different designs. The environments include a tabular example with binary observation variables, an example with continuous observations, a small-scale synthetic dispatch example, and a city-scale real data-based dispatch simulator. We investigate the proposed treatment allocation strategies designed for NMDPs and MDPs. We also implement the following three allocation designs for comparison:
Figure 2: Boxplots of the MSEs of the allocation methods for  $p_s = 0.8$  and  $T = 50$  in Example 5.1: the four panels correspond to  $\delta = 0, 3, 6$ , and 9, respectively.
(i) Random: This uniform random treatment allocation design assumes  $\mathbb{P}(A_{i,t} = 1) = 1 / 2$ . All  $A_{i,t}$ s are independent.
(ii) Half-half: This design applies the global treatment for the first  $n/2$  days and the global control for the remaining days.
(iii) Greedy: This design uses the  $\epsilon$ -greedy algorithms that select the current-best treatment, which maximizes the Q-function, with probability  $1 - \epsilon$ , and employs a uniform random policy with probability  $\epsilon$ .
We note that (iii) is widely used in the RL literature for online regret minimization. We compare the mean squared errors (MSEs) of the estimates of the average treatment effect based on 200 replicates, and present the results subsequently. We set the burn-in period  $m_0$  to  $n / 4$  in all the experiments. Detailed information on the data-generating processes can be found in Section S2 of the supplementary material.
Example 5.1 (Binary Observations). In this example, we consider binary observation variables. To better understand the results, we introduce some hyper-parameters that describe the system dynamics. Let  $p_s$  denote the marginal probability that the future observation  $O_{t+1}$  equals 1 if  $A_t = 1$ , and it is  $1 - p_s$  if  $A_t = 0$ . Additionally, we use  $\delta \in \{0,3,6,9\}$  to characterize the difference of the conditional variance of the reward between the treatment and the control. A larger value of  $\delta$  indicates a greater difference.
Figure 2 presents boxplots of the MSEs for the allocation methods when  $p_s = 0.8$ ,  $T = 50$ , and  $n = 50$ . It is evident that the proposed treatment allocation methods generally outperform the alternatives, offering lower median MSE and smaller variability in most scenarios. However, when  $\delta = 0$ , the half-half design outperforms ours. This outcome is expected since in this case, the optimal allocation rule assigns the initial action with equal probability, i.e.,  $\pi_1^{b*} = 0.5$ . As  $\delta$  increases, our proposed estimator surpasses the others in terms of achieving a smaller MSE. Table 1 in the supplementary material presents the means and standard deviations of the MSEs for  $T \in \{10, 30, 50\}$  and  $p_s \in \{0.5, 0.8\}$ . In these scenarios, our proposed designs consistently outperform the alternatives.
Example 5.2 (Continuous Observations). In this example, we use  $\delta_s$  to denote the difference in the conditional variance of the observation variable between the treatment and the control, and  $\delta$  to describe the difference of the conditional variance of the reward.
Figure 3 presents the boxplots of all MSEs for  $\delta_s = 1$  and  $T = 50$  with  $n = 50$ . Table 2 of the supplementary material includes the Monte Carlo averages and standard errors of MSEs for  $T \in \{10, 30, 50\}$  and  $\delta_s \in \{0, 1\}$ . These results are comparable to those in Example 5.1, demonstrating the superior performance of the proposed design when  $\delta$  is moderately large. This is expected, as the proposed method takes into account the conditional variance of the temporal difference error.
Example 5.3 (Synthetic Dispatch). Following Xu et al. (2018), we construct a small-scale synthetic dispatch environment to estimate the treatment effect of different order dispatch policies. Specifically, we simulate drivers and orders in a  $9 \times 9$  spatial grid with a duration of 20 time steps per day. Orders will be canceled if not being responded for a long time. We compare the MDP order dispatch strategy (Xu et al., 2018) against the distance-based dispatch method which minimizes the total distance between drivers and passengers. The reward of interest is given by the total revenue. The observation variables are set to the number of orders and the number of drivers at each time. The number of days is set to be  $n \in \{30, 50, 100\}$ . All the methods are tested using 100 orders with the number of drivers being either generated from the uniform distribution  $U(25, 30)$ , or being fixed to 25, 50. A detailed description of the environment can be found in the supplementary document.
Figure 3: Boxplots of the MSEs of the allocation methods corresponding to  $\delta_s = 1$  and  $T = 50$  in Example 5.2: the four panels correspond to  $\delta = 0, 3, 6, 9$ , respectively.
Figure 4: Boxplots of the MSEs of the allocation methods in Example 5.3 with  $n = 50$ : the three panels correspond to drivers generated from  $U(25, 30)$  or fixed as 25 and 50, respectively.
Figure 4 presents the boxplots of MSEs for  $n = 50$  with different numbers of drivers. Table 3 of the supplementary material reports the values of MSEs of the four methods. As expected, the MSEs of all the methods decrease as the increase of the number of days  $n$ . The proposed method achieves the best performance in almost all scenarios.
Example 5.4 (Real-Data Based Dispatch). We evaluate the proposed treatment allocation method on a dispatch simulator based on a city-scale order-driver historical dataset from a world-leading ride-sharing platform. The dataset consists of temporal spatial information of orders or drivers and numerical features of them. We adopt the simulator in Tang et al. (2019) to generate data based on the historical dataset. The distributions of drivers and orders are set to be identical to the distributions of historical data. Similar to Example 5.3, the two policies being compared are the MDP strategy and the distance-based dispatch method. The reward of interest is the total Gross Merchandise Volume (GMV), and the observation variables include the number of drivers and the number of orders.
Since the ground truth of the average treatment effect is large, we calculate the relative MSE of the estimated average treatment effect for each method. RMSEs of ATE estimates for distinct allocation designs are reported in Figure 5 with the number of days  $n = 7$  and  $n = 10$ . It can be seen that the proposed method outperforms all its counterparts. Meanwhile, as the episode  $n$  gets larger, the RMSEs of most allocation designs become smaller.
Li's research is partially supported by the National Science Foundation of China 12101388, CCF-DiDi GAIA Collaborative Research Funds for Young Scholars and Program for Innovative Research Team of Shanghai University of Finance and Economics. Shi's research is partially supported by an EPSRC grant EP/W014971/1. Zhou's work is partially supported by National Natural Science Foundation of China 12001356, "Chenguang Program" supported by Shanghai Education Development
Figure 5: Boxplots of the RMSEs of the allocation methods in Example 5.4 with  $n = 7$  and  $n = 10$ .
Foundation and Shanghai Municipal Education Commission, Open Research Projects of Zhejiang Lab NO.2022RC0AB06, Shanghai Research Center for Data Science and Decision Technology. We thank the anonymous referees and the meta reviewer for their constructive comments, which have led to a significant improvement of the earlier version of this article.
Antos, A., C. Szepesvári, and R. Munos (2008). Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning 71, 89-129.
Atkinson, A., A. Donev, and R. Tobias (2007). Optimum experimental designs, with SAS, Volume 34. OUP Oxford.
Atkinson, A. and D. Pedrosa (2017). Optimum design and sequential treatment allocation in an experiment in deep brain stimulation with sets of treatment combinations. Statistics in Medicine 36(30), 4804-4815.
Baird, S., J. A. Bohren, C. McIntosh, and B. Özler (2018). Optimal design of experiments in the presence of interference. Review of Economics and Statistics 100(5), 844-860.
Bajari, P., B. Burdick, G. W. Imbens, L. Masoero, J. McQueen, T. Richardson, and I. M. Rosen (2021). Multiple randomization designs. arXiv preprint arXiv:2112.13495.
Begg, C. B. and B. Iglewicz (1980). A treatment allocation procedure for sequential clinical trials. Biometrics 36(1), 81-90.
Bhandari, J., D. Russo, and R. Singal (2018). A finite time analysis of temporal difference learning with linear function approximation. In Conference on Learning Theory, pp. 1691-1692. PMLR.
Bian, Z., C. Shi, Z. Qi, and L. Wang (2023). Off-policy evaluation in doubly inhomogeneous environments. arXiv preprint arXiv:2306.08719.
Bibaut, A., M. Dimakopoulou, N. Kallus, A. Chambaz, and M. van Der Laan (2021). Post-contextual-bandit inference. Advances in Neural Information Processing Systems 34, 28548-28559.
Bickel, P. J., C. A. Klaassen, P. J. Bickel, Y. Ritov, J. Klaassen, J. A. Wellner, and Y. Ritov (1993). Efficient and adaptive estimation for semiparametric models, Volume 4. Springer.
Blau, T., E. V. Bonilla, I. Chades, and A. Dezfouli (2022). Optimizing sequential experimental design with deep reinforcement learning. In International Conference on Machine Learning, pp. 2107-2128. PMLR.
Bojinov, I. and N. Shephard (2019). Time series experiments and causal estimands: exact randomization tests and trading. Journal of the American Statistical Association 114(528), 1665-1682.
Bojinov, I., D. Simchi-Levi, and J. Zhao (2023). Design and analysis of switchback experiments. Management Science 69(7), 3759-3777.
Chandereng, T., X. Wei, and R. Chappell (2020). Imbalanced randomization in clinical trials. Statistics in Medicine 39(16), 2185-2196.
Chen, X. and Z. Qi (2022). On well-posedness and minimax optimal rates of nonparametric q-function estimation in off-policy evaluation. In International Conference on Machine Learning, pp. 3558-3582. PMLR.
Díaz, I. (2020). Machine learning in the estimation of causal effects: targeted minimum loss-based estimation and double/debiased machine learning. *Biostatistics* 21(2), 353–358.
Dudík, M., D. Erhan, J. Langford, and L. Li (2014). Doubly robust policy evaluation and optimization. Statistical Science 29(4), 485-511.
Efron, B. (1971). Forcing a sequential experiment to be balanced. Biometrika 58(3), 403-417.
Farajtabar, M., Y. Chow, and M. Ghavamzadeh (2018). More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pp. 1447-1456. PMLR.