SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathbb{E}_{P,\pi}$  denotes an expectation with respect to the randomness of the trajectory  $\{(x_t,u_t):t\in [T]\}$  obtained by following a policy  $\pi$  and the transition probability  $P$  at each step  $t\in [T]$ . The Bellman optimality equations can then be expressed as
$$
Q _ {t} ^ {\star} (x, u) = \left(c _ {t} + \mathbb {P} _ {t} V _ {t + 1} ^ {\star}\right) (x, u), \quad V _ {t} ^ {\star} (x) = \inf  _ {v \in \mathcal {U}} Q _ {t} ^ {\star} (x, v), \quad V _ {t} ^ {\star} (x) = 0 \tag {3}
$$
for all  $(x,u)\in \mathcal{X}\times \mathcal{U},t\in [T]$  and  $t\in [T]$ , where we write  $(\mathbb{P}_tV^{\star})(x,u)\coloneqq \mathbb{E}_{x'\sim P_t(\cdot |x,u)}[V^{\star}(x')]$ . This indicates that for each time step  $t\in [T]$ ,  $\pi_t^\star$  is the greedy policy with respect to its optimal Q-value functions  $(Q_{t}^{\star}:t\in [T])$ . Note that for any  $t\in [T]$ ,  $Q_{t}^{\star}(x,u) = 0$ . Given this setup, the value-based policies  $\widetilde{\pi}\coloneqq (\widetilde{\pi}_t:t\in [T])$  take the following form. For any  $t\in [T]$ , a value-based policy  $\widetilde{\pi}_t:\mathcal{X}\to \mathcal{U}$  produces an action  $\widetilde{u}_t\in \arg \min_{v\in \mathcal{U}}\widetilde{Q}_t(x_t,v)$  by minimizing an estimate of the optimal Q-value function  $\tilde{Q}_t$ .
We make the following assumption on the machine-learned untrusted policy  $\widetilde{\pi}$  and the Q-value advice.
Assumption 3. The machine-learned untrusted policy  $\widetilde{\pi}$  is value-based. The  $Q$ -value advice  $\widetilde{Q}_t: \mathcal{X} \times \mathcal{U} \to \mathbb{R}$  is Lipschitz continuous with respect to  $u \in \mathcal{U}$  for any  $x \in \mathcal{X}$ , with a Lipschitz constant  $L_Q$  for all  $t \in [T]$ . Moreover,  $\widetilde{Q}_t(x,u) - Q_t^\star(x,u) = o(T)$  for all  $(x,u) \in \mathcal{X} \times \mathcal{U}$  and  $t \in [T]$ .
We can now define a consistency measure for Q-value advice  $\widetilde{Q}_t$ , which measures the error of the estimates of the Q-value functions due to approximation error and time-varying environments, etc. Let  $p \in (0, \infty]$ . Fix a sequence of distributions  $\rho = (\rho_t : t \in [T])$  whose support set is  $\mathcal{X} \times \mathcal{U}$  and let  $\phi_t$  be the marginal distribution of  $\rho_t$  on  $\mathcal{X}$ . We define a quantity representing the error of the Q-value
$$
\varepsilon (p, \rho) := \sum_ {t \in [ T ]} \left(\left\| \widetilde {Q} _ {t} - Q _ {t} ^ {\star} \right\| _ {p, \rho_ {t}} + \left\| \inf  _ {v \in \mathcal {U}} \widetilde {Q} _ {t} - \inf  _ {v \in \mathcal {U}} Q _ {t} ^ {\star} \right\| _ {p, \phi_ {t}}\right) \tag {4}
$$
where  $\| \cdot \|_{p,\rho} \coloneqq \left( \int |\cdot|^p \, \mathrm{d}\rho \right)^{1/p}$  denotes the  $L_{p,\rho}$ -norm. A policy with Q-value functions  $\{Q_t : t \in [T]\}$  is said to be  $(\varepsilon, p, \rho)$ -consistent if there exists an  $\varepsilon$  satisfying (4). In addition, a policy is  $(0, \infty)$ -consistent if  $\widetilde{Q}_t$  is a Lebesgue-measurable function for all  $t \in [T]$  and  $(\infty, \varepsilon)$ -consistent if the  $L_{\infty}$ -norm satisfies  $\sum_{t \in [T]} \| \widetilde{Q}_t - Q_t^\star \|_{\infty} \leq \varepsilon$ . The consistency error of a policy in (4) quantifies how the Q-value advice is close to optimal Q-value functions. It depends on various factors such as the function approximation error or training error due to the distribution shift, and has a close connection to a rich literature on value function approximation [57-61]. The results in [59] generalized the worst-case  $L_{\infty}$  guarantees to arbitrary  $L_{p,\rho}$ -norms under some mixing assumptions via policy iteration for a stationary Markov decision process (MDP) with a continuous state space and a discrete action space. Recently, approximation guarantees for the average case for parametric policy classes (such as a neural network) of value functions have started to appear [57, 58, 60]. These bounds are useful in lots of supervised machine learning methods such as classification and regression, whose bounds are typically given on the expected error under some distribution. These results exemplify richer instances of the consistency definition (see (4)) and a summary of these bounds can be found in [61].
Now, we are ready to introduce our definition of consistency and robustness with respect to the ratio of expectations, similar to the growing literature on learning-augmented algorithms [3, 22, 23, 4]. We write the ratio of expectations  $\mathrm{RoE}(\varepsilon)$  of a policy  $\pi$  as a function of the Q-value advice error  $\varepsilon$  in terms of the  $L_{\infty}$  norm, defined in (4).
Definition 4 (Consistency and Robustness). An algorithm  $\pi$  is said to be  $k$ -consistent if its worst-case (with respect to the MDP model  $\mathsf{MDP}(\mathcal{X},\mathcal{U},T,P,c)$ ) ratio of expectations satisfies  $\mathsf{RoE}(\varepsilon) \leq k$  for  $\varepsilon = 0$ . On the other hand, it is  $l$ -robust if  $\mathsf{RoE}(\varepsilon) \leq l$  for any  $\varepsilon > 0$ .
4 The Projection Pursuit Policy (PROP)
In this section we introduce our proposed algorithm (Algorithm 1), which achieves near-optimal consistency while bounding the robustness by leveraging a robust baseline (Section 3.1) in combination with value-based advice (Section 3.2). A key challenge in the design is how to exploit the benefits of good value-based advice while avoiding following it too closely when it performs poorly. To address this challenge, we propose to judiciously project the value-based advice into a neighborhood of the robust baseline. By doing so, the actions we choose can follow the value-based advice for consistency while staying close to the robust baseline for robustness. More specifically, at each step  $t \in [T]$ , we choose  $u_{t} = \mathrm{Proj}_{\overline{\mathcal{U}}_{t}}(\widetilde{u}_{t})$  where a projection operator  $\mathrm{Proj}_{\overline{\mathcal{U}}_t}(\cdot):\mathcal{U}\to \mathcal{U}$  is defined as
$$
\operatorname {P r o j} _ {\overline {{\mathcal {U}}} _ {t}} (u) := \underset {v \in \mathcal {U}} {\arg \min } \| u - v \| _ {\mathcal {U}} \text {s u b j e c t} \quad \| v - \bar {\pi} _ {t} \left(x _ {t}\right) \| _ {\mathcal {U}} \leq R _ {t}, \tag {5}
$$
corresponding to the projection of  $u$  onto a ball  $\overline{\mathcal{U}}_t \coloneqq \{u \in \mathcal{U} : \| u - \overline{\pi}_t(x_t) \|_{\mathcal{U}} \leq R_t\}$ . Note that when the optimal solution of (5) is not unique, we choose the one on the same line with  $\overline{\pi}_t(x_t) - u$ .
The PROjection Pursuit policy, abbreviated as PROP, can be described as follows. For a time step  $t \in [T]$ , let  $\widetilde{\pi}_t: \mathcal{X} \to \mathcal{U}$  denote a policy that chooses an action  $\widetilde{u}_t$  (arbitrarily choose one if there are multiple minimizers of  $\widetilde{Q}_t$ ), given the current system state  $x_t$  at time  $t \in [T]$  and step  $t \in [T]$ . An action  $u_t = \mathrm{Proj}_{\overline{\mathcal{U}}_t}(\widetilde{u}_t(x_t))$  is selected by projecting the machine-learned action  $\widetilde{u}_t(x_t)$  onto a norm ball  $\overline{\mathcal{U}}_t$  defined by the robust policy  $\overline{\pi}$  given a radius  $R_t \geq 0$ . Finally, PROP applies to both black-box and grey-box settings (which differ from each other in terms of how the radius  $R_t$  is decided). The results under both settings are provided in Section 5, revealing a tradeoff between consistency and robustness.
The radii  $(R_{t}: t \in [T])$  can be interpreted as robustness budgets and are key design parameters that determine the consistency and robustness tradeoff. Intuitively, the robustness budgets reflect the trustworthiness on the value-based policy  $\widetilde{\pi}$  — the larger budgets, the more trustworthiness and hence the more freedom for PROP to follow  $\widetilde{\pi}$ . How the robustness budget is chosen differentiates the grey-box setting from the black-box one.
Algorithm 1 PROjection Pursuit Policy (PROP)
Procedure 1 BLACK-BOX Procedure at  $t\in [T]$  (Input: state  $x_{t}$  and hyper-parameter  $0\leq \lambda \leq 1$
Initialize: Untrusted policy  $\widetilde{\pi} = (\widetilde{\pi}_t : t \in [T])$  and baseline policy  $\overline{\pi} = (\overline{\pi}_t : t \in [T])$   
1 for  $t = 0, \ldots, T - 1$  do  
2 // Implement black-box (Section 4.1) or grey-box (Section 4.2) procedures  
3  $(\widetilde{u}_t, R_t) \gets \text{BLACK-BOX}(x_t)$  or  $(\widetilde{u}_t, R_t) \gets \text{GREY-BOX}(x_t)$   
4 Set action  $u_t = \text{Proj}_{\overline{\mathcal{U}}_t}(\widetilde{u}_t)$  where  $\overline{\mathcal{U}}_t := \{u \in \mathcal{U} : \|u - \overline{\pi}_t(x_t)\|_{\mathcal{U}} \leq R_t\}$   
5 Sample next state  $x_{t+1} \sim P_t(\cdot | x_t, u_t)$   
6 end
4.1 Black-Box Setting
In the black-box setting, the only information provided by  $\widetilde{\pi}$  is a suggested action  $\widetilde{u}$  for the learning-augmented algorithm. Meanwhile, the robust policy  $\overline{\pi}$  can also be queried to provide advice  $\overline{u}$ . Thus, without additional information, a natural way to utilize both  $\widetilde{\pi}$  and  $\overline{\pi}$  is to decide a projection radius at each time based on the how the obtained  $\widetilde{u}$  and  $\overline{u}$ . More concretely, at each time  $t\in [T]$ , the robustness budget  $R_{t}$  is chosen by the following BLACK-BOX Procedure, where we set  $R_{t} = \lambda \eta_{t}$  with  $\eta_t\coloneqq \| \widetilde{u}_t - \overline{u}_t\|_{\mathcal{U}}$  representing the difference between the two advice measured in terms of the norm  $\| \cdot \|_{\mathcal{U}}$  and  $0\leq \lambda \leq 1$  being a tradeoff hyper-parameter that measures the trustworthiness on the machine-learned advice. The choice of  $R_{t} = \lambda \eta_{t}$  can be explained as follows. The value of  $\eta_t$  indicates the intrinsic discrepancy between the robust advice and the machine-learned untrusted advice — the larger discrepancy, the more difficult to achieve good consistency and robustness simultaneously. Given a robust policy and an untrusted policy, by setting a larger  $\lambda$ , we allow the actual action to deviate more from the robust advice and to follow the untrusted advice more closely, and vice versa.  $\lambda$  is a crucial hyper-parameter that can be pre-determined to yield a desired consistency and robustness tradeoff. The computation of  $R_{t}$  is summarized in Procedure below.
Implement  $\widetilde{\pi}_t$  and  $\overline{\pi}_t$  to obtain  $\widetilde{u}_t$  and  $\overline{u}_t$ , respectively.
Set robustness budget  $R_{t} = \lambda \eta_{t}$  where  $\eta_t\coloneqq \| \widetilde{u}_t - \overline{u}_t\|_{\mathcal{U}}$  ; Return  $(\widetilde{u}_t,R_t)$
4.2 Grey-Box Setting
In the grey-box setting, along with the suggested action  $\widetilde{u}$ , the value-based untrusted policy  $\widetilde{\pi}$  also provides an estimate of the Q-value function  $\widetilde{Q}$  that indicates the long-term cost impact of an action. To utilize such additional information informed by  $\widetilde{Q}_t$  at each time  $t\in [T]$ , we propose a novel algorithm that dynamically adjusts the budget  $R_{t}$  to further improve the consistency and robustness tradeoff. More concretely, let us consider the Temporal-Difference (TD) error  $\mathsf{T}\mathsf{D}_t = c_{t - 1} + \mathbb{P}_{t - 1}\widetilde{V}_t - \widetilde{Q}_{t - 1}$ . Intuitively, if a non-zero TD-error is observed, the budget  $R_{t}$  needs to be decreased so as to minimize the impact of the learning error. However, the exact TD-error is difficult to compute in practice, since it requires complete knowledge of the transition kernels  $(P_t:t\in [T])$ . To address this challenge, we use the following estimated TD-error based on previous trajectories:
$$
\delta_ {t} \left(x _ {t}, x _ {t - 1}, u _ {t - 1}\right) := c _ {t - 1} \left(x _ {t - 1}, u _ {t - 1}\right) + \inf  _ {v \in \mathcal {U}} \widetilde {Q} _ {t} \left(x _ {t}, v\right) - \widetilde {Q} _ {t - 1} \left(x _ {t - 1}, u _ {t - 1}\right). \tag {6}
$$
Denote by  $\beta > 0$  a hyper-parameter. Based on the estimated TD-error in (6), the robustness budget in Algorithm 1 is set as
$$
R _ {t} := \left[ \underbrace {\| \widetilde {\pi} _ {t} (x _ {t}) - \bar {\pi} _ {t} (x _ {t}) \| _ {\mathcal {U}}} _ {\text {D e c i s i o n D i s c r e p a n c y} \eta_ {t}} - \frac {\beta}{L _ {Q}} \sum_ {s = 1} ^ {t} \underbrace {\delta_ {s} (x _ {s} , x _ {s - 1} , u _ {s - 1})} _ {\text {A p p r o x i m a t e T D - E r r o r}} \right] ^ {+}, \tag {7}
$$
which constitutes two terms. The first term  $\eta_t \coloneqq \|\widetilde{\pi}_t(x_t) - \overline{\pi}_t(x_t)\|$  measures the decision discrepancy between the untrusted policy  $\widetilde{\pi}$  and the baseline policy  $\overline{\pi}$ , which normalizes the total budget, similar to the one used in the black-box setting in Procedure 1. The second term is the approximate TD-error, which is normalized by the Lipschitz constant  $L_Q$  of Q-value functions. With these terms defined, the GREY-BOX Procedure below first chooses a suggested action  $\widetilde{u}_t$  by minimizing  $\widetilde{Q}_t$  and then decides a robustness budget  $R_t$  using ⑦.
Procedure 2 GREY-BOX Procedure at  $t\in [T]$  (Input: state  $x_{t}$  and hyper-parameter  $0\leq \beta \leq 1$
Obtain advice  $\widetilde{Q}_t$  and  $\widetilde{u}_t$  where  $\widetilde{u}_t\in \arg \inf_{v\in \mathcal{U}}\widetilde{Q}_t(x_t,v)$
Implement  $\overline{\pi}_t$  and obtain  $\overline{u}_t$
Set robustness budget  $R_{t}$  as [7]; Return  $(\widetilde{u}_t,R_t)$
We now formally present the main results for both the black-box and grey-box settings. Our results not only quantify the tradeoffs between consistency and robustness formally stated in Definition 4 with respect to the ratio of expectations, but also emphasize a crucial role that additional information about the estimated Q-values plays toward improving the consistency and robustness tradeoff.
5.1 Black-Box Setting
In the existing learning-augmented algorithms, the untrusted machine-learned policy  $\widetilde{\pi}$  is often treated as a black-box that generates action advice  $\widetilde{u}_t$  at each time  $t\in [T]$ . Our first result is the following general dynamic regret bound for the black-box setting (Section 4.1). We utilize the Big-O notation, denoted as  $\mathcal{O}(\cdot)$  and  $o(\cdot)$  to disregard inessential constants.
Theorem 5.1. Suppose the machine-learned policy  $\widetilde{\pi}$  is  $(\infty, \varepsilon)$ -consistent. For any MDP model satisfying Assumption 12, and 3 the expected dynamic regret of PROP with the BLACK-BOX Procedure is bounded by DR(PROP)  $\leq \min \{\mathcal{O}(\varepsilon) + \mathcal{O}((1 - \lambda)\gamma T), \mathcal{O}((\mathsf{ROB} + \lambda\gamma - 1)T)\}$  where  $\varepsilon$  is defined in (4),  $\gamma$  is the diameter of the action space  $\mathcal{U}$ ,  $T$  is the length of the time horizon, ROB is the ratio of expectations of the robust baseline  $\overline{\pi}$ , and  $0 \leq \lambda \leq 1$  is a hyper-parameter.
When  $\lambda$  increases, the actual action can deviate more from the robust policy, making the dynamic regret potentially closer to that of the value-based policy. While the regret bound in Theorem 5.1 clearly shows the role of  $\lambda$  in terms of controlling how closely we follow the robust policy, the dynamic regret given a fixed  $\lambda \in [0,1]$  grows linearly in  $\mathcal{O}(T)$ . In fact, the linear growth of dynamic regret holds even if the black-box policy  $\widetilde{\pi}$  is consistent, i.e.,  $\varepsilon$  is small. This can be explained by noting the lack of dynamically tuning  $\lambda$  to follow the better of the two policies — even when one policy is nearly perfect, the actual action still always deviates from it due to the fixed choice of  $\lambda$ .
Consider any MDP model satisfying Assumptions 1, 2, and 3. Following the classic definitions of consistency and robustness (see Definition 4), we summarize the following characterization of PROP, together with a negative result in Theorem 5.3. Proofs of Theorem 5.1, 5.2, and 5.3 are detailed in Appendix C
Theorem 5.2 (BLACK-BOX Consistency and Robustness). PROP with the BLACK-BOX Procedure is  $(1 + \mathcal{O}((1 - \lambda)\gamma))$ -consistent and  $(\mathsf{ROB} + \mathcal{O}(\lambda \gamma))$ -robust where  $0 \leq \lambda \leq 1$  is a hyper-parameter.
Theorem 5.3 (BLACK-BOX Impossibility). PROP with the BLACK-BOX Procedure cannot be both  $(1 + o((1 - \lambda)\gamma))$ -consistent and  $(\mathsf{ROB} + o(\lambda \gamma))$ -robust for any  $0 \leq \lambda \leq 1$ .
5.2 Grey-Box Setting
To overcome the impossibility result in the black-box setting, we dynamically tune the robustness budgets by tapping into additional information informed by the estimated Q-value functions using the GREY-BOX Procedure (Section 4.2). By setting the robustness budgets in (7), an analogous result of Theorem 5.1 is given in Appendix D, which leads to a dynamic regret bound of PROP in the grey-box setting (Theorem D.1 in Appendix D). Consider any MDP model satisfying Assumptions 1, 2, and 3. Our main result below indicates that knowing more structural information about a black-box policy can indeed bring additional benefits in terms of the consistency and robustness tradeoff, even if the black-box policy is untrusted.
Theorem 5.4 (GREY-BOX Consistency and Robustness). PROP with the GREY-BOX Procedure is 1-consistent and  $(\mathsf{ROB} + o(1))$  -robust for some  $\beta >0$
Theorem 5.3 implies that using the BLACK-BOX Procedure, PROP cannot be 1-consistent and  $(\mathsf{ROB} + o(1))$ -robust, while this can be achieved using the GREY-BOX Procedure. On one hand, this theorem validates the effectiveness of the PROP policy with value-based machine-learned advice that
may not be fully trusted. On the other hand, this sharp contrast between the black-box and grey-box settings reveals that having access to information of value function can improve the tradeoff between consistency and robustness (see Definition 4) non-trivially. A proof of Theorem 5.4 can be found in Appendix D. Applications of our main results are discussed in Appendix A.
Our results contribute to the growing body of literature on learning-augmented algorithms for MDPs and highlight the importance of considering consistency and robustness in this context. In particular, we have shown that by utilizing the structural information of machine learning methods, it is possible to achieve improved performance over a black-box approach. The results demonstrate the potential benefits of utilizing value-based policies as advice; however, there remains room for future work in exploring other forms of structural information.
Limitations and Future Work. One limitation of our current work is the lack of analysis of more general forms of black-box procedures. Understanding and quantifying the available structural information in a more systematic way is another future direction that could lead to advances in the design of learning-augmented online algorithms and their applications in various domains.
We would like to thank the anonymous reviewers for their helpful comments. This work was supported in part by the National Natural Science Foundation of China (NSFC) under grant No. 72301234, the Guangdong Key Lab of Mathematical Foundations for Artificial Intelligence, and the start-up funding UDF01002773 of CUHK-Shenzhen. Yiheng Lin was supported by the Caltech Kortschak Scholars program. Shaolei Ren was supported in part by the U.S. U.S. National Science Foundation (NSF) under grant CNS-1910208. Adam Wierman was supported in part by the U.S. NSF under grants CNS-2146814, CPS-2136197, CNS-2106403, NGSDI-2105648.
[1] Michael Mitzenmacher and Sergei Vassilvitskii. Algorithms with predictions. Communications of the ACM, 65(7):33-35, 2022.
[2] Tongxin Li. Learning-Augmented Control and Decision-Making: Theory and Applications in Smart Grids. PhD thesis, California Institute of Technology, 2023.
[3] Manish Purohit, Zoya Svitkina, and Ravi Kumar. Improving online algorithms via ml predictions. Advances in Neural Information Processing Systems, 31, 2018.
[4] Nicolas Christianson, Tinashe Handina, and Adam Wierman. Chasing convex bodies and functions with black-box advice. In Conference on Learning Theory, pages 867-908. PMLR, 2022.
[5] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30, 2017.
[6] Zhuoran Yang, Chi Jin, Zhaoran Wang, Mengdi Wang, and Michael I Jordan. On function approximation in reinforcement learning: optimism in the face of large state spaces. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 13903-13916, 2020.
[7] Noah Golowich and Ankur Moitra. Can q-learning be improved with advice? In Conference on Learning Theory, pages 4548-4619. PMLR, 2022.
[8] Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Conference on Learning Theory, pages 4300-4354. PMLR, 2021.
[61] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res., 22(98):1-76, 2021.
[62] Yiheng Lin, James A Preiss, Emile Timothy Anand, Yingying Li, Yisong Yue, and Adam Wierman. Online adaptive policy selection in time-varying systems: No-regret via contractive perturbations. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[63] Runyu Zhang, Yingying Li, and Na Li. On the regret analysis of online LQR control with predictions. In 2021 American Control Conference (ACC), pages 697-703. IEEE, 2021.
[64] James R Norris. Markov chains. Cambridge university press, 1998.
[65] Steve Lalley. Markov chains: Basic theory. https://galton.uchicago.edu/~lalley/Courses/312/MarkovChains.pdf, 2016.
[66] Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems, 31, 2018.
[67] Yingying Li and Na Li. Online learning for markov decision processes in nonstationary environments: A dynamic regret analysis. In 2019 American Control Conference (ACC), pages 1232-1237. IEEE, 2019.
[68] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[69] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. Pmlr, 2014.
[70] Leonid Kantorovitch. On the translocation of masses. Management science, 5(1):1-4, 1958.
[71] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: no regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pages 1015-1022, 2010.