SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2 COMPILER/COMPILER-E
Input: Pairwise Demonstrations  $\Gamma^{+},\Gamma^{-}$  ; Expert ratio  $\alpha$  (Unknown for COMPILER-E); Environ-   
ment env.   
Output: The learner policy  $\pi_{\theta}$    
1: Initialize the learner policy  $\pi_{\theta}$  and the discriminator  $D_{\omega}$    
2: if  $\alpha$  is unknown then  $\triangleright$  COMPILER-E   
3: Obtain the estimated expert ratio  $\alpha \gets$  ExpertRatioEstimation(  $\Gamma^{+},\Gamma^{-})$    
4: else  $\triangleright$  COMPILER   
5:  $\alpha \gets$  input(a).   
6: end if   
7: for each training steps do   
8: Sample a batch of learner's data  $(s,a)\sim p_{\pi_{\theta}}$  by the interactions between  $\pi_{\theta}$  and env.   
9: Sample a batch of demonstrations data  $(s,a)\sim \widehat{p}_{\pi_+}$  and  $(s,a)\sim \widehat{p}_{\pi_-}$  from  $\Gamma^{+}$  and  $\Gamma^{-}$    
10: Update  $D_{\omega}$  by maximizing (8).   
11: Update  $\pi_{\theta}$  with  $(s,a)\sim p_{\pi_{\theta}}$  and the reward - log  $D_{\omega}(s,a)$  using off-the-shelf RL algorithm.   
12: end for
and its estimation would be highly unstable. Thus, we choose to estimate  $\alpha$  with  $\beta_{1}$  in our algorithm. The empirical studies in Section 6.3 also supported our choice.
The relationship (2) between the expert and non-expert data shares a similar formulation to that of the unlabeled unlabeled (UU) classification [34], which is a specific kind of weakly supervised learning problem studying how to train a classifier from two unlabeled datasets and requires the knowledge of mixture proportions. However, how to estimate these mixture proportions is still an open problem in the weakly supervised learning literature. Although our reduction is developed for estimating the expert ratio for IL problems, it can be applied to a UU learning setting for independent interest.
Reconstruct Expert Policy by Data Selection. To handle the MPE task for high dimensional data, a practice is to first train a classifier with probability output and then conduct the MPE on the probability outputs of samples [35]. However, the quality of the trained classifier turns out to depend on the estimation accuracy of  $\alpha$ . On the other hand, we found that it is possible to filter out the undesired component of the pairwise datasets  $(\widehat{p}_{\pi_E}$  in  $\Gamma^{-}$  and  $\widehat{p}_{\pi_{\mathrm{NE}}}$  in  $\Gamma^{+})$  and keep the desired data to enhance estimating  $\alpha$ . Therefore, also inspired by [31, 36] of the distribution shift problem and weakly supervised learning, we propose an iteration-based learning framework. In each iteration, we throw away the non-expert data with higher confidence in  $\Gamma^{+}$  and the expert data with higher confidence in  $\Gamma^{-}$  after estimating  $\alpha$ , and train a classifier with the datasets after selection. The detailed learning process can be found in Algorithm 1.
After we obtained the estimated  $\widehat{\alpha}$ , we can recover  $p_{\pi_E}$  as by Eq. (10).
5 COMParative Imitation LEarning with Risk rewriting (COMPILER)
We have illustrated how to estimate the occupancy measure of the expert policy  $p_{\pi_E}$  from  $(\Gamma^+, \Gamma^-)$  under known and unknown  $\alpha$  respectively. Here we describe how to adopt these learning frameworks into end-to-end algorithms for learning an policy. We name our algorithm for VPIL with known  $\alpha$  COMParative Imitation LEarning with Risk-rewriting (COMPIER), and with unknown  $\alpha$  COMParative Imitation LEarning with Risk-rewriting by Estimation (COMPIER-E).
Current state-of-the-art adversarial IL methods [37, 1] aim to learn a policy by matching the occupancy measure between the learner and the expert policies. We fuse our methods with one of the representative adversarial IL methods, Generative Adversarial Imitation Learning (GAIL) [1], whose optimization problem is given as follows:
$$
\min  _ {\theta \in \Theta} \max  _ {w \in \mathcal {W}} \mathbb {E} _ {(s, a) \sim p _ {\pi_ {\theta}}} \log D _ {w} (s, a) + \mathbb {E} _ {(s, a) \sim p _ {\pi_ {\mathsf {E}}}} \log (1 - D _ {w} (s, a)),
$$
where  $D_w: S \times \mathcal{A} \mapsto [0,1]$  parameterized by  $w$  is a discriminator.  $p_{\pi_\theta}$  parameterized by  $\theta$  is the occupancy measure of a policy. In our setting, the expert demonstration of  $p_{\pi_{\mathrm{F}}}$  is unavailable.
COMPILER. As suggested by Theorem 2, we can recover  $p_{\pi_E}$  with the occupancy measures of  $\Gamma^+$  and  $\Gamma^-$ . Then, we can train the policy  $p_{\pi_\theta}$  by mincing  $p_{\pi_E} = \frac{1 + \alpha}{2\alpha} p_{\pi_+} - \frac{1 - \alpha}{2\alpha} p_{\pi_-}$  in Eq. (10). Specifically, plugging (2) into (8) and approximating  $p_{\pi_+}$  and  $p_{\pi_-}$  with their empirical versions, we can obtain the desirable target. However, such rewriting can introduce a negative term and lead to overfitting [38, 39]. To avoid such an undesired phenomenon, instead of training the policy  $\pi_\theta$  to make  $p_{\pi_\theta}$  match  $\frac{1 + \alpha}{2\alpha} \widehat{p}_{\pi_+} - \frac{1 - \alpha}{2\alpha} \widehat{p}_{\pi_-}$ , we twist the objective function of GAIL (8) to minimize the discrepancy between  $\frac{2\alpha}{1 + \alpha} p_{\pi_\theta} + \frac{1 - \alpha}{1 + \alpha} \widehat{p}_{\pi_-}$  and  $\widehat{p}_{\pi_+}$ , which gives the following optimization problem without the negative term:
$$
\begin{array}{l} \min  _ {\theta \in \Theta} \max  _ {w \in \mathcal {W}} 2 \alpha \mathbb {E} _ {(s, a) \sim p _ {\pi_ {\theta}}} \log D _ {w} (s, a) + (1 - \alpha) \mathbb {E} _ {(s, a) \sim \widehat {p} _ {\pi_ {-}}} \log (D _ {w} (s, a)) \\ + (1 + \alpha) \mathbb {E} _ {(s, a) \sim \widehat {p} _ {\pi_ {+}}} \log (1 - D _ {w} (s, a)). \tag {8} \\ \end{array}
$$
Let  $\widehat{V} (\pi_{\theta},D_w)$  be the objective function in (8) and  $V(\pi_{\theta},D_w)$  be its expectation established on  $p_{\pi_+}$  and  $p_{\pi_-}$ . We have the following theorem for the estimation error of the discriminator trained by (8).
Theorem 3. Let  $\mathcal{W}$  be a parameter space for training the discriminator and  $D_{\mathcal{W}} = \{D_w \mid w \in \mathcal{W}\}$  be the hypothesis space. Assume the functions  $|\log D_w(s, a)| \leq B$  and  $|\log (1 - D_w(s, a))| \leq B$  are upper-bounded for any state-action pair  $(s, a) \in S \times \mathcal{A}$  and  $w \in \mathcal{W}$ . Further assume both the functions  $\log D_w(s, a)$  and  $\log (1 - D_w(s, a))$  are  $L$ -Lipschitz continuous in the state-action space. For a fixed policy  $\pi_\theta$ , let  $\Gamma^\theta = \{\tau_i^\theta\}_{i=1}^{n_\theta}$  be trajectories generated from  $\pi_\theta$ . Then, for any  $\delta \in (0, 1)$  with probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} V \left(\pi_ {\theta}, D _ {w} ^ {*}\right) - V \left(\pi_ {\theta}, \widehat {D} _ {w}\right) \leq 4 L (1 + \alpha) \mathcal {R} _ {n _ {+}} \left(D _ {\mathcal {W}}\right) + 4 L (1 - \alpha) \mathcal {R} _ {n _ {-}} \left(D _ {\mathcal {W}}\right)) \\ + 8 L \mathcal {R} _ {n _ {\theta}} (D _ {\mathcal {W}}) + C (\delta) \left(\frac {1}{\sqrt {n _ {\theta}}} + \frac {1}{\sqrt {n _ {+}}} + \frac {1}{\sqrt {n _ {-}}}\right), \\ \end{array}
$$
where  $\widehat{D}_w = \arg \max_{w\in \mathcal{W}}\widehat{V} (\pi_\theta ,D_w)$  and  $D_w^* = \arg \max_{w\in \mathcal{W}}V(\pi_\theta ,D_w)$ . The constants  $n_+$  and  $n_-$  are the number of trajectories in  $\Gamma^{+}$  and  $\Gamma^{-}$ . We define  $C(\delta) = 4B\sqrt{\log(6 / \delta)}$ . The empirical Radamacher complexities [40] on datasets  $\Gamma^{+}$ ,  $\Gamma^{-}$ , and  $\Gamma^{\theta}$  are denoted by  $\mathcal{R}_{n_+}$ ,  $\mathcal{R}_{n_-}$ , and  $\mathcal{R}_{n_\theta}$ .
A proof is given in Appendix A. The theorem shows that the discriminator trained by (8) converges to the one optimized with the true distribution  $p_{\pi_+}$  and  $p_{\pi_-}$  at each step of the training.
COMPILER-E. For the VPIL problem with unknown  $\alpha$ , first we estimate  $\widehat{\alpha}$  by Algorithm 1, then we learn the policy by (8) with  $\widehat{\alpha}$ .
We integrate the two algorithmic processes COMPILER and COMPILER-E into Algorithm 2.
In this section, we conducted extensive experiments under the setting of VPIL. Through the experiments, we want to investigate the following questions: (1) Can COMPILER and COMPILER-E solve VPIL problems under various expert ratios in demonstrations? (2) Is COMPILER-E still valid when using different MPE estimators to obtain  $\widehat{\alpha}$ ? (3) How is the  $\alpha$  estimation with  $\beta_{1}$  and  $\beta_{2}$  as in (7)?
Table 1: The detailed information of demonstrations in the empirical studies.
Dimension of S	Dimension of A	Non-Expert 1	Non-Expert 2	Expert	α
Hopper	11	3	1142.16±159.28	1817.80±819.69	3606.11±43.95	{0.1, 0.2, 0.3, 0.4, 0.5}
Swimmer	8	2	46.28±1.87	62.64±16.82	100.63±4.69	{0.1, 0.2, 0.3, 0.4, 0.5}
Walker2d	17	6	410.98±56.91	766.67±398.85	3204.37±848.55	{0.1, 0.2, 0.3, 0.4, 0.5}
HalfCheetah	17	6	532.50±58.66	864.18±335.90	1599.06±41.97	{0.1, 0.2, 0.3, 0.4, 0.5}
To investigate the answer to the first question, we chose 20 different VPIL tasks with four MuJoCo benchmark environments to evaluate the performance of the contenders and our approaches under five different  $\alpha$  levels. The detailed setups are reported as follows.
**Environments and Demonstrations.** We set HalfCheetah, Hopper, Swimmer, and Walker2d in MuJoCo as the basic environments. For each experiment, the demonstration pool contains 100 trajectories with different expert ratios  $\alpha = \{0.1, 0.2, 0.3, 0.4, 0.5\}$ , in which  $\alpha = 0.1$  is the most difficult situation of the VPIL problem. We also trained an expert-level RL agent and two non-expert RL agents as demonstrators by DDPG algorithm [41]. The details of the environment and the demonstrations can be found in Table 1.
Dataset generation process. We start with the demonstration pool  $\{\tau_i\}_{i = 1}^N$ , which contains  $\alpha N$  trajectories sampled from the optimal policy  $\pi_{\mathsf{E}}$  and  $(1 - \alpha)N$  trajectories from the non-optimal policy  $\pi_{\mathrm{NE}}$ . For notation simplicity, we denote by  $\mathcal{D}_+$  the part containing expert demonstrations only and by  $\mathcal{D}_{-}$  the part for the non-expert demonstrations. Then, we can generate the dataset as follows,
$$
\left\{ \begin{array}{l l} \Gamma^ {+} = & \mathrm {s a m p l e} (\mathcal {D} _ {+}, (2 \alpha - \alpha^ {2}) N) \bigcup \mathrm {s a m p l e} (\mathcal {D} _ {-}, (1 - \alpha) ^ {2} N), \\ \Gamma^ {-} = & \mathrm {s a m p l e} (\mathcal {D} _ {+}, \alpha^ {2} N) \bigcup \mathrm {s a m p l e} (\mathcal {D} _ {-}, (1 - \alpha^ {2}) N), \end{array} \right.
$$
where  $\mathrm{sample}(\mathcal{D},N)$  is the function that samples  $N$  trajectories from the dataset  $\mathcal{D}$ .
Contenders. Since  $\Gamma^{+}$  contains more expert demonstrations, here we use Behavior Cloning [42], GAIL [1], and AIRL [43] with  $\Gamma^{+}$  only as basic baselines. Also, we provide GAIL with expert-level demonstrations (GAIL_expert) as the skyline of all methods. Besides, we set T-REX [13], a state-of-the-art preference-based IL method, by taking that every data in  $\Gamma^{+}$  is more expert-like than that in  $\Gamma^{-}$  as a form of preference to train its reward function. We also set the variants of T-REX algorithms, D-REX [22] and SSRR [44], that directly generate the ranking information through the imperfect datasets. CAIL proposed a confidence-based method to imitate from mixed demonstrations [14]. In the experiment, we provide  $5\%$  trajectory labels to meet its requirement as suggested in their paper. Each method ran 4e7 steps. 5 trials were conducted with different seeds for each task.
Meanwhile, to investigate the answer to the second question, we conducted Best Bin Estimation (BBE) [31] and Kernel Mean (KM) [28] algorithm to estimate  $\beta$  in algorithm 1 of COMPILER-E, as COMPILER-E (BBE) and COMPILER-E (KM) respectively. We note that COMPILER-E cannot obtain the true  $\alpha$  during the whole experiment.
Implementation of  $\alpha$  estimation. We implement a four-layer fully connected neural network as the binary classifier  $f_{\psi}$  in Algorithm 1. The neurons in each layer are 1000, 1000, 100, and 50 respectively. The activation function of each layer is ReLU, and the output value will go through the sigmoid function to obtain the score for  $\Gamma^{+}$  and  $\Gamma^{-}$  as  $S^{+}$  and  $S^{-}$  respectively. The optimizer for  $f_{\psi}$  is stochastic gradient descent. We totally train  $f_{\psi}$  for 1000 epochs with bath size 1024. The initial learning rate is 5e-4, with an exponential decay rate of 0.99 at each step.
Implementation of policy training. We choose Proximal Policy Optimization (PPO) [45] as the basic RL algorithm, and set all hyper-parameters, update frequency, and network architectures of the policy part the same as [46]. Besides, the hyper-parameters of the discriminator for all methods were the same: The discriminator was updated using Adam with a decayed learning rate of  $3 \times 10^{-4}$ ; the batch size was 256. The ratio of update frequency between the learner and discriminator was 3: 1. For T-REX algorithm [13], we use the codes with default hyperparameters and model architecture of their official implementation.
6.2 Empirical Results
The final results are gathered in Table 2. We can see that since AIRL and GAIL are standard IL algorithms, they cannot adaptively filter out non-expert data, so the underlying non-expert data reduces the performance of them. T-REX did not obtain promising results on VPII tasks, which
verifies that vague pairwise comparison can not be considered as full-ranking information, so the preference-based method cannot be used to tackle such problems. In addition, both SSRR and D-REX are updated versions of T-REX using noise to generate ranking information. They still cannot achieve good results due to its inability to address the core challenges of the VPIL problem. This phenomenon further demonstrates that preference-based algorithms cannot solve VPIL problems without explicit preference information, and some assumptions may not be held in VPIL problems. As  $\alpha$  grows, we can see that the performances of almost all the contenders increased more or less, except for the skyline GAIL_expert, while COMPILER and COMPILER-E got the biggest performance boost and achieved the best performances. Even under the  $\alpha = 0.1$  situation, COMPILER and COMPILER-E have achieved expert-level performance on Hopper and Walker2d environments. For CAIL, it shows competitive performance with COMPILER and COMPILER-E under high  $\alpha$  situations, but still falls short in low  $\alpha$  cases. The reason for this phenomenon is that when  $\alpha$  is relatively low, the trajectory quality corresponding to the provided trajectory labels may also be poor. Under such circumstances, estimating confidence is extremely difficult. As a result, CAIL generally performs suboptimally in low  $\alpha$  cases. On the other hand, our algorithms can achieve comparable results with CAIL in high alpha scenarios. This indicates that our methods reasonably utilize the structure of the VPIL problem and can achieve results on par with the state-of-the-art algorithm without introducing additional information. Meanwhile, COMPILER (BBE) and COMPILER (KM) both achieved promising and comparable performances, which indicates that COMPILER-E is a general algorithm and can be operated with any effective MPE method.
6.3 Case Study: Estimation Effect on Data Selection
Overestimation and underestimation comparisons. To investigate if the superiority of COMPILER-E comes from precisely estimating  $\alpha$ , and also to answer the third question about using two different estimations in (7) in Section 6, we reported the estimation curves under different true  $\alpha$  on HalfCheetah with BBE and KM estimators, as shown in Figure 3. We can see that the overestimation method obtained  $\widehat{\alpha}$  that far exceeded the true  $\alpha$ , especially when  $\alpha$  was smaller; while the underestimation method was very accurate despite obtaining a slightly lower  $\widehat{\alpha}$  value compared to the true  $\alpha$ , for both BBE and KM estimators. The results also suggested that we indeed need to use the underestimation method for COMPILER-E when the expert ratio  $\alpha$  is small. It also reflects that the power of COMPILER-E for solving VPIL problems indeed comes from accurately estimating the  $\alpha$  value. More environmental results can be found in Appendix B.
Thrown data effect. To further investigate what caused the performance gap between the overestimation and underestimation, we analyze the effect on the proportion of data selections in Algorithm 1. Also connected to the expert and non-expert proportions of  $\Gamma^{+}$  and  $\Gamma^{-}$  in (2), we can calculate the theoretical value of the ratio differences of thrown data under different estimations with that under true  $\alpha$ , as shown in Figure 3.
We can see that the overestimated method threw away more data in  $\Gamma^{-}$  and fewer data in  $\Gamma^{+}$  (corresponding to the upper left triangles of the red boxes in the figures); meanwhile, the underestimated method did the opposite (corresponding to the lower right triangles). However, as analyzed in Section 4.2, the overestimated method relied heavily on  $p_{\pi_E}$  of  $\Gamma^{-}$  dataset, whose components, were quite small. In the case of finite data, once  $p_{\pi_E}$  of  $\Gamma^{-}$  were overthrown, the remaining part of  $p_{\pi_E}$  became less, making the estimation further high, which led to a vicious circle. The phenomenon was especially severe in the case of low  $\alpha$ . This is the reason why  $\hat{\alpha}$  of the overestimated method became
(a) BBE estimator HalfCheetah
on(b) KM estimator HalfCheetah
on(c) Thrown data ratio differ(d) Thrown data ratio difference on  $\Gamma^{+}$  dataset ence on  $\Gamma^{-}$  dataset
Figure 3: (a, b) The comparisons of overestimated  $\left(\frac{2\beta}{1 + \beta}\right)$  and underestimated  $\left(\frac{1 - \beta}{1 + \beta}\right)$  methods on various tasks with BBE and KM estimators. (c,d) The distribution map of thrown data ratio using estimated  $\widehat{\alpha}$  and true  $\alpha$ . The thrown data ratio difference on  $\Gamma^{+}$  is  $(1 - \widehat{\alpha})^2 - (1 - \alpha)^2$ , while that on  $\Gamma^{-}$  is  $\widehat{\alpha}^2 - \alpha^2$ . The range marked by the red box is considered in our experiments, in which the expert ratio is smaller than (or equal to) the non-expert one.
bigger as true  $\alpha$  decreased. On the other hand, although the underestimated method also overthrew the non-expert data in  $\Gamma^{+}$ , the proportion of these data was relatively high. So even if more were thrown away at the beginning, it will not affect the accuracy of the estimation. This is the reason why using  $\frac{1 - \beta}{1 + \beta}$  to estimate  $\widehat{\alpha}$  is relatively stable and accurate.
6.4 Case Study: Assessing the Robustness of the COMPILER Algorithm
To evaluate the robustness of the COMPILER algorithm, a comprehensive set of experiments were conducted. These were carried out in four distinct environments, implementing tasks with  $\alpha = 0.5$ . Each experiment was repeated five times to ensure the reliability of the results. To emulate real-world conditions, different levels of noise were introduced to the parameter  $\alpha$ , resulting in the generation of corresponding noisy datasets. This noise was denoted as  $\varepsilon$ , thereby resulting in  $\widehat{\alpha} = \alpha + \varepsilon$ .
Figure 4 presents the findings from these experiments. Remarkably, the COMPILER algorithm demonstrated consistent performance, achieving at least  $96\%$  of the standard performance even in the presence of various noise levels. This robustness, as exhibited in the results, underlines the capacity of COMPILER to effectively handle and perform under noisy conditions, specifically when the provided  $\alpha$  is subject to disturbances. This emphasizes not only the resilience of the algorithm but also its potential for deployment in real-world situations where data is seldom perfect.
(a) Hopper
(b) Walker2d
(c) HalfCheetah
Figure 4: The performance of COMPILER under different noise levels.
(d) Swimmer
In this work, we formulated the problem of Vaguely Pairwise Imitation Learning (VPIL), in which mixed expert and non-expert demonstrations are present, and the data collector only provides vague pairwise information of demonstrations. To solve this problem, we proposed two learning paradigms, with risk rewriting and mixture proportion estimations (MPE), to recover the expert distribution with the known expert ratio  $\alpha$  and unknown one respectively. Afterward, we showed that these paradigms can be integrated with off-the-shelf IL methods, such as GAIL, to form the algorithm COMParative Imitation LEearning with Risk rewriting (COMPILER) and that by Estimation (COMPILER-E) to solve the VPIL problem with known and unknown  $\alpha$  respectively. The experimental results showed that our methods outperformed standard and preference-based IL methods on a variety of tasks. In the future, we hope to use our algorithms to address more challenging problems, such as VPIL with multiple and noisy annotators.
This research was supported by the Institute for AI and Beyond, UTokyo; JST SPRING, Grant Number JPMJSP2108. The authors would like to thank Johannes Ackermann and the anonymous reviewers for their insightful comments and suggestions.
[38] Ryuichi Kiryo, Gang Niu, Martinus Christoffel du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NIPS 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1675-1685, 2017.
[39] Nan Lu, Tianyi Zhang, Gang Niu, and Masashi Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. In The 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1115-1125, 2020.
[40] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. The MIT press, 2012.
[41] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, 2016.
[42] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, UK, July 1995], pages 103-129. Oxford University Press, 1995.
[43] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[44] Letian Chen, Rohan R. Paleja, and Matthew C. Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pages 1262-1277. PMLR, 2020.
[45] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
[46] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, 2017.
[47] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning - From Theory to Algorithms. Cambridge University Press, 2014.
A Proof for the Theorems
In this section, we provide the proof for Theorem 2, Proposition 1, and Theorem 3 in the main body of the paper.