SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 2. Assume the pairwise datasets  $(\Gamma^{+},\Gamma^{-})$  are generated following the procedure in Section 3.2 in the main paper. Let  $p_{\pi_+}(s,a) = \mathbb{E}_{\Gamma^+}\left[\widehat{p}_{\pi_+}(s,a)\right]$  and  $p_{\pi_-}(s,a) = \mathbb{E}_{\Gamma^-}\left[\widehat{p}_{\pi_-}(s,a)\right]$  be the expected occupancy measures of  $\Gamma^{+}$  and  $\Gamma^{-}$ , where the randomness is taken over the draws of  $\Gamma^{+}$  and  $\Gamma^{-}$ . Then, we have
$$
\left\{ \begin{array}{l l} p _ {\pi_ {+}} (s, a) = & \left(2 \alpha - \alpha^ {2}\right) p _ {\pi_ {E}} (s, a) + (1 - \alpha) ^ {2} p _ {\pi_ {N E}} (s, a), \\ p _ {\pi_ {-}} (s, a) = & \alpha^ {2} p _ {\pi_ {E}} (s, a) + (1 - \alpha^ {2}) p _ {\pi_ {N E}} (s, a), \end{array} \right. \tag {9}
$$
$$
\left\{ \begin{array}{l l} p _ {\pi_ {E}} (s, a) = & \frac {1 + \alpha}{2 \alpha} p _ {\pi_ {+}} (s, a) - \frac {1 - \alpha}{2 \alpha} p _ {\pi_ {-}} (s, a), \\ p _ {\pi_ {N E}} (s, a) = & - \frac {\alpha}{2 (1 - \alpha)} p _ {\pi_ {+}} (s, a) + \frac {2 - \alpha}{2 - 2 \alpha} p _ {\pi_ {-}} (s, a). \end{array} \right. \tag {10}
$$
Proof. We first prove the first line of (9). Considering the data collection process introduced in Section 3.2, where a pair of trajectories  $(\tau^{(1)},\tau^{(2)})$  is independently sampled from the demonstrator pool. When one trajectory enjoys a better quality than another, the collector would put the better one into  $\Gamma^{+}$ . Otherwise, the two trajectories will be randomly allocated. Let  $\widehat{p}_i^+(s,a) = (1 - \gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{1}[(s_{t,i},a_{t,i}) = (s,a)]$  be the empirical occupancy measure for the trajectory  $\tau_i \in \Gamma^+$ . Then, further define the even  $E_1 = \{\tau_i\}$  and event  $E_1 = \{\tau_i\}$  is sampled from  $\pi_{\mathsf{NE}}$ . We have
$$
\begin{array}{l} \mathbb {E} \left[ \widehat {p _ {i} ^ {+}} (s, a) \right] = \Pr \left[ E _ {1} \right] \mathbb {E} _ {\Gamma^ {+}} \left[ \widehat {p _ {i} ^ {+}} (s, a) \mid E _ {1} \right] + \Pr \left[ E _ {2} \right] \mathbb {E} _ {\Gamma^ {+}} \left[ \widehat {p _ {i} ^ {+}} (s, a) \mid E _ {2} \right] \\ = \Pr [ E _ {1} ] p _ {\pi_ {\mathrm {E}}} + \Pr [ E _ {2} ] p _ {\pi_ {\mathrm {N E}}}. \tag {11} \\ \end{array}
$$
The second equality is due to  $\tau_{i}$  is i.i.d. sampled from  $p_{\pi_E}$  under event  $E_{1}$  and is i.i.d. sampled from  $p_{\pi_{-}}$  under event  $E_{2}$ . Then, let the even  $A_{1} = \{\text{both } \tau^{(1)}, \tau^{(2)} \text{ are sampled from } \pi_E\}$ ,  $A_{2} = \{\text{both } \tau^{(1)}, \tau^{(2)} \text{ are sampled from } \pi_{\mathsf{NE}}\}$  and  $A_{3} = \{\text{other situation}\}$ . Then, we have
$$
\begin{array}{l} \Pr \left[ E _ {1} \right] = \Pr \left[ A _ {1} \right] \Pr \left[ E _ {1} \mid A _ {1} \right] + \Pr \left[ A _ {2} \right] \Pr \left[ E _ {1} \mid A _ {2} \right] + \Pr \left[ A _ {3} \right] \Pr \left[ E _ {1} \mid A _ {3} \right] \\ = \alpha^ {2} + 2 \alpha (1 - \alpha) = 2 \alpha - \alpha^ {2}, \\ \end{array}
$$
where the second inequality is due to  $\operatorname{Pr}[E_1 \mid A_1] = \operatorname{Pr}[E_3 \mid A_3] = 1$  and  $\operatorname{Pr}[E_1 \mid A_2] = 0$  according to the data collection process. We can also show the probability of event  $E_2$  is  $\operatorname{Pr}[E_1] = (1 - \alpha)^2$ , then we have
$$
\mathbb {E} \left[ \widehat {p _ {i}} ^ {+} (s, a) \right] = \left(2 \alpha - \alpha^ {2}\right) p _ {\pi_ {\mathsf {E}}} (s, a) + (1 - \alpha) ^ {2} p _ {\pi_ {\mathsf {N E}}} (s, a).
$$
Since  $\widehat{p}_{\pi_+}(s,a) = \sum_{i=1}^{n_+} \widehat{p}_i^+(s,a)/n_+$ , we have  $\mathbb{E}[\widehat{p}_{\pi_+}(s,a)] = \sum_{i=1}^{n_+} \mathbb{E}[\widehat{p}_i^+(s,a)]/n_+$ $(2\alpha - \alpha^2)p_{\pi_E}(s,a) + (1 - \alpha)^2 p_{\pi_{NE}}(s,a)$ , which completes the proof for the first equality of (9). The second equality of (9) can be proved following a similar arguments
Proposition 2. Suppose the distributions  $p_{\pi_E}$  and  $p_{\pi_{NE}}$  are mutually irreducible such that there exists no decomposition of the form  $p_{\pi_E} = (1 - \eta)Q + \eta p_{\pi_{NE}}$  and  $p_{\pi_{NE}} = (1 - \eta')Q' + \eta' p_{\pi_E}$  for any probability distributions  $Q, Q'$  and scalars  $\eta, \eta' \in (0,1]$ . Then, the true mixture proportions  $\beta_1$  and  $\beta_2$  are unique and can be identified by
$$
\left\{ \begin{array}{l} \beta_ {1} = \sup  \left\{\eta \mid p _ {\pi_ {+}} = \eta p _ {\pi_ {-}} + (1 - \eta) K, K \in \mathcal {C} \right\}, \\ \beta_ {2} = \sup  \left\{\eta^ {\prime} \mid p _ {\pi_ {-}} = \eta^ {\prime} p _ {\pi_ {+}} + (1 - \eta^ {\prime}) K ^ {\prime}, K ^ {\prime} \in \mathcal {C} \right\}, \end{array} \right. \tag {12}
$$
$$
\alpha = (1 - \beta_ {1}) / (1 + \beta_ {1}) \quad o r \quad \alpha = 2 \beta_ {2} / (1 + \beta_ {2}). \tag {13}
$$
Proof. We first prove the first line of (12). Since  $p_{\pi_E}$  is irreducible w.r.t.  $p_{\pi_{\mathrm{NE}}}$ , such that there exists no decomposition  $p_{\pi_E} = (1 - \eta)Q + \eta p_{\pi_{\mathrm{NE}}}$  for any  $\eta \in (0,1]$  and distribution  $Q$ , we have  $p_{\pi_E}$  is also irreducible w.r.t.  $p_{\pi_-}$ , such that  $p_{\pi_E}$  can not be rewritten as a mixture  $p_{\pi_E} = (1 - \eta')Q' + \eta'p_{\pi_-}$  for any distribution  $Q'$  and  $\eta' \in (0,1]$ . Then, according to the relationship (5) of the main paper and Proposition 5 of [16], we can show  $\beta_1$  is identifiable and  $\beta_1 = \sup \{\eta | p_{\pi_+} = \eta p_{\pi_-} + (1 - \eta)K, K \in \mathcal{C}\}$ . We can prove the second line of (12) by a similar argument. Then, combined with  $\beta_1 = \frac{1 - \alpha}{1 + \alpha}$  and  $\beta_2 = \frac{\alpha}{2 - \alpha}$  in (5) of the main paper, we can obtain identifiable  $\alpha$  estimation in (13).
Theorem 3. Let  $\mathcal{W}$  be a parameter space for training the discriminator and  $D_{\mathcal{W}} = \{D_w \mid w \in \mathcal{W}\}$  be the hypothesis space. Assume the functions  $|\log D_w(s, a)| \leq B$  and  $|\log (1 - D_w(s, a))| \leq B$  are upper-bounded for any state-action pair  $(s, a) \in S \times \mathcal{A}$  and  $w \in \mathcal{W}$ . Further assume both the functions  $\log D_w(s, a)$  and  $\log (1 - D_w(s, a))$  are  $L$ -Lipschitz continuous in the state-action space. For a fixed policy  $\pi_\theta$ , let  $\Gamma^\theta = \{\tau_i^\theta\}_{i=1}^{n_\theta}$  be trajectories generated from  $\pi_\theta$ . Then, for any  $\delta \in (0,1)$  with probability at least  $1 - \delta$ , we have
Proof. Since  $\widehat{D}_w$  and  $D_w^*$  are the maximizer of the objective functions  $\widehat{V}(\pi_\theta, D_w)$  and  $V(\pi_\theta, D_w)$ , respectively.
$$
\begin{array}{l} V \left(\pi_ {\theta}, D _ {w} ^ {*}\right) - V \left(\pi_ {\theta}, \widehat {D} _ {w}\right) = V \left(\pi_ {\theta}, D _ {w} ^ {*}\right) - \widehat {V} \left(\pi_ {\theta}, D _ {w} ^ {*}\right) + \widehat {V} \left(\pi_ {\theta}, D _ {w} ^ {*}\right) - \widehat {V} \left(\pi_ {\theta}, \widehat {D} _ {w}\right) (14) \\ + \widehat {V} (\pi_ {\theta}, \widehat {D} _ {w}) - V (\pi_ {\theta}, \widehat {D} _ {w}) \\ \leq V \left(\pi_ {\theta}, D _ {w} ^ {*}\right) - \widehat {V} \left(\pi_ {\theta}, D _ {w} ^ {*}\right) + \widehat {V} \left(\pi_ {\theta}, \widehat {D} _ {w}\right) - V \left(\pi_ {\theta}, \widehat {D} _ {w}\right) \\ \leq 2 \sup  _ {w \in \mathcal {W}} | V \left(\pi_ {\theta}, D _ {w}\right) - \widehat {V} \left(\pi_ {\theta}, D _ {w}\right) |, (15) \\ \end{array}
$$
where the inequality is due to the optimality of  $\widehat{D}_w$ . The according to the definition of  $V(\pi_{\theta}, D_w)$  and  $\widehat{V}(\pi_{\theta}, D_w)$ ,
$$
\begin{array}{l} V \left(\pi_ {\theta}, D _ {w}\right) = 2 \alpha \mathbb {E} _ {(s, a) \sim p _ {\pi_ {\theta}}} \left[ \log D _ {w} (s, a) \right] + (1 - \alpha) \mathbb {E} _ {(s, a) \sim p _ {\pi_ {-}}} [ \log \left(D _ {w} (s, a)\right) ] \tag {16} \\ + (1 + \alpha) \mathbb {E} _ {(s, a) \sim p _ {\pi_ {+}}} [ \log (1 - D _ {w} (s, a)) ] \\ \end{array}
$$
$$
\begin{array}{l} \widehat {V} \left(\pi_ {\theta}, D _ {w}\right) = 2 \alpha \mathbb {E} _ {(s, a) \sim \widehat {p} _ {\pi_ {\theta}}} \left[ \log D _ {w} (s, a) \right] + (1 - \alpha) \mathbb {E} _ {(s, a) \sim \widehat {p} _ {\pi_ {-}}} \left[ \log \left(D _ {w} (s, a)\right) \right] \tag {17} \\ + (1 + \alpha) \mathbb {E} _ {(s, a) \sim \widehat {p} _ {\pi_ {+}}} [ \log (1 - D _ {w} (s, a)) ], \\ \end{array}
$$
where  $\widehat{p}_{\pi_{\theta}}(s,a) = \frac{1}{n_{\theta}}\sum_{i = 1}^{n_{\theta}}\widehat{p}_{i}^{\theta}(s,a)$  and  $\widehat{p}_i^\theta (s,a) = (1 - \gamma)\sum_{t = 0}^\infty \gamma^t\mathbb{1}\left[(s_{t,i},a_{t,i}) = (s,a)\right]$  is the empirical occupancy measure for the trajectory  $\tau_{i}$  sampled from  $\pi_{\theta}$  The empirical occupancy measure is unbiased w.r.t.  $\pi_{\theta}$  such that  $\mathbb{E}[\widehat{p}_i^\theta (s,a)] = p_{\pi_\theta}(s,a)$ . Then, we can decompose the R.H.S. of (15) as
$$
\sup  _ {w \in \mathcal {W}} | V (\pi_ {\theta}, D _ {w}) - \widehat {V} (\pi_ {\theta}, D _ {w}) | = \text {t e r m (a) + t e r m (b) + t e r m (c)}.
$$
In above, the first term
$$
\begin{array}{l} \text {t e r m} (\mathsf {a}) = 2 \alpha \left| \mathbb {E} _ {(s, a) \sim \widehat {p} _ {\pi_ {\theta}}} [ \log D _ {w} (s, a) ] - \mathbb {E} _ {(s, a) \sim p _ {\pi_ {\theta}}} [ \log D _ {w} (s, a) ] \right| \\ = 2 \alpha \left| \frac {1}{n _ {\theta}} \sum_ {i = 1} ^ {n _ {\theta}} \mathbb {E} _ {(s, a) \sim \hat {p} _ {i} ^ {\theta}} [ \log D _ {w} (s, a) ] - \mathbb {E} _ {(s, a) \sim p _ {\pi_ {\theta}}} [ \log D _ {w} (s, a) ] \right| \\ \end{array}
$$
measures the generalization gap between  $\widehat{p}_{\pi_{\theta}}$  and  $p_{\pi_{\theta}}$ . The second term
$$
\text {t e r m} (b) = (1 - \alpha) \left| \frac {1}{n _ {-}} \sum_ {i = 1} ^ {n _ {-}} \mathbb {E} _ {(s, a) \sim \widehat {p} _ {i} ^ {-}} [ \log D _ {w} (s, a) ] - \mathbb {E} _ {(s, a) \sim p _ {\pi_ {-}}} [ \log D _ {w} (s, a) ] \right|
$$
measures the generalization gap between  $\widehat{p}_{\pi_{-}}$  and  $p_{\pi_{-}}$  and the last term
$$
\text {t e r m (c)} = (1 + \alpha) \left| \frac {1}{n _ {+}} \sum_ {i = 1} ^ {n _ {-}} \mathbb {E} _ {(s, a) \sim \widehat {p} _ {i} ^ {+}} [ \log (1 - D _ {w} (s, a)) ] - \mathbb {E} _ {(s, a) \sim p _ {\pi_ {+}}} [ \log (1 - D _ {w} (s, a)) ] \right|
$$
measures the gap between  $\widehat{p}_{\pi_+}$  and  $p_{\pi_+}$ .
Under the condition that  $|\log D_w(s,a)|\leq B$  for any  $(s,a)\in S\times \mathcal{A}$  and  $w\in \mathcal{W}$ , the standard analysis generalization analysis with Rademacher complexity shows (e.g. Theorem 26.5 of [47]), for all  $w\in \mathcal{W}$ , we have
$$
\text {t e r m} (\mathbf {a}) \leq 8 \alpha \mathcal {R} _ {n _ {\theta}} \left(\log \circ D _ {\mathcal {W}}\right) + 4 \alpha B \sqrt {\frac {2 \ln \left(2 / \delta^ {\prime}\right)}{n _ {\theta}}},
$$
with probability at least  $1 - \delta'$ , where  $\mathcal{R}_{n_{\theta}}$  is the empirical Rademacher complexity. Then, according to the Talagrand's contraction inequality (Lemma 26.9 of [47]), we have
$$
\mathcal {R} _ {n _ {\theta}} (\log \circ D _ {\mathcal {W}}) \leq L \mathcal {R} _ {n _ {\theta}} (D _ {\mathcal {W}}),
$$
$$
\text {t e r m} (\mathsf {a}) \leq 8 \alpha L \mathcal {R} _ {n _ {\theta}} \left(D _ {\mathcal {W}}\right) + 4 \alpha B \sqrt {\frac {2 \ln \left(2 / \delta^ {\prime}\right)}{n _ {\theta}}}. \tag {18}
$$
Since  $|\log (1 - D_w)|\leq B$  and  $\log (1 - D_w)$  is  $L$ -Lipschitz continuous, the a similar arguments ensures,
$$
\text {t e r m} (\mathrm {b}) \leq 4 (1 - \alpha) L \mathcal {R} _ {n _ {-}} \left(D _ {\mathcal {W}}\right) + 2 (1 - \alpha) B \sqrt {\frac {2 \ln \left(2 / \delta^ {\prime}\right)}{n _ {-}}}. \tag {19}
$$
$$
\text {t e r m} (c) \leq 4 (1 + \alpha) \mathcal {R} _ {n _ {+}} \left(D _ {\mathcal {W}}\right) + 2 (1 + \alpha) B \sqrt {\frac {2 \ln \left(2 / \delta^ {\prime}\right)}{n _ {+}}}, \tag {20}
$$
with probability at least  $1 - \delta'$ . Let  $\delta' = \delta / 3$ , we complete the proof by combining (18), (19), and (20) with (15).
B Full Results of  $\alpha$  Estimations
In the main body of the paper, we only reported the effect of overestimation and underestimation on HalfCheetah environment due to the space limitation. Here we provide the full results on four environments in our experiments, as shown in Figure 5. The experimental results demonstrated the same conclusion as in the main body of the paper, that the underestimation method is better than the overestimation one.
on(c) BBE estimator on Hop(d) KM estimator on Hopper per
(e) BBE estimator on Swimmer
(f) KM estimator on Swi
(g) BBE estimator Walker2d
on(h) KM estimator on Walker2d
Figure 5: The comparisons of overestimated  $(\frac{2\beta}{1 + \beta})$  and underestimated  $(\frac{1 - \beta}{1 + \beta})$  methods on various tasks with BBE and KM estimators.