SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {P ^ {+} (\leq y)}{P ^ {\prime -} (\leq y)} \leq \sup  _ {\lambda} \frac {\mathbb {P} \left(M _ {\lambda} (D) \leq y\right)}{\mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) \leq y\right)} \cdot \frac {C}{c} \quad \text {a n d} \quad \frac {P ^ {+} (<   y)}{P ^ {\prime -} (<   y)} \leq \sup  _ {\lambda} \frac {\mathbb {P} \left(M _ {\lambda} (D) <   y\right)}{\mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) <   y\right)} \cdot \frac {C}{c}. \tag {A.11}
$$
It is also straightforward to verify from the definition that
$$
P ^ {+} (\leq y) = P ^ {+} (<   y) + P ^ {+} (y) \quad \text {a n d} \quad P ^ {\prime +} (\leq y) = P ^ {\prime +} (<   y) + P ^ {\prime +} (y) \tag {A.12}
$$
$$
P ^ {+} - (\leq y) = P ^ {-} (<   y) + P ^ {-} (y) \quad \text {a n d} \quad P ^ {\prime -} (\leq y) = P ^ {\prime -} (<   y) + P ^ {\prime -} (y). \tag {A.13}
$$
Lemma A.3. Suppose if  $a_{\lambda}, b_{\lambda}$  are non-negative and  $c_{\lambda}, c_{\lambda}'$  are positive for all  $\lambda$ . Then we have
$$
\frac {\sum_ {\lambda} a _ {\lambda} c _ {\lambda}}{\sum_ {\lambda} b _ {\lambda} c _ {\lambda} ^ {\prime}} \leq \frac {\sum_ {\lambda} a _ {\lambda}}{\sum_ {\lambda} b _ {\lambda}} \cdot \sup  _ {\lambda , \lambda^ {\prime}} \left| \frac {c _ {\lambda}}{c _ {\lambda} ^ {\prime}} \right|.
$$
Proof of Lemma A.3. This lemma is pretty straight forward by comparing the coefficient for each term in the full expansion. Specifically, we re-write the inequality as
$$
\sum_ {\lambda} a _ {\lambda} c _ {\lambda} \sum_ {\lambda^ {\prime}} b _ {\lambda} ^ {\prime} \leq \sum_ {\lambda} a _ {\lambda} \sum_ {\lambda^ {\prime}} b _ {\lambda} ^ {\prime} c _ {\lambda} ^ {\prime} \cdot \sup  _ {\lambda , \lambda^ {\prime}} \left| \frac {c _ {\lambda}}{c _ {\lambda} ^ {\prime}} \right|. \tag {A.14}
$$
For each term  $a_{\lambda}b_{\lambda}'$ , its coefficient on the left hand side of (A.14) is  $c_{\lambda}$ , but its coefficient on the right hand side of (A.14) is  $c_{\lambda}' \cdot \sup_{\lambda, \lambda'} \left|\frac{c_{\lambda}}{c_{\lambda}'}\right|$ . Since we always have  $c_{\lambda}' \cdot \sup_{\lambda, \lambda'} \left|\frac{c_{\lambda}}{c_{\lambda}'}\right| \geq c_{\lambda}$ , and  $a_{\lambda}b_{\lambda}' \geq 0$ , we know the inequality (A.14) holds.
Next, in order to present our results in terms of RDP guarantees, we prove the following lemma.
Lemma A.4. The Rényi divergence between  $P^{+}$  and  $P^{-}$  is be bounded as follows:
$$
\mathrm {D} _ {\alpha} \left(P ^ {+} \| P ^ {\prime -}\right) \leq \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \sup  _ {\lambda \in \Lambda} \mathrm {D} _ {\alpha} \left(M _ {\lambda} (D) \| M _ {\lambda} \left(D ^ {\prime}\right)\right)
$$
Proof of Lemma A.4. We write that
$$
e ^ {(\alpha - 1) \mathrm {D} _ {\alpha} \left(P ^ {+} \| P ^ {\prime -}\right)} = \sum_ {y \in \mathcal {Y}} P ^ {+} (y) ^ {\alpha} \cdot P ^ {\prime -} (y) ^ {1 - \alpha} = \sum_ {y \in \mathcal {Y}} \frac {\left(\sum_ {\lambda} \pi^ {+} (\lambda) \mathbb {P} \left(M _ {\lambda} (D) = y\right)\right) ^ {\alpha}}{\left(\sum_ {\lambda} \pi^ {\prime -} (\lambda) \mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) = y\right)\right) ^ {\alpha - 1}} \tag {A.15}
$$
Here,  $\pi^{+}$  and  $\pi^{\prime -}$  are defined in (A.5) and (A.6), so they are essentially  $\pi_y^+$  and  $\pi_y^{\prime -}$  as they depend on the value of  $y$ . Therefore, we need to "remove" this dependence on  $y$  to leverage the RDP guarantees for each base algorithm  $M_{\lambda}$ . We accomplish this task by bridging via  $\pi$ , the uniform density on  $\Lambda$  (that is  $\pi (\lambda) = \pi (\lambda ')$  for any  $\lambda ,\lambda '\in \Lambda$ ). Specifically, we define  $a_{\lambda} = \pi (\lambda)\mathbb{P}(M_{\lambda}(D) = y)$ ,  $b_{\lambda} = \pi (\lambda)\mathbb{P}(M_{\lambda}(D') = y)$ ,  $c_{\lambda} = \frac{\pi_y^+(\lambda)}{\pi(\lambda)}$ , and  $c_{\lambda}' = \frac{\pi_y'^-(\lambda)}{\pi(\lambda)}$ . We see that
$$
\sup  _ {\lambda , \lambda^ {\prime}} \left| \frac {c _ {\lambda}}{c _ {\lambda} ^ {\prime}} \right| = \sup  _ {\lambda , \lambda^ {\prime}} \left| \frac {\pi_ {y} ^ {+} (\lambda) / \pi (\lambda)}{\pi_ {y} ^ {\prime -} (\lambda^ {\prime}) / \pi (\lambda^ {\prime})} \right| = \sup  _ {\lambda , \lambda^ {\prime}} \left| \frac {\pi_ {y} ^ {+} (\lambda))}{\pi_ {y} ^ {\prime -} (\lambda^ {\prime})} \right| \leq C / c, \tag {A.16}
$$
since  $\pi$  is the uniform, and  $\pi_y^+$  and  $\pi_y^{\prime -}$  belongs to  $S_{C,c}$ . We now apply Lemma A.3 with the above notations for each  $y$  to (A.15), and we have
$$
\begin{array}{l} \sum_ {y \in \mathcal {Y}} \frac {(\sum_ {\lambda} \pi^ {+} (\lambda) \mathbb {P} (M _ {\lambda} (D) = y)) ^ {\alpha}}{(\sum_ {\lambda} \pi^ {\prime -} (\lambda) \mathbb {P} (M _ {\lambda} (D ^ {\prime}) = y)) ^ {\alpha - 1}} \\ = \sum_ {y \in \mathcal {Y}} \frac {\left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} (M _ {\lambda} (D) = y) \cdot \frac {\pi^ {+} (\lambda)}{\pi (\lambda)}\right) ^ {\alpha - 1} \left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} (M _ {\lambda} (D) = y) \cdot \frac {\pi^ {+} (\lambda)}{\pi (\lambda)}\right)}{\left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} (M _ {\lambda} (D ^ {\prime}) = y) \cdot \frac {\pi^ {\prime -} (\lambda)}{\pi (\lambda)}\right) ^ {\alpha - 1}} \\ = \sum_ {y \in \mathcal {Y}} \frac {\left(\sum_ {\lambda} a _ {\lambda} \cdot c _ {\lambda}\right) ^ {\alpha - 1} \left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} (M _ {\lambda} (D) = y) \cdot \frac {\pi^ {+} (\lambda)}{\pi (\lambda)}\right)}{\left(\sum_ {\lambda} b _ {\lambda} \cdot c _ {\lambda} ^ {\prime}\right) ^ {\alpha - 1}} \\ \leq \sum_ {y \in \mathcal {Y}} \sup  _ {\lambda , \lambda^ {\prime}} \left| \frac {c _ {\lambda}}{c _ {\lambda} ^ {\prime}} \right| ^ {\alpha - 1} \frac {\left(\sum_ {\lambda} a _ {\lambda}\right) ^ {\alpha - 1} \left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} (M _ {\lambda} (D) = y) \cdot \frac {\pi^ {+} (\lambda)}{\pi (\lambda)}\right)}{\left(\sum_ {\lambda} b _ {\lambda}\right) ^ {\alpha - 1}} \\ = \sum_ {y \in \mathcal {Y}} \sup  _ {\lambda , \lambda^ {\prime}} \left| \frac {c _ {\lambda}}{c _ {\lambda} ^ {\prime}} \right| ^ {\alpha - 1} \frac {\left(\sum_ {\lambda} a _ {\lambda}\right) ^ {\alpha - 1} \left(\sum_ {\lambda} a _ {\lambda} \cdot c _ {\lambda}\right)}{\left(\sum_ {\lambda} b _ {\lambda}\right) ^ {\alpha - 1}} \\ \leq \sum_ {y \in \mathcal {Y}} \sup  _ {\lambda , \lambda^ {\prime}} \left| \frac {c _ {\lambda}}{c _ {\lambda} ^ {\prime}} \right| ^ {\alpha - 1} \frac {\left(\sum_ {\lambda} a _ {\lambda}\right) ^ {\alpha - 1} \left(\sum_ {\lambda} a _ {\lambda}\right) \cdot \sup  _ {\lambda} c _ {\lambda}}{\left(\sum_ {\lambda} b _ {\lambda}\right) ^ {\alpha - 1}} \\ \leq \sum_ {y \in \mathcal {Y}} \left(\frac {C}{c}\right) ^ {\alpha - 1} \frac {\left(\sum_ {\lambda} a _ {\lambda}\right) ^ {\alpha - 1} \left(\sum_ {\lambda} a _ {\lambda}\right) \cdot \left(\frac {C}{c}\right)}{\left(\sum_ {\lambda} b _ {\lambda}\right) ^ {\alpha - 1}} \\ = \sum_ {y \in \mathcal {Y}} \left(\frac {C}{c}\right) ^ {\alpha} \cdot \frac {\left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} \left(M _ {\lambda} (D) = y\right)\right) ^ {\alpha}}{\left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) = y\right)\right) ^ {\alpha - 1}} \\ \end{array}
$$
The first inequality is due to Lemma A.3, the second inequality is because  $a_{\lambda}$  are non-negative, and the last inequality is because of (A.16) and the fact that both  $\pi^{+}(\lambda)$  and  $\pi (\lambda)$  are defined in  $\mathbf{S}_{C,c}$ , and thus their ratio is upper bounded by  $\frac{C}{c}$  for any  $\lambda$ .
Now we only need to prove that for any fixed distribution  $\pi$  that doesn't depend on value  $y$ , we have
$$
\sum_ {y \in \mathcal {Y}} \frac {\left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} \left(M _ {\lambda} (D) = y\right)\right) ^ {\alpha}}{\left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} \left(M _ {\lambda} \left(D ^ {\prime}\right) = y\right)\right) ^ {\alpha - 1}} \leq \sup  _ {\lambda \in \Lambda} e ^ {(\alpha - 1) \mathrm {D} _ {\alpha} \left(M _ {\lambda} (D) \| M _ {\lambda} \left(D ^ {\prime}\right)\right)}. \tag {A.17}
$$
With this result, we immediately know the result holds for uniform distribution  $\pi$  as a special case. To prove this result, we first observe that the function  $f(u,v) = u^{\alpha}v^{1 - \alpha}$  is a convex function. This is because the Hessian of  $f$  is
$$
\left( \begin{array}{c c} \alpha (\alpha - 1) u ^ {\alpha - 2} v ^ {1 - \alpha} & - \alpha (\alpha - 1) u ^ {\alpha - 1} v ^ {- \alpha} \\ - \alpha (\alpha - 1) u ^ {\alpha - 1} v ^ {- \alpha} & \alpha (\alpha - 1) u ^ {\alpha} v ^ {- \alpha - 1} \end{array} \right),
$$
which is easy to see to be positive semi-definite. And now, consider any distribution  $\pi$ , denote  $u(\lambda) = \mathbb{P}(M_{\lambda}(D) = y)$  and  $v(\lambda) = \mathbb{P}(M_{\lambda}(D') = y)$  by Jensen's inequality, we have
$$
f \left(\sum_ {\lambda} \pi (\lambda) u (\lambda), \sum_ {\lambda} \pi (\lambda) v (\lambda)\right) \leq \sum_ {\lambda} \pi (\lambda) f (u (\lambda), v (\lambda)).
$$
By adding the summation over  $y$  on both side of the above inequality, we have
$$
\begin{array}{l} \sum_ {y \in \mathcal {Y}} \frac {\left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} (M _ {\lambda} (D) = y)\right) ^ {\alpha}}{\left(\sum_ {\lambda} \pi (\lambda) \mathbb {P} (M _ {\lambda} (D ^ {\prime}) = y)\right) ^ {\alpha - 1}} \leq \sum_ {y \in \mathcal {Y}} \sum_ {\lambda} \pi (\lambda) \frac {\mathbb {P} (M _ {\lambda} (D) = y) ^ {\alpha}}{\mathbb {P} (M _ {\lambda} (D ^ {\prime}) = y) ^ {\alpha - 1}} \\ = \sum_ {\lambda} \sum_ {y \in \mathcal {Y}} \pi (\lambda) \frac {\mathbb {P} (M _ {\lambda} (D) = y) ^ {\alpha}}{\mathbb {P} (M _ {\lambda} (D ^ {\prime}) = y) ^ {\alpha - 1}} \\ \leq \sup  _ {\lambda} \sum_ {y \in \mathcal {Y}} \frac {\mathbb {P} (M _ {\lambda} (D) = y) ^ {\alpha}}{\mathbb {P} (M _ {\lambda} (D ^ {\prime}) = y) ^ {\alpha - 1}}. \\ \end{array}
$$
The first equality is due to Fubini's theorem. And the second inequality is straight forward as one observe  $\pi (\lambda)$  only depends on  $\lambda$ . This concludes the proof as we know that
$$
\begin{array}{l} e ^ {(\alpha - 1) \mathrm {D} _ {\alpha} (P ^ {+} \| P ^ {\prime -})} \leq \left(\frac {C}{c}\right) ^ {\alpha} \sup  _ {\lambda} \sum_ {y \in \mathcal {Y}} \frac {\mathbb {P} (M _ {\lambda} (D) = y) ^ {\alpha}}{\mathbb {P} (M _ {\lambda} (D ^ {\prime}) = y) ^ {\alpha - 1}} \\ = \left(\frac {C}{c}\right) ^ {\alpha} \sup  _ {\lambda} e ^ {(\alpha - 1) \mathrm {D} _ {\alpha} (M _ {\lambda} (D) \| M _ {\lambda} (D ^ {\prime})} \\ \end{array}
$$
$$
\mathrm {D} _ {\alpha} \left(P ^ {+} \| P ^ {\prime -}\right) \leq \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \sup  _ {\lambda \in \Lambda} \mathrm {D} _ {\alpha} \left(M _ {\lambda} (D) \| M _ {\lambda} \left(D ^ {\prime}\right)\right).
$$
We now present our crucial technical lemma for adaptive hyperparameter tuning with any distribution on the number of repetitions  $T$ . This is a generalization from [32].
Lemma A.5. Fix  $\alpha > 1$ . Let  $T$  be a random variable supported on  $\mathbb{N}_{\geq 0}$ . Let  $f:[0,1] \to \mathbb{R}$  be the probability generating function of  $K$ , that is,  $f(x) = \sum_{k=0}^{\infty} \mathbb{P}[T=k] x^{k}$ .
Let  $M_{\lambda}$  and  $M_{\lambda}'$  be the base algorithm for  $\lambda \in \Lambda$  on  $\mathcal{V}$  on  $D$  and  $D'$  respectively. Define  $A_1 \coloneqq \mathcal{A}(D, \pi^{(0)}, \mathcal{T}, C, c)$ , and  $A_2 \coloneqq \mathcal{A}(D', \pi^{(0)}, \mathcal{T}, C, c)$ . Then
$$
\mathrm {D} _ {\alpha} \left(A _ {1} \| A _ {2}\right) \leq \sup  _ {\lambda} \mathrm {D} _ {\alpha} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \frac {1}{\alpha - 1} \log \left(f ^ {\prime} (q) ^ {\alpha} \cdot f ^ {\prime} \left(q ^ {\prime}\right) ^ {1 - \alpha}\right),
$$
where applying the same postprocessing to the bounding probabilities  $P^{+}$  and  $P^{\prime -}$  gives probabilities  $q$  and  $q^{\prime}$  respectively. This means that, there exist a function set  $g: \mathcal{V} \to [0,1]$  such that  $q = \underset{X \leftarrow P^{+}}{\mathbb{E}}[g(X)]$  and  $q^{\prime} = \underset{X^{\prime} \leftarrow P^{\prime -}}{\mathbb{E}}[g(X^{\prime})]$ .
Proof of Lemma A.5. We consider the event that  $A_{1}$  outputs  $y$ . By definition, we have
$$
\begin{array}{l} A _ {1} (y) = \sum_ {k = 1} ^ {\infty} \mathbb {P} (T = k) [ \prod_ {j = 1} ^ {k} P _ {j} (\leq y) - \prod_ {i = 1} ^ {k} P _ {j} (<   y) ] \\ = \sum_ {k = 1} ^ {\infty} \mathbb {P} (T = k) [ \sum_ {i = 1} ^ {k} P _ {i} (y) \prod_ {j = 1} ^ {i - 1} P _ {j} (<   y) \cdot \prod_ {j = i + 1} ^ {k} P _ {j} (\leq y) ] \\ \leq \sum_ {k = 1} ^ {\infty} \mathbb {P} (T = k) [ \sum_ {i = 1} ^ {k} P ^ {+} (y) \prod_ {j = 1} ^ {i - 1} P ^ {+} (<   y) \cdot \prod_ {j = i + 1} ^ {k} P ^ {+} (\leq y) ] \\ = \sum_ {k = 1} ^ {\infty} \mathbb {P} (T = k) \left[ \sum_ {i = 1} ^ {k} P ^ {+} (y) \cdot P ^ {+} (<   y) ^ {i - 1} \cdot P ^ {+} (\leq y) ^ {k - i} \right] \\ = \sum_ {k = 1} ^ {\infty} \mathbb {P} (T = k) [ P ^ {+} (\leq y) ^ {k} - P ^ {+} (<   y) ^ {k} ] \\ = f (P ^ {+} (\leq y)) - f (P ^ {+} (<   y)) = P ^ {+} (y) \cdot \underset {X \leftarrow \mathrm {U n i f o r m} ([ P ^ {+} (<   y), P ^ {+} (\leq y) ])} {\mathbf {E}} [ f ^ {\prime} (X) ]. \\ \end{array}
$$
The second equality is by partitioning on the events of the first time of getting  $y$ , we use  $i$  to index such a time. The third inequality is using (A.7), (A.9), and (A.10). The third to last equality is by (A.12) and algebra. The second to last equality is by definition of the probability generating function  $f$ . The last equality follows from definition of integral.
$$
A _ {2} (y) \geq \sum_ {k = 1} ^ {\infty} \mathbb {P} (T = k) \left[ P ^ {\prime -} (\leq y) ^ {k} - P ^ {\prime -} (<   y) ^ {k} \right] = P ^ {\prime -} (y) \cdot \underset {X \leftarrow \text {U n i f o r m} ([ P ^ {\prime -} (<   y), P ^ {\prime -} (\leq y) ])} {\mathbf {E}} [ f ^ {\prime} (X) ].
$$
The rest part of the proof is standard and follows similarly as in [32]. Specifically, we have
$$
\begin{array}{l} e ^ {(\alpha - 1) \mathrm {D} _ {\alpha} (A _ {1} \| A _ {2})} \\ = \sum_ {y \in \mathcal {Y}} A _ {1} (y) ^ {\alpha} \cdot A _ {2} (y) ^ {1 - \alpha} \\ \leq \sum_ {y \in \mathcal {Y}} P ^ {+} (y) ^ {\alpha} \cdot P ^ {\prime -} (y) ^ {1 - \alpha} \cdot \underset {X \leftarrow [ P ^ {+} (<   y), P ^ {+} (\leq y) ]} {\mathbb {E}} \left[ f ^ {\prime} (X) \right] ^ {\alpha} \cdot \underset {X ^ {\prime} \leftarrow [ P ^ {\prime -} (<   y), P ^ {\prime -} (\leq y) ]} {\mathbb {E}} \left[ f ^ {\prime} \left(X ^ {\prime}\right) \right] ^ {1 - \alpha} \\ \leq \sum_{y\in \mathcal{V}}P^{+}(y)^{\alpha}\cdot P^{\prime -}(y)^{1 - \alpha}  .\quad \underset { \begin{array}{c}X\leftarrow [P^{+}(<  y),P^{+}(\leq y)]\\ X^{\prime}\leftarrow \left[P^{\prime -}(<  y),P^{\prime -}(\leq y)\right] \end{array} }{\mathbb{E}}\left[f^{\prime}(X)^{\alpha}\cdot f^{\prime}\left(X^{\prime}\right)^{1 - \alpha}\right] \\ \leq \left(\frac{C}{c}\right)^{\alpha}\sup_{\lambda}e^{(\alpha -1)\mathrm{D}_{\alpha}\big(M_{\lambda}(D)\| M_{\lambda}(D^{\prime})\big)}\cdot \max_{y\in \mathcal{Y}}\underset { \begin{array}{c}X\gets [P^{+}(<   y),P^{+}(\leq y)]\\ X^{\prime}\gets [P^{\prime -}(<   y),P^{\prime -}(\leq y)] \end{array} }{\mathbb{E}}\left[f^{\prime}(X)^{\alpha}\cdot f^{\prime}\left(X^{\prime}\right)^{1 - \alpha}\right]. \\ \end{array}
$$
The last inequality follows from Lemma A.4. The second inequality follows from the fact that, for any  $\alpha \in \mathbb{R}$ , the function  $h:(0,\infty)^2\to (0,\infty)$  given by  $h(u,v) = u^{\alpha}\cdot v^{1 - \alpha}$  is convex. Therefore,  $\mathbb{E}[U]^{\alpha}\mathbb{E}[V]^{1 - \alpha} = h(\mathbb{E}[(U,V)])\leq \mathbb{E}[h(U,V)] = \mathbb{E}\left[U^{\alpha}\cdot V^{1 - \alpha}\right]$  all positive random variables  $(U,V)$ . Note that  $X$  and  $X^{\prime}$  are required to be uniform separately, but their joint distribution can be arbitrary. As in [32], we will couple them so that  $\frac{X - P^{+}(<y)}{P^{+}(y)} = \frac{X^{\prime} - P^{\prime - }( < y)}{P^{\prime - }(y)}$ . In particular, this implies that, for each  $y\in \mathcal{V}$ , there exists some  $t\in [0,1]$  such that
$$
\operatorname *{\mathbb{E}}_{\substack{X\leftarrow [P^{+}(<   y),P^{+}(\leq y)]\\ X^{\prime}\leftarrow \left[P^{\prime -}(<   y),P^{\prime -}(\leq y)\right]}}\left[f^{\prime}(X)^{\alpha}\cdot f^{\prime}(X^{\prime})^{1 - \alpha}\right]\leq f^{\prime}(P^{+}(<   y) + t\cdot P^{+}(y))^{\alpha}\cdot f^{\prime}\left(P^{\prime -}(<   y) + t\cdot P^{\prime -}(y)\right)^{1 - \alpha}
$$
$$
\begin{array}{l} \mathrm {D} _ {\alpha} \left(A _ {1} \| A _ {2}\right) \leq \sup  _ {\lambda} \mathrm {D} _ {\alpha} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} \\ +\frac{1}{\alpha - 1}\log \left(\max_{\substack{y\in \mathcal{Y}\\ t\in [0,1]}}f^{\prime}(P^{+}(<   y) + t\cdot P^{+}(y))^{\alpha}\cdot f^{\prime}\left(P^{\prime -}(<   y) + t\cdot P^{\prime -}(y)\right)^{1 - \alpha}\right). \\ \end{array}
$$
To prove the result, we simply fix  $y_* \in \mathcal{V}$  and  $t_* \in [0,1]$  achieving the maximum above and define
$$
g (y) := \left\{ \begin{array}{l l} 1 & \text {i f} y <   y _ {*} \\ t _ {*} & \text {i f} y = y _ {*} \\ 0 & \text {i f} y > y _ {*} \end{array} \right.
$$
The result directly follows by setting  $q = \mathbb{E}_{X \leftarrow P^{+}}[g(X)]$  and  $q' = \mathbb{E}_{X' \leftarrow P'^{-}}[g(X')]$ .
Now we can prove Theorem 1, given the previous technical lemma. The proof share similarity to the proof of Theorem 2 in [32] with the key difference from the different form in Lemma A.5. We demonstrate this proof as follows for completeness.
Proof of Theorem 1. We first specify the probability generating function of the truncated negative binomial distribution
$$
f (x) = \underset {T \sim \operatorname {N e g B i n} (\theta , \gamma)} {\mathbb {E}} \left[ x ^ {T} \right] = \left\{ \begin{array}{l l} \frac {(1 - (1 - \gamma) x) ^ {- \theta} - 1}{\gamma^ {- \theta} - 1} & \text {i f} \theta \neq 0 \\ \frac {\log (1 - 1 - \gamma) x)}{\log (\gamma)} & \text {i f} \theta = 0 \end{array} \right.
$$
$$
\begin{array}{l} f ^ {\prime} (x) = (1 - (1 - \gamma) x) ^ {- \theta - 1} \cdot \left\{ \begin{array}{l l} \frac {\theta \cdot (1 - \gamma)}{\gamma - \theta - 1} & \text {i f} \theta \neq 0 \\ \frac {1 - \gamma}{\log (1 / \gamma)} & \text {i f} \theta = 0 \end{array} \right. \\ = (1 - (1 - \gamma) x) ^ {- \theta - 1} \cdot \gamma^ {\theta + 1} \cdot \mathbb {E} [ T ]. \\ \end{array}
$$
By Lemma A.5, for appropriate values  $q, q' \in [0,1]$  and for all  $\alpha > 1$  and all  $\hat{\alpha} > 1$ , we have
$$
\begin{array}{l} \mathrm {D} _ {\alpha} \left(A _ {1} \| A _ {2}\right) \\ \leq \sup  _ {\lambda} \mathrm {D} _ {\alpha} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \frac {1}{\alpha - 1} \log \left(f ^ {\prime} (q) ^ {\alpha} \cdot f ^ {\prime} \left(q ^ {\prime}\right) ^ {1 - \alpha}\right) \\ \leq \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \frac {1}{\alpha - 1} \log \left(\gamma^ {\theta + 1} \cdot \mathbb {E} [ T ] \cdot (1 - (1 - \gamma) q) ^ {- \alpha (\theta + 1)} \cdot (1 - (1 - \gamma) q ^ {\prime}) ^ {- (1 - \alpha) (\theta + 1)}\right) \\ = \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} \\ + \frac {1}{\alpha - 1} \log \left(\gamma^ {\theta + 1} \cdot \mathbb {E} [ T ] \cdot \left((\gamma + (1 - \gamma) (1 - q)) ^ {1 - \hat {\alpha}} \cdot (\gamma + (1 - \gamma) (1 - q ^ {\prime})) ^ {\hat {\alpha}}\right) ^ {\nu} \cdot (\gamma + (1 - \gamma) (1 - q)) ^ {u}\right) \\ \end{array}
$$
(Here, we let  $\hat{\alpha}\nu = (\alpha -1)(1 + \theta)$  and  $(1 - \hat{\alpha})\nu +u = -\alpha (\theta +1))$
$$
\leq \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \frac {1}{\alpha - 1} \log \left(\gamma^ {\theta + 1} \cdot \mathbb {E} [ T ] \cdot \left(\gamma + (1 - \gamma) \cdot e ^ {(\hat {\alpha} - 1) \mathrm {D} _ {\hat {\alpha}} (P ^ {+} \| P ^ {-})}\right) ^ {\nu} \cdot (\gamma + (1 - \gamma) (1 - q)) ^ {u}\right)
$$
(Here,  $1 - q$  and  $1 - q^{\prime}$  are postprocessings of some  $P^{+}$  and  $P^{\prime -}$  respectively and  $e^{(\hat{\alpha} -1)\mathrm{D}_{\hat{\alpha}}(\cdot ||\cdot)}$  is convex )
$$
\leq \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \frac {1}{\alpha - 1} \log \left(\gamma^ {\theta + 1} \cdot \mathbb {E} [ T ] \cdot \left(\gamma + (1 - \gamma) \cdot e ^ {(\hat {\alpha} - 1) \sup  _ {\lambda} D _ {\hat {\alpha}} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + \hat {\alpha} \log \frac {C}{c}}\right) ^ {\nu} \cdot (\gamma + (1 - \gamma) (1 - q)) ^ {u}\right)
$$
(By Lemma A.4)
$$
\begin{array}{l} \leq \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \frac {1}{\alpha - 1} \log \left(\gamma^ {\theta + 1} \cdot \mathbb {E} [ T ] \cdot \left(\gamma + (1 - \gamma) \cdot e ^ {(\hat {\alpha} - 1) \sup  _ {\lambda} D _ {\hat {\alpha}} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + \hat {\alpha} \log \frac {C}{c}}\right) ^ {\nu} \cdot (\gamma + (1 - \gamma) (1 - q)) ^ {u}\right) \\ \leq \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \frac {1}{\alpha - 1} \log \left(\gamma^ {\theta + 1} \cdot \mathbb {E} [ T ] \cdot \left(\gamma + (1 - \gamma) \cdot e ^ {(\hat {\alpha} - 1) \sup  _ {\lambda} D _ {\hat {\alpha}} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + \hat {\alpha} \log \frac {C}{c}}\right) ^ {\nu} \cdot \gamma^ {u}\right) \\ \end{array}
$$
(Here  $\gamma \leq \gamma +(1 - \gamma)(1 - q)$  and  $u\le 0$
$$
= \varepsilon +\frac{\alpha}{\alpha - 1}\log \frac{C}{c} +\frac{\nu}{\alpha - 1}\log \left(\gamma +(1 - \gamma)\cdot e^{(\hat{\alpha} -1)\sup_{\lambda}D_{\hat{\alpha}}\left(M_{\lambda}\| M_{\lambda}^{\prime}\right) + \hat{\alpha}\log \frac{C}{c}}\right) + \frac{1}{\alpha - 1}\log \left(\gamma^{\theta +1}\cdot \mathbb{E}[T]\cdot \gamma^{u}\right)
$$
$$
\begin{array}{l} = \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + \frac {\nu}{\alpha - 1} \left(\left(\hat {\alpha} - 1\right) \sup  _ {\lambda} \mathrm {D} _ {\hat {\alpha}} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + \hat {\alpha} \log \frac {C}{c} \right. \\ \left. + \log \left(1 - \gamma \cdot \left(1 - e ^ {- (\hat {\alpha} - 1) \sup  _ {\lambda} D _ {\hat {\alpha}} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + \hat {\alpha} \log \frac {C}{c}}\right)\right)\right) + \frac {1}{\alpha - 1} \log \left(\gamma^ {u + \theta + 1} \cdot \mathbb {E} [ T ]\right) \\ = \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + (1 + \theta) \left(1 - \frac {1}{\hat {\alpha}}\right) \sup  _ {\lambda} D _ {\hat {\alpha}} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + (1 + \theta) \log \frac {C}{c} \\ + \frac {1 + \theta}{\hat {\alpha}} \log \left(1 - \gamma \cdot \left(1 - e ^ {- (\hat {\alpha} - 1) \sup  _ {\lambda} \mathrm {D} _ {\hat {\alpha}} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + \hat {\alpha} \log \frac {C}{c}\right)\right) + \frac {\log (\mathbb {E} [ T ])}{\alpha - 1} + \frac {1 + \theta}{\hat {\alpha}} \log (1 / \gamma) \\ \end{array}
$$
(Here we have  $\nu = \frac{(\alpha - 1)(1 + \theta)}{\hat{\alpha}}$  and  $u = -(1 + \theta)\left(\frac{\alpha - 1}{\hat{\alpha}} +1\right)$
$$
\begin{array}{l} = \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + (1 + \theta) \left(1 - \frac {1}{\hat {\alpha}}\right) \sup  _ {\lambda} D _ {\hat {\alpha}} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) + (1 + \theta) \log \frac {C}{c} \\ + \frac {1 + \theta}{\hat {\alpha}} \log \left(\frac {1}{\gamma} - 1 + e ^ {- (\hat {\alpha} - 1) \sup  _ {\lambda} \mathrm {D} _ {\hat {\alpha}} \left(M _ {\lambda} \| M _ {\lambda} ^ {\prime}\right) - \hat {\alpha} \log \frac {C}{c}}\right) + \frac {\log (\mathbb {E} [ T ])}{\alpha - 1} \\ \leq \varepsilon + \frac {\alpha}{\alpha - 1} \log \frac {C}{c} + (1 + \theta) \left(1 - \frac {1}{\hat {\alpha}}\right) \hat {\varepsilon} + (1 + \theta) \log \frac {C}{c} + \frac {1 + \theta}{\hat {\alpha}} \log \left(\frac {1}{\gamma}\right) + \frac {\log (\mathbb {E} [ T ])}{\alpha - 1}, \\ \end{array}
$$