SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Md Manjurul Ahsan, Shahana Akter Luna, and Zahed Siddique. Machine-learning-based disease diagnosis: A comprehensive review. Healthcare, 10(3):541, 2022.
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv:2107.07511, 2021.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In Int. Conf. on Machine Learning, pages 254-263, 2018.
Yu Bai, Song Mei, Huan Wang, Yingbo Zhou, and Caiming Xiong. Efficient and differentiable conformal prediction with general function classes. In Int. Conf. on Learning Representations, 2022.
Peter L. Bartlett, Dylan J. Foster, and Matus J Telgarsky. Spectrally-Normalized Margin Bounds for Neural Networks. In Conf. on Neural Information Processing Systems, volume 30, 2017.
Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Occam's razor. Information Processing Letters, 24(6):377-380, 1987.
Matthew Cleaveland, Insup Lee, George J Pappas, and Lars Lindemann. Conformal prediction regions for time series using linear complementarity programming. arXiv:2304.01075, 2023.
Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Differentiable ranking and sorting using optimal transport. In Conf. on Neural Information Processing Systems, volume 32, 2019.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183-212, 1983.
Gintare Karolina Dziugaite and Daniel M. Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. In Proc. Conf. on Uncertainty in Artificial Intelligence, 2017.
Gintare Karolina Dziugaite and Daniel M Roy. Data-dependent pac-bayes priors via differential privacy. In Conf. on Neural Information Processing Systems, volume 31, 2018.
Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertainty-aware classifiers with conformalized deep learning. In Conf. on Neural Information Processing Systems, volume 35, 2022.
Mahdi Milani Fard, Joelle Pineau, and Csaba Szepesvári. Pac-bayesian policy evaluation for reinforcement learning. arXiv:1202.3717, 2012.
Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization of sorting networks via continuous relaxations. In Int. Conf. on Learning Representations, 2019.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic Generalization Measures and Where to Find Them. In Int. Conf. on Learning Representations, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press, 2014.
Apoorva Sharma, Navid Azizan, and Marco Pavone. Sketching curvature for efficient out-of-distribution detection for deep neural networks. In Proc. Conf. on Uncertainty in Artificial Intelligence, pages 1958-1967, 2021.
David Stutz, Ali Taylan Cemgil, Arnaud Doucet, et al. Learning optimal conformal classifiers. In Int. Conf. on Learning Representations, 2021.
Jessica Vamathevan, Dominic Clark, Paul Czodrowski, Ian Dunham, Edgardo Ferran, George Lee, Bin Li, Anant Madabhushi, Parantu Shah, Michaela Spitzer, et al. Applications of machine learning in drug discovery and development. Nature Reviews Drug Discovery, 18(6):463-477, 2019.
Vladimir N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Dokl. Akad. Nauk, 181(4), 1968.
Sushant Veer and Anirudha Majumdar. Probably approximately correct vision-based planning using motion primitives. In Conf. on Robot Learning, pages 1001-1014, 2020.
Paul Viallard, Pascal Germain, Amaury Habrard, and Emilie Morvant. A general framework for the practical disintegration of pac-bayesian bounds. Machine Learning, 2023.
Vladimir Vovk. Conditional validity of inductive conformal predictors. In *Asian Conf. on Machine Learning*, pages 475–490, 2012.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer, 2005.
Waymo. The Waymo Driver Handbook: Teaching an autonomous vehicle how to perceive and understand the world around it, 2021. Available at https://blog.waymo.com/2021/10/the-waymo-driver-handbook-perception.html.
Omry Yadan. Hydra - a framework for elegantly configuring complex applications. Github, 2019. URL https://github.com/facebookresearch/hydra.
Heng Yang and Marco Pavone. Object pose estimation with statistical guarantees: Conformal keypoint detection and geometric uncertainty propagation. In IEEE Conf. on Computer Vision and Pattern Recognition, 2023.
Yachong Yang and Arun Kumar Kuchibhotla. Finite-sample efficient conformal prediction. arXiv:2104.13871, 2021.
Our coverage and efficiency generalization bounds follow a similar form to many PAC-Bayes bounds. A fundamental building block of these proofs is the Donsker-Varadhan variational formula [Donsker and Varadhan, 1983], which we restate below.
Lemma 1 (Donsker-Varadhan variational formula). For any measurable bounded function  $h: \Theta \to \mathbb{R}$ , we have,
$$
\log \operatorname * {E} _ {\theta \sim P} [ \exp (h (\theta)) ] = \sup  _ {Q \in \mathcal {P} (\theta)} \left[ \operatorname * {E} _ {\theta \sim Q} [ h (\theta) ] - \mathrm {K L} (Q | | P) \right].
$$
We then apply this lemma to obtain a core result relating to quantities depending on parameters  $\theta$  and sampled data  $D_N$ , which we derive first as a second lemma which is common to the proof of both generalization bounds.
Lemma 2. Let  $\Phi(\theta, D_N): \Theta \times (\mathcal{X} \times \mathcal{Y})^N \to \mathbb{R}$  be a measurable function mapping a parameter and a set of sampled data to a scalar. Let  $P$  be an arbitrary distribution over  $\Theta$ , independent of  $D_N$ . Let  $Q$  denote another arbitrary probability distribution over  $\Theta$ . Suppose there exists a function  $B(N) > 0$  such that, for any fixed  $\theta \in \Theta$ ,
$$
\underset {D _ {N}} {\mathbb {E}} \left[ \exp (\Phi (\theta , D _ {N})) \right] \leq B (N).
$$
$$
\underset {D _ {N}} {\mathbb {P}} \left(\forall Q: \underset {\theta \sim Q} {\mathbb {E}} [ \Phi (\theta , D _ {N}) ] \leq \operatorname {K L} (Q | | P) + \log \left(\frac {B (N)}{\delta}\right)\right) \geq 1 - \delta .
$$
Proof of Lemma 2. We start with our assumed property of  $B(N)$ , and integrate it with respect to  $\theta$ .
$$
\underset {D _ {N}} {\mathbb {E}} \left[ \exp (\Phi (\theta , D _ {N})) \right] \leq B (N)
$$
$$
\underset {\theta \sim P} {\mathbb {E}} \underset {D _ {N}} {\mathbb {E}} \left[ \exp (\Phi (\theta , D _ {N})) \right] \leq B (N)
$$
The integrand is non-negative and  $P$  is independent of  $D_N$ ; so, by Tonelli's theorem, we can swap the order of the integration with respect to the parameters  $\theta$  and the dataset  $D_N$ , yielding
$$
\mathop{\mathbb{E}}_{D_{N}}\mathop{\mathbb{E}}_{\theta \sim P}\left[\exp (\Phi (\theta ,D_{N}))\right]\leq B(N)
$$
Now, we can apply the Donsker-Varadhan variational formula to get
$$
\underset {D _ {N}} {\mathbb {E}} \left[ e ^ {\sup  _ {Q \in \mathcal {P} (\theta)} \left[ \mathbb {E} _ {\theta \sim Q} \left[ \Phi \left(\theta , D _ {N}\right) \right] - \mathrm {K L} (Q | | P) \right]} \right] \leq B (N) \tag {12}
$$
$$
\mathbb {E} _ {D _ {N}} \left[ e ^ {\sup  _ {Q \in \mathcal {P} (\theta)} \left[ \mathbb {E} _ {\theta \sim Q} \left[ \Phi \left(\theta , D _ {N}\right) \right] - \operatorname {K L} (Q | | P) - \log B (N) \right]} \right] \leq 1. \tag {13}
$$
Now, by the Chernoff bound, for any random variable  $X$ ,
$$
\mathbb {P} (X > s) \leq \mathbb {E} [ e ^ {X} ] e ^ {- s},
$$
so if we let  $\delta = e^{-s}$ ,  $s = -\log \delta$ , we have
$$
\mathbb {P} \left(X > - \log \delta\right) \leq \mathbb {E} [ e ^ {X} ] \delta .
$$
Now, applying this bound to  $X = \sup_{Q\in \mathcal{P}(\theta)}[\mathbb{E}_{\theta \sim Q}\left[\Phi (\theta ,D_N)\right] - \mathrm{KL}(Q||P) - \log B(N)]$ , we obtain
$$
\begin{array}{l} \mathbb {P} _ {D _ {N}} \left(\sup  _ {Q \in \mathcal {P} (\theta)} \left[ \underset {\theta \sim Q} {\mathbb {E}} [ \Phi (\theta , D _ {N}) ] - \operatorname {K L} (Q | | P) - \log B (N) \right] > - \log \delta\right) \\ \leq \mathbb {E} _ {D _ {N}} \left[ e ^ {\sup  _ {Q \in \mathcal {P} (\theta)} \left[ \mathbb {E} _ {\theta \sim Q} [ \Phi (\theta , D _ {N}) ] - \mathrm {K L} (Q | | P) - \log B (N) ] \right]} \right] \delta \\ \leq \delta \tag {14} \\ \end{array}
$$
Now, since
$$
\begin{array}{l} \left(\exists Q \in \mathcal {P} (\theta): \underset {\theta \sim Q} {\mathbb {E}} [ \Phi (\theta , D _ {N}) ] - \operatorname {K L} (Q | | P) - \log B (N) > - \log \delta\right) \\ \Rightarrow \left(\sup  _ {Q \in \mathcal {P} (\theta)} \left[ \underset {\theta \sim Q} {\mathbb {E}} \left[ \Phi (\theta , D _ {N}) \right] - \operatorname {K L} (Q | | P) - \log B (N) \right] > - \log \delta\right), \\ \end{array}
$$
the result from (14) implies that
$$
\underset {D _ {N}} {\mathbb {P}} \left(\exists Q \in \mathcal {P} (\theta): \left[ \underset {\theta \sim Q} {\mathbb {E}} [ \Phi (\theta , D _ {N}) ] - \operatorname {K L} (Q | | P) - \log B (N) \right] \leq - \log \delta\right) \leq \delta
$$
Taking the complement, we have
$$
\underset {D _ {N}} {\mathbb {P}} \left(\forall Q \in \mathcal {P} (\theta): \left[ \underset {\theta \sim Q} {\mathbb {E}} [ \Phi (\theta , D _ {N}) ] - \operatorname {K L} (Q \| P) - \log B (N) \right] \leq - \log \delta\right) \geq 1 - \delta
$$
Rearranging terms completes the proof.
Equipped with this lemma, we can now prove our core theorems. We start by proving the coverage bound.
Proof of Theorem 1. First, let  $\mathcal{L}_{\mathrm{cov}}(\theta ,D_N,\hat{\alpha})$  represent the miscoverage rate of a conformal predictor calibrated to achieve  $1 - \hat{\alpha}$  empirical coverage on  $D_N$ :
$$
\mathcal {L} _ {\mathrm {c o v}} (\theta , D _ {N}, \hat {\alpha}) = \underset {\boldsymbol {x}, \boldsymbol {y} \sim \mathcal {D}} {\mathbb {P}} (\boldsymbol {y} \notin C (\boldsymbol {x}; \tau^ {*} (D _ {N}, \hat {\alpha}; \theta), \theta)).
$$
Note that for a fixed  $\theta$ , the distribution of coverage (and hence, mis-coverage) conditioned on a particular calibration dataset follows a Beta distribution [Angelopoulos and Bates, 2021, Vovk, 2012]. Specifically,  $\mathcal{L}_{\mathrm{cov}}(\theta, D_N, \hat{\alpha}) \sim \mathrm{Beta}((\lfloor(N + 1)\hat{\alpha}\rfloor, N + 1 - \lfloor(N + 1)\hat{\alpha}\rfloor))$ .
Now, choose  $\Phi (\theta ,D_N)$  to be
$$
\Phi (\theta , D _ {N}) = (N - 1) \mathrm {k l} \left(\frac {\lfloor (N + 1) \hat {\alpha} \rfloor - 1}{N - 1} \Bigg \| \mathcal {L} _ {\operatorname {c o v}} (\theta , D _ {N}, \hat {\alpha})\right).
$$
First, we derive the bound  $B(N)$  required to apply Lemma 2. Let  $k = \lfloor (N + 1)\hat{\alpha}\rfloor$ ,  $a = \frac{k - 1}{N - 1}$ , and  $b = \mathcal{L}_{\mathrm{cov}}(\theta ,D_N,\hat{\alpha})$ . Then,
$$
\begin{array}{l} \underset {D _ {N}} {\mathbb {E}} \left[ \exp \left(\Phi (\theta , D _ {N})\right) \right] \\ = \underset {D _ {N}} {\mathbb {E}} \left[ \exp \left(\left(N - 1\right) \operatorname {k l} (a \| b)\right) \right] \\ = \underset {D _ {N}} {\mathbb {E}} \left[ \exp \left((N - 1) a \log \frac {a}{b} + (N - 1) (1 - a) \log \frac {1 - a}{1 - b}\right) \right] \\ = \underset {D _ {N}} {\mathbb {E}} \left[ \left(\frac {a}{b}\right) ^ {(N - 1) a} \left(\frac {1 - a}{1 - b}\right) ^ {(N - 1) (1 - a)} \right] \\ \end{array}
$$
Note that  $(N - 1)a = k - 1$ , and  $(N - 1)(1 - a) = N - k$ . Substituting, we have
$$
\begin{array}{l} \underset {D _ {N}} {\mathbb {E}} \left[ \exp (\Phi (\theta , D _ {N})) \right] = \underset {D _ {N}} {\mathbb {E}} \left[ \left(\frac {a}{b}\right) ^ {k - 1} \left(\frac {1 - a}{1 - b}\right) ^ {N - k} \right] \\ = a ^ {k - 1} (1 - a) ^ {N - k} \underset {D _ {N}} {\mathbb {E}} \left[ b ^ {1 - k} (1 - b) ^ {k - N} \right] \\ = a ^ {k - 1} (1 - a) ^ {N - k} \int_ {0} ^ {1} b ^ {1 - k} (1 - b) ^ {k - N} \operatorname {B e t a} (b; k, N + 1 - k) d b \\ = a ^ {k - 1} (1 - a) ^ {N - k} \int_ {0} ^ {1} b ^ {1 - k} (1 - b) ^ {k - N} \frac {b ^ {k - 1} (1 - b) ^ {N - k} N !}{(k - 1) ! (N - k) !} d b \\ = a ^ {k - 1} (1 - a) ^ {N - k} \frac {N !}{(k - 1) ! (N - k) !} \\ = \operatorname {B e t a} (a; k, N + 1 - k) \\ \end{array}
$$
Thus, in this case we can analytically evaluate this expectation as
$$
\underset {D _ {N}} {\mathbb {E}} \left[ \exp \left(\Phi (\theta , D _ {N})\right) \right] = B (N) := \operatorname {B e t a} ((k - 1) / (N - 1); k, N + 1 - k). \tag {15}
$$
where  $\operatorname{Beta}(x; \alpha, \beta)$  is the pdf of the Beta distribution with parameters  $\alpha, \beta$ .
To compare our bound with other PAC-Bayes bounds and to quantify the asymptotic behavior of  $B(N)$ , we can derive an upper bound. First, we plug in our definition of  $a$ , and group terms
$$
B (N) = a ^ {k - 1} (1 - a) ^ {N - k} \frac {N !}{(k - 1) ! (N - k) !} = N \frac {(N - 1) !}{(N - 1) ^ {N - 1}} \frac {(k - 1) ^ {k - 1}}{(k - 1) !} \frac {(N - k) ^ {N - k}}{(N - k) !}
$$
By Stirling's approximation of  $a!$ , we know
$$
\sqrt {2 \pi a} a ^ {a} e ^ {- a + \frac {1}{1 2 a + 1}} <   a! <   \sqrt {2 \pi a} a ^ {a} e ^ {- a + \frac {1}{1 2 a}}
$$
$$
\frac {a ^ {a}}{a !} <   \frac {1}{\sqrt {2 \pi a}} e ^ {a - \frac {1}{1 2 a + 1}}
$$
$$
\frac {a !}{a ^ {a}} <   \sqrt {2 \pi a} e ^ {- a + \frac {1}{1 2 a}}
$$
Plugging in these bounds, we can obtain a simplified expression which removes the factorials and high powers of  $N$ :
$$
B (N) <   N \frac {\sqrt {(N - 1)}}{\sqrt {2 \pi (k - 1) (N - k)}} \exp \left(\frac {1}{1 2 N - 1 2} - \frac {1}{1 2 k - 1 1} - \frac {1}{1 2 N - 1 2 k + 1}\right)
$$
Notice that for  $0 < k < N$ , the term in the exponent is negative. Given the definition of  $k$ , this is true as long as  $N \geq \frac{1}{\alpha} - 1$ . In this case, due to monotonicity, the exponential term is less than 1. This yields
$$
B (N) <   N \frac {\sqrt {(N - 1)}}{\sqrt {2 \pi (k - 1) (N - k)}} := O \left(\sqrt {\frac {N}{\hat {\alpha} (1 - \hat {\alpha})}}\right). \tag {16}
$$
Having computed  $B(N)$ , we can now apply Lemma 2 to obtain
$$
\mathbb {P} _ {D _ {N}} \left( \right.\forall Q: \underset {\theta \sim Q} {\mathbb {E}} \left[ \right. (N - 1) \mathrm {k l} \left( \right.\frac {\lfloor (N + 1) \hat {\alpha} \rfloor - 1}{N - 1} \left\| \right. \mathcal {L} _ {\mathrm {c o v}} (\theta , D _ {N}, \hat {\alpha})\left. \right)\left. \right] \leq \mathrm {K L} (Q | | P) + \log \left(\frac {B (N)}{\delta}\right)\left. \right) \geq 1 - \delta .
$$
Since  $\mathrm{kl}(\cdot ,\cdot)$  is convex in both arguments, we can use Jensen's inequality to bring the expectation over  $\theta$  inside each argument, yielding,
$$
\mathbb{P}_{D_{N}}\left( \right.\forall Q:(N - 1)\mathrm{kl}\left( \right.\frac{\lfloor(N + 1)\hat{\alpha}\rfloor - 1}{N - 1}\left\| \right. \mathbb{E}_{\theta \sim Q}\left[\mathcal{L}_{\mathrm{cov}}(\theta ,D_{N},\hat{\alpha})\right]\left. \right) \leq \mathrm{KL}(Q||P) + \log \left(\frac{B(N)}{\delta}\right)\left. \right) \geq 1 - \delta .
$$
Note that by definition in equation (6),  $\mathcal{L}_{\mathrm{cov}}(Q) = \mathbb{E}_{\theta \sim Q}[\mathcal{L}_{\mathrm{cov}}(\theta ,D_N,\hat{\alpha})]$ . Rearranging terms completes the proof.
Proof of Corollary 2.1. Let  $q = \frac{\lfloor(n + 1)\hat{\alpha}\rfloor - 1}{n - 1}$ , and for compactness, let  $\mathcal{L}_{\mathrm{cov}} = \mathcal{L}_{\mathrm{cov}}(\theta ,\tau (\theta ,D_N,\hat{\alpha}))$  Let  $C$  be the event that our KL constraint holds, i.e.
$$
\operatorname {K L} (Q \| P) \leq (n - 1) \operatorname {k l} (q \| \alpha) - \log \left(\frac {B (n)}{\delta}\right)
$$
Consider the event  $E$  that  $\mathbb{E}_{\theta \sim Q}[\mathcal{L}_{\mathrm{cov}}] \leq q$ . Since  $q \leq \alpha$ , we have that
$$
\mathbb{P}\left(\underset {\theta \sim Q}{\mathbb{E}}[\mathcal{L}_{\mathrm{cov}}]\leq \alpha \Bigg|E,C\right) = 1.
$$
Now, consider the complement,  $\bar{E}$ . We know that  $\mathrm{kl}(q||x)$  is monotonically increasing in  $x$  for  $x\geq q$ . Thus, conditioned on  $\bar{E}$ , we have
$$
\mathbb {P} \left(\underset {\theta \sim Q} {\mathbb {E}} [ \mathcal {L} _ {\mathrm {c o v}} ] \leq \alpha \Big | \bar {E}, C\right) = \mathbb {P} \left(\operatorname {k l} \left(q \| \underset {\theta \sim Q} {\mathbb {E}} [ \mathcal {L} _ {\mathrm {c o v}} ]\right) \leq \operatorname {k l} (q \| \alpha) \Big | \bar {E}\right).
$$
Given these statements, we have
$$
\begin{array}{l} \mathbb {P} \left(\underset {\theta \sim Q} {\mathbb {E}} [ \mathcal {L} _ {\mathrm {c o v}} ] \leq \alpha \Bigg | C\right) \\ = \mathbb {P} (E \mid C) \mathbb {P} \left(\underset {\theta \sim Q} {\mathbb {E}} [ \mathcal {L} _ {\mathrm {c o v}} ] \leq \alpha \Bigg | E, C\right) + \mathbb {P} (\bar {E} \mid C) \mathbb {P} \left(\underset {\theta \sim Q} {\mathbb {E}} [ \mathcal {L} _ {\mathrm {c o v}} ] <   \alpha \Bigg | \bar {E}, C\right) \\ = * \mathbb {P} (E \mid C) + \mathbb {P} (\bar {E} \mid C) \mathbb {P} \left(\operatorname {k l} \left(q \| _ {\theta \sim_ {Q}} \left[ \mathcal {L} _ {\mathrm {c o v}} \right]\right) \leq \operatorname {k l} (q \| \alpha) \Big | \bar {E}, C\right) \\ \geq \mathbb {P} (E \mid C) \mathbb {P} \left(\operatorname {k l} \left(q \| _ {\theta \sim_ {Q}} \mathbb {E} [ \mathcal {L} _ {\mathrm {c o v}} ]\right) \leq \operatorname {k l} (q \| \alpha) \Big | E, C\right) \\ + \mathbb {P} (\bar {E} \mid C) \mathbb {P} \left(\operatorname {k l} \left(q \| _ {\theta \sim_ {Q}} \mathbb {E} [ \mathcal {L} _ {\mathrm {c o v}} ]\right) \leq \operatorname {k l} (q \| \alpha) \Big | \bar {E}, C\right) \\ = \mathbb {P} \left(\operatorname {k l} \left(q \| _ {\theta \sim Q} \mathbb {E} [ \mathcal {L} _ {\text {c o v}} ]\right) \leq \operatorname {k l} (q \| \alpha) \mid C\right) \tag {17} \\ \end{array}
$$
Now, due to the convexity of  $\mathrm{kl}$ , we have through Jensen's inequality,
$$
\underset {\theta \sim Q} {\mathbb {E}} \left[ \mathrm {k l} \left(q \| \mathcal {L} _ {\mathrm {c o v}}\right) \right] \geq \mathrm {k l} \left(q \| \underset {\theta \sim Q} {\mathbb {E}} \left[ \mathcal {L} _ {\mathrm {c o v}} \right]\right).
$$
Plugging this in to the result of Theorem 1, we have
$$
\mathbb {P} \left(\forall Q, \operatorname {k l} \left(q \| _ {\theta \sim Q} \mathbb {E} [ \mathcal {L} _ {\mathrm {c o v}} ]\right) \leq \frac {\operatorname {K L} (Q \| P) + \log \left(\frac {B (n)}{\delta}\right)}{n - 1}\right) \geq 1 - \delta
$$
Now, since this holds for all  $Q$ , it must also hold for those  $Q$  satisfying the constraint. Thus,
$$
\mathbb {P} \left(\forall Q, \operatorname {k l} \left(q \| _ {\theta \sim Q} \mathbb {E} [ \mathcal {L} _ {\text {c o v}} ]\right) \leq \frac {\operatorname {K L} (Q \| P) + \log \left(\frac {B (n)}{\delta}\right)}{n - 1} \Bigg | C\right) \geq 1 - \delta
$$
$$
\begin{array}{l} \mathbb {P} \left(\forall Q, \operatorname {k l} \left(q \| _ {\theta \sim Q} \mathbb {E} \left[ \mathcal {L} _ {\mathrm {c o v}} \right]\right) \leq \frac {(n - 1) \operatorname {k l} (q \| \alpha) - \log \left(\frac {B (n)}{\delta}\right) + \log \left(\frac {B (n)}{\delta}\right)}{n - 1} \Bigg | C\right) \geq 1 - \delta \\ \mathbb {P} \left(\forall Q, \operatorname {k l} \left(q \| _ {\theta \sim Q} \mathbb {E} _ {[ \mathcal {L} _ {\mathrm {c o v}} ]}\right) \leq \operatorname {k l} (q \| \alpha) \mid C\right) \geq 1 - \delta \\ \end{array}
$$
Plugging this result into (17), we have
$$
\mathbb {P} \left(\underset {\theta \sim Q} {\mathbb {E}} \left[ \mathcal {L} _ {\text {c o v}} \right] \leq \alpha \mid C\right) \geq 1 - \delta ,
$$
Next, we prove our bound on efficiency.
Proof of Theorem 2. We wish to bound  $\mathcal{L}_{\mathrm{eff}}(Q)$  as a function of the observed empirical efficiency. Define  $\hat{\mathcal{L}}_{\mathrm{eff}}(\theta, \tau, D_N)$  as the empirical mean efficiency observed for a particular value of parameter  $\theta$  and threshold  $\tau$ , and  $\mathcal{L}_{\mathrm{eff}}(\theta, \tau)$  as the expected efficiency of the prediction sets constructed with  $\theta$  and  $\tau$  on new data sampled from  $\mathcal{D}$ :
$$
\begin{array}{l} \hat {\mathcal {L}} _ {\mathrm {e f f}} (\theta , \tau , D _ {N}) := \frac {1}{N} \sum_ {i = 1} ^ {N} \ell_ {\mathrm {e f f}} (C (\boldsymbol {x} _ {i}; \tau , \theta)) \\ \mathcal {L} _ {\mathrm {e f f}} (\theta , \tau) := \underset {\boldsymbol {x} \sim \mathcal {D}} {\mathbb {E}} \left[ \ell_ {\mathrm {e f f}} (C (\boldsymbol {x}; \tau , \theta)) \right]. \\ \end{array}
$$
In ICP, the threshold  $\tau = \tau^{*}(D_{N},\hat{\alpha},\theta)$  itself depends on  $D_N$ . This means that even for a fixed  $\theta$ , the observed efficiency for the prediction set on each sample in  $D_N$  are not independent from one another. To avoid this complication, we define the quantity
$$
\hat {R} (\theta , D _ {N}) := \sup _ {\tau} \| \hat {\mathcal {L}} _ {\mathrm {e f f}} (\theta , \tau , D _ {N}) - \mathcal {L} _ {\mathrm {e f f}} (\theta , \tau) \|,
$$
which measures the worst-case difference between the empirical and true mean efficiencies over the worst-case choice of threshold. Note that
$$
\mathcal {L} _ {\text {e f f}} \left(\theta , \tau^ {*} \left(D _ {N}, \hat {\alpha}, \theta\right)\right) - \hat {\mathcal {L}} _ {\text {e f f}} \left(\theta , \tau^ {*} \left(D _ {N}, \hat {\alpha}, \theta\right), D _ {N}\right) \leq \hat {R} (\theta , D _ {N}). \tag {18}
$$
Next, we consider the expectation of this quantity over the sampling of  $D_N$ :
$$
R (\theta) := \underset {D _ {N}} {\mathbb {E}} \left[ \sup  _ {\tau} \| \hat {\mathcal {L}} _ {\mathrm {e f f}} (\theta , \tau , D _ {N}) - \mathcal {L} _ {\mathrm {e f f}} (\theta , \tau) \| \right]
$$
Given that, by assumption,  $\ell_{\mathrm{eff}}$  is  $L_{\tau}$ -Lipschitz in  $\tau$ , and  $\tau$  is bounded by  $\beta$  (due to  $s(\pmb{x},\pmb{y},\theta)$  being bounded), we use the result shown in Proposition 4 of Bai et al. [2022] to show that  $R(\theta)$  can be bounded for a fixed  $\theta$ :
$$
R (\theta) \leq \frac {2 \beta L _ {\tau}}{\sqrt {N}}. \tag {19}
$$
Furthermore, assuming  $\ell_{\mathrm{eff}}$  is bounded by [0, 1], then, for a fixed  $\theta$ ,  $\hat{R}(\theta, D_N)$  satisfies the bounded differences property with bound  $1/N$ . Therefore, applying McDairmid's inequality, we obtain
$$
\mathbb {P} _ {D _ {N}} \left(\left| \hat {R} (\theta , D _ {N}) - R (\theta) \right| \geq s\right) \leq 2 \exp (- 2 N s ^ {2})) \tag {20}
$$
This bound only holds for a fixed  $\theta$  independent of  $D_N$ .
To obtain a bound which holds for  $\theta \sim Q$  for all  $Q$ , we can follow the standard PAC-Bayes bound sub-Gaussian random variables. Specifically, we turn to Lemma 2, this time defining
$$
\Phi (\theta , D _ {N}) = 2 (N - 1) \left(\hat {R} (\theta , D _ {N}) - R (\theta)\right) ^ {2}
$$
Note that given (20), applying Lemma 1.5, we can bound
$$
\underset {D _ {N}} {\mathbb {E}} \left[ \exp (\Phi (\theta , D _ {N})) \right] \leq B (N) := 2 N.
$$
Now, we can apply Lemma 2 to obtain
$$
\underset {D _ {N}} {\mathbb {P}} \left(\forall Q: \underset {\theta \sim Q} {\mathbb {E}} \left[ 2 (N - 1) (\hat {R} (\theta , D _ {N}) - R (\theta)) ^ {2} \right] \leq \mathrm {K L} (Q \| P) + \log \left(\frac {2 N}{\gamma}\right)\right) \geq 1 - \gamma
$$
Bringing the expectation into the quadratic using Jensen's inequality, and rearranging terms, we obtain
$$
\underset {D _ {N}} {\mathbb {P}} \left(\forall Q: \underset {\theta \sim Q} {\mathbb {E}} \left[ \hat {R} (\theta , D _ {N}) \right] \leq \underset {\theta \sim Q} {\mathbb {E}} [ R (\theta) ] + \sqrt {\frac {\operatorname {K L} (Q \| P) + \log \left(\frac {2 N}{\gamma}\right)}{2 (N - 1)}}\right) \geq 1 - \gamma
$$
Now, substituting (19) and (18), we obtain that with probability greater than  $1 - \gamma$ , for all  $Q$ , it holds that