SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Luc De Raedt, Sebastijan Dumančić, Robin Manhaeve, and Giuseppe Marra. From statistical relational to neural-symbolic artificial intelligence. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 4943–4950, 2021.
[2] Artur d'Avila Garcez, Sebastian Bader, Howard Bowman, Luis C Lamb, Leo de Penning, BV Illuminoo, Hoifung Poon, and COPPE Gerson Zaverucha. Neural-symbolic learning and reasoning: A survey and interpretation. Neuro-Symbolic Artificial Intelligence: The State of the Art, 342:1, 2022.
[3] Eleonora Giunchiglia, Mihaela Catalina Stoian, and Thomas Lukasiewicz. Deep learning with logical constraints. arXiv preprint arXiv:2205.00523, 2022.
[77] Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit B Patel, and Anima Anandkumar. Semi-supervised stylegan for disentanglement learning. In Proceedings of the 37th International Conference on Machine Learning, pages 7360-7369, 2020.
[78] Hao Tang and Kevin Ellis. From perception to programs: regularize, overparameterize, and amortize. In International Conference on Machine Learning, pages 33616-33631. PMLR, 2023.
[79] Alessandro Daniele, Tommaso Campari, Sagar Malhotra, and Luciano Serafini. Deep symbolic learning: Discovering symbols and rules from perceptions. arXiv preprint arXiv:2208.11561, 2022.
[80] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338-5348. PMLR, 2020.
[81] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprint arXiv:2209.09056, 2022.
[82] Francesco Giannini, Michelangelo Diligenti, Marco Gori, and Marco Maggini. On a convex logic fragment for learning and reasoning. IEEE Transactions on Fuzzy Systems, 2018.
[83] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[84] Kareem Ahmed, Kai-Wei Chang, and Guy Van den Broeck. A pseudo-semantic loss for deep generative models with logical constraints. In Knowledge and Logical Reasoning in the Era of Data-driven Learning Workshop, July 2023.
[85] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[86] Tommaso Carraro. LTNtorch: PyTorch implementation of Logic Tensor Networks, mar 2022. URL https://doi.org/10.5281/zenodo.6394282.
[87] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980.
[88] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
A Other NeSy Predictors
In this appendix, we outline the NeSy prediction approaches used in our experiments and then show that they share deterministic RSs as DPL.
The semantic loss (SL) [8] is a penalty term that encourages a neural network to place all probability mass on predictions that are consistent with prior knowledge  $\mathsf{K}$ . In our setting, the SL is applied to a predictor  $p_{\theta}(\mathbf{Y} \mid \mathbf{C})$  placed on top of a concept extractor  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$  and it can be written as:
$$
\mathsf {S L} \left(p _ {\theta}, (\mathbf {x}, \mathbf {y}), K\right) := - \log \sum_ {\mathbf {c}} \mathbb {1} \left\{\left(\mathbf {c}, \mathbf {y}\right) \vDash K \right\} p _ {\theta} (\mathbf {c} \mid \mathbf {x}) \tag {11}
$$
Like DPL, the SL relies on knowledge compilation to efficiently implement Eq. (11). Importantly, if the distribution  $p_{\theta}(\mathbf{c} \mid \mathbf{x})$  allocates mass only to concepts  $\mathbf{c}$  that satisfy the knowledge given  $\mathbf{y}$ , these are optimal solutions, in the sense that the SL is exactly zero. We will make use of this fact in Appendix A.1.
During training, the SL is combined with any regular supervised loss  $\ell$ , for instance the cross-entropy, leading to the overall training loss:
$$
\frac {1}{| \mathcal {D} |} \sum_ {(\mathbf {x}, \mathbf {y}) \in \mathcal {D}} \ell \left(p _ {\theta}, (\mathbf {x}, \mathbf {y})\right) + \mu \mathrm {S L} \left(p _ {\theta}, (\mathbf {x}, \mathbf {y}), K\right) \tag {12}
$$
with  $\mu > 0$  a hyperparameter. During inference, the SL plays no role: the predicted label is obtained by simply taking the most likely configuration through a forward pass over the network.
Logic tensor networks (LTNs) [6] are another state-of-the-art NeSy architecture that combines elements of reasoning- and penalty-based approaches. The core idea behind LTNs, and of all other NeSy predictors based of fuzzy logic [82], is to relax the prior knowledge  $\mathsf{K}$  into a real-valued function  $\mathcal{T}[\mathsf{K}]$  quantifying how close a prediction is to satisfying  $\mathsf{K}$ . LTNs perform this transformation using product real logic [6]. In our context, this function takes the form  $\mathcal{T}[\mathsf{K}]:[0,1]^k\times [0,1]^n\to [0,1]$ , and it takes as input the probabilities of the various concepts  $\mathbf{C}$  and labels  $\mathbf{Y}$  and outputs a degree of satisfaction.
Crucially, fuzzy logics are designed such that, if all probability mass is allocated to configurations  $(\mathbf{c},\mathbf{y})$  that do satisfy the logic, then the degree of satisfaction is exactly 1, i.e., maximal. We will leverage this fact in Appendix A.1.
During training, LTNs guide the concept extractor  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$  towards predicting concepts that satisfy the prior knowledge  $\mathsf{K}$  by penalizing it proportionally to how far away their predictions are from satisfying  $\mathsf{K}$  given the ground-truth label  $\mathbf{y}$ , that is,  $1 - \mathcal{T}[\mathsf{K}](p(\mathbf{C} \mid \mathbf{x}), \mathbb{1}\{\mathbf{Y} = \mathbf{y}\})$ . During inference, LTNs first predict the most likely concepts  $\hat{\mathbf{c}} = \operatorname{argmax}_{\mathbf{c}} p_{\theta}(\mathbf{c} \mid \mathbf{x})$  using a forward pass through the network, and then predict a label  $\hat{\mathbf{y}}$  that maximally satisfies the knowledge given  $\hat{\mathbf{c}}$ , again according to  $\mathcal{T}[\mathsf{K}]$ .
A.1 Deterministic Optima are Shared
The bulk of our theoretical analysis focuses on DPL because it offers a clear probabilistic framework, however in the following we show that deterministic optima do transfer to other NeSy predictors approaches. Specifically, under A2, DPL, SL, and LTN all admit the same deterministic optima (det-opts).
To see this, let  $\mathcal{C}_{\mathbf{y}}$  be the set of concepts vectors  $\mathbf{c}$  from which the label  $\mathbf{y}$  can be inferred, that is,  $\mathcal{C}_{\mathbf{y}} = \{\mathbf{c} \in \mathcal{C} : p^{*}(\mathbf{y} \mid \mathbf{c}) > 0\}$ . Now, take a deterministic RS  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$  for DPL, i.e., a concept distribution that satisfies Definition 1 when the likelihood is computed as in Eq. (2). By determinism, this distribution can be equivalently written as a function mapping inputs  $\mathbf{x}$  to concept vectors  $\hat{\mathbf{c}}(\mathbf{x}) \in \mathcal{C}$ , that is,  $p_{\theta}(\mathbf{C} \mid \mathbf{X} = \mathbf{x}) = \mathbb{1}\{\mathbf{C} = \hat{\mathbf{c}}(\mathbf{x})\}$ . At the same time, by optimality and A2 we have that, for every  $(\mathbf{x}, \mathbf{y}) \in \mathcal{D}$ , the NeSy predictor  $p_{\theta}(\mathbf{Y} \mid \mathbf{X}; \mathsf{K}) = \sum_{\mathbf{c}} u_{\mathsf{K}}(\mathbf{Y} \mid \mathbf{c}) \cdot p_{\theta}(\mathbf{c} \mid \mathbf{X})$  allocates all probability mass to the correct label  $\mathbf{y}$ . As a consequence,  $\hat{\mathbf{c}}(\mathbf{x}) \in \mathcal{C}_{\mathbf{y}}$ , for every  $(\mathbf{x}, \mathbf{y}) \in \mathcal{D}$ .
Consider a NeSy predictor that first predicts concepts using  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$  and then labels using a reasoning layer based on fuzzy logic, such that a prediction  $\hat{\mathbf{y}} \in \mathcal{Y}$  is chosen so that it minimial distance from satisfaction w.r.t.  $K$ , and the distance from satisfaction is also used as a training objective. Since  $\hat{\mathbf{c}} \in \mathcal{C}_{\mathbf{y}}$  for every  $(\mathbf{x}, \mathbf{y}) \in \mathcal{D}$ , by definition of T-norms, the label with minimal distance from satisfaction will necessarily be  $\mathbf{y}$ : all other labels cannot be inferred from the knowledge,
so they have larger distance from satisfaction. Hence, training loss will be minimal for this predictor as well, meaning that  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$  is a deterministic RS for it too.
Next, consider a NeSy penalty-based predictor where predictions are obtained by first inferring concepts using MAP over the concept extractor, that is,  $\hat{\mathbf{c}} = \operatorname{argmax}_{\mathbf{c}} p_{\theta}(\mathbf{c} \mid \mathbf{X})$ , and then performing a forward pass over a neural prediction layer  $p_{\theta}(\mathbf{Y} \mid \mathbf{C})$ . By construction, the penalty will be minimal whenever  $(\mathbf{y}, \hat{\mathbf{c}}(\mathbf{x})) \models \mathsf{K}$ . We already established that this is the case for all  $(\mathbf{x}, \mathbf{y}) \in \mathcal{D}$ , meaning that  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$  is a deterministic RS for penalty-based predictors too.
In the proofs, we suppress  $\mathsf{K}$  from the notation for readability.
B.1 Proof of Lemma 1: Upper Bound of the Log-Likelihood
Proof plan. The proof of the first point of our claim is split in three parts:
1. We show that in expectation the log-likelihood in Eq. (2) is upper bounded by a term containing the KL divergence for  $p_{\theta}(\mathbf{Y} \mid \mathbf{G})$ .
2. We prove that, under A1, any optimum of Eq. (2) minimizes also the KL. In this step, we make use of Information Theory [83] to connect the two.
3. We show that assuming A1 and A2 the optima  $p_{\theta}(\mathbf{Y} \mid \mathbf{G})$  for the KL are given only by optima  $p_{\theta}(\mathbf{Y} \mid \mathbf{X})$  for Eq. (2).
We proceed to prove the second point by leveraging the fact that  $p_{\theta}(\mathbf{C} \mid \mathbf{G})$  is given by marginalizing  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$  over the generating distribution  $p^{*}(\mathbf{X} \mid \mathbf{G}, \mathbf{S})$ .
Point (i). (1) We upper bound the log-likelihood in Eq. (1) as follows:
$$
\begin{array}{l} \mathcal {L} \left(p _ {\theta}, \mathcal {D}\right) = \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \mathbb {E} _ {\mathbf {s} \sim p (\mathbf {S})} \mathbb {E} _ {\mathbf {x} \sim p ^ {*} (\mathbf {X} | \mathbf {g}, \mathbf {s})} \mathbb {E} _ {\mathbf {y} \sim p ^ {*} (\mathbf {Y} | \mathbf {g})} \left[ \log p _ {\theta} (\mathbf {y} \mid \mathbf {x}) \right] (13) \\ = \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \mathbb {E} _ {\mathbf {x} \sim p ^ {*} (\mathbf {X} | \mathbf {g})} \mathbb {E} _ {\mathbf {y} \sim p ^ {*} (\mathbf {y} | \mathbf {g})} \left(\log p _ {\theta} (\mathbf {y} \mid \mathbf {x})\right) (14) \\ \leq \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \mathbb {E} _ {\mathbf {y} \sim p ^ {*} (\mathbf {Y} | \mathbf {g})} \left(\log \mathbb {E} _ {\mathbf {x} \sim p ^ {*} (\mathbf {X} | \mathbf {g})} [ p _ {\theta} (\mathbf {y} \mid \mathbf {x}) ]\right) (15) \\ = \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \mathbb {E} _ {\mathbf {y} \sim p ^ {*} (\mathbf {Y} | \mathbf {g})} \left(\log p _ {\theta} (\mathbf {y} \mid \mathbf {g})\right) (16) \\ = \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \mathbb {E} _ {\mathbf {y} \sim p ^ {*} (\mathbf {Y} | \mathbf {g})} \left(\log \frac {p _ {\theta} (\mathbf {y} \mid \mathbf {g})}{p ^ {*} (\mathbf {y} \mid \mathbf {g})} - \log p ^ {*} (\mathbf {y} \mid \mathbf {g})\right) (17) \\ = \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \left(- \operatorname {K L} \left[ p ^ {*} (\mathbf {y} \mid \mathbf {g}) \| p _ {\theta} (\mathbf {y} \mid \mathbf {g}) \right] - \mathsf {H} \left[ p ^ {*} (\mathbf {y} \mid \mathbf {g}) \right]\right) (18) \\ \end{array}
$$
In the second line, we introduced the distribution  $p^*(\mathbf{x} \mid \mathbf{g}) \coloneqq \mathbb{E}_{\mathbf{s} \sim p(\mathbf{S})}[p^*(\mathbf{x} \mid \mathbf{g}, \mathbf{s})]$ , in the third line we applied Jensen's inequality, and in the fifth line we added and subtracted  $p^*(\mathbf{y} \mid \mathbf{g})$ . This proves our claim.
(2) We proceed showing that for a deterministic  $p^*(\mathbf{X} \mid \mathbf{G}, \mathbf{S})$  every optimum  $p_{\theta}(\mathbf{Y} \mid \mathbf{X})$  for the log-likelihood leads to an optimum  $p_{\theta}(\mathbf{Y} \mid \mathbf{G})$  for the RHS of Eq. (29). We can rewrite the joint distribution as:
$$
\begin{array}{l} p (\mathbf {x}, \mathbf {y}) := \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \mathbb {E} _ {\mathbf {s} \sim p (\mathbf {S})} \left[ p ^ {*} (\mathbf {x} \mid \mathbf {g}, \mathbf {s}) p ^ {*} (\mathbf {y} \mid \mathbf {g}) \right] (19) \\ = \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \left[ p ^ {*} (\mathbf {y} \mid \mathbf {g}) \mathbb {E} _ {\mathbf {s} \sim p (\mathbf {S})} \left[ p ^ {*} (\mathbf {x} \mid \mathbf {g}, \mathbf {s}) \right] \right] (20) \\ = \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \left[ p ^ {*} (\mathbf {y} \mid \mathbf {g}) p ^ {*} (\mathbf {x} \mid \mathbf {g}) \right] (21) \\ = \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \left[ p ^ {*} (\mathbf {y} \mid \mathbf {g}) p ^ {*} (\mathbf {g} \mid \mathbf {x}) p (\mathbf {x}) / p (\mathbf {g}) \right] (22) \\ = \mathbb {E} _ {\mathbf {g} \sim p ^ {*} (\mathbf {G} | \mathbf {x})} [ p ^ {*} (\mathbf {y} \mid \mathbf {g}) ] p (\mathbf {x}) (23) \\ = p ^ {*} (\mathbf {y} \mid \mathbf {x}) p (\mathbf {x}) (24) \\ \end{array}
$$
where  $p^{*}(\mathbf{y}\mid \mathbf{x}) = \mathbb{E}_{\mathbf{g}\sim p^{*}(\mathbf{g}|\mathbf{x})}[p^{*}(\mathbf{y}\mid \mathbf{g})]$  and  $p^{\ast}(\mathbf{G}\mid \mathbf{x})$  is the posterior underlying the data generative process. Hence, the log-likelihood in Eq. (1) can be equivalently written as:
$$
\begin{array}{l} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim p (\mathbf {X}, \mathbf {Y})} [ \log p _ {\theta} (\mathbf {y} \mid \mathbf {x}) ] = \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {X})} \mathbb {E} _ {\mathbf {y} \sim p ^ {*} (\mathbf {Y} | \mathbf {x})} [ \log p _ {\theta} (\mathbf {y} \mid \mathbf {x}) ] (25) \\ = \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {X})} \mathbb {E} _ {\mathbf {y} \sim p ^ {*} (\mathbf {Y} | \mathbf {x})} \left[ \log \frac {p _ {\theta} (\mathbf {y} \mid \mathbf {x})}{p ^ {*} (\mathbf {y} \mid \mathbf {x})} - \log p ^ {*} (\mathbf {y} \mid \mathbf {x}) \right] (26) \\ = \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {X})} \left(- \operatorname {K L} \left[ p ^ {*} (\mathbf {y} \mid \mathbf {x}) \| p _ {\theta} (\mathbf {y} \mid \mathbf {x}) \right] - \mathsf {H} \left[ p ^ {*} (\mathbf {y} \mid \mathbf {x}) \right]\right) (27) \\ \end{array}
$$
In the first line, we used  $p(\mathbf{X}, \mathbf{Y}) = p^{*}(\mathbf{Y} \mid \mathbf{X})p(\mathbf{X})$ , and then added and subtracted  $\log p^{*}(\mathbf{y} \mid \mathbf{x})$ . By comparing Eq. (18) and Eq. (27), we obtain:
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {X})} \left(- K L \left[ p ^ {*} (\mathbf {Y} \mid \mathbf {x}) \| p _ {\theta} (\mathbf {Y} \mid \mathbf {x}) \right] - H [ p ^ {*} (\mathbf {Y} \mid \mathbf {x}) ]\right) \\ \leq \mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \left(- \operatorname {K L} \left[ p ^ {*} (\mathbf {Y} \mid \mathbf {g}) \| p _ {\theta} (\mathbf {y} \mid \mathbf {g}) \right] - \mathbb {H} \left[ p ^ {*} (\mathbf {Y} \mid \mathbf {g}) \right]\right) \tag {28} \\ \end{array}
$$
Now, consider a distribution  $p_{\theta}(\mathbf{Y}\mid \mathbf{X})$  that attains maximal likelihood. Then,  $\mathrm{KL}[p^{*}(\mathbf{y}\mid \mathbf{x})\| p_{\theta}(\mathbf{y}\mid \mathbf{x})] = 0$ , and we can rearrange the inequality in Eq. (28) to obtain:
$$
\mathbb {E} _ {\mathbf {g} \sim p (\mathbf {G})} \left(\mathrm {K L} \left[ p ^ {*} (\mathbf {y} \mid \mathbf {g}) \| p _ {\theta} (\mathbf {y} \mid \mathbf {g}) \right]\right) \leq \mathrm {H} [ \mathbf {Y} \mid \mathbf {X} ] - \mathrm {H} [ \mathbf {Y} \mid \mathbf {G} ] \tag {29}
$$
Here,  $\mathsf{H}[\mathbf{Y}\mid \mathbf{X}] = \mathbb{E}_{\mathbf{x}\sim p(\mathbf{X})}[\mathsf{H}[p^{*}(\mathbf{Y}\mid \mathbf{x})]$  and  $\mathsf{H}[\mathbf{Y}\mid \mathbf{G}] = \mathbb{E}_{\mathbf{g}\sim p(\mathbf{G})}\mathsf{H}[p^{*}(\mathbf{Y}\mid \mathbf{g})]$  are conditional entropies.
We want to show that, under A1, the right-hand side of Eq. (29) is in fact zero. As for the other conditional entropy term, recall that [83]:
$$
\mathrm {H} [ \mathbf {Y} \mid \mathbf {X} ] = \mathrm {H} [ \mathbf {Y} \mid \mathbf {X} ] - \mathrm {H} [ \mathbf {Y} ] + \mathrm {H} [ \mathbf {Y} ] = - \mathrm {I} [ \mathbf {X}: \mathbf {Y} ] + \mathrm {H} [ \mathbf {Y} ] \tag {30}
$$
where  $\mathsf{I}[\cdot ,\cdot ]$  is the mutual information. By the chain rule of the mutual information, cf. [83, Theorem 2.8.1], we have:
$$
\left. \right.\left[ \mathbf {Y}: \mathbf {X}, \mathbf {G} \right] = \left. \right.\left[ \mathbf {Y}: \mathbf {X} \right] + \left| \right.\left[ \right. \mathbf {Y}: \mathbf {G} \left. \right| \mathbf {X} \left. \right\rbrack = \left. \right.\left[ \mathbf {Y}: \mathbf {G} \right] + \left| \right.\left[ \right. \mathbf {X}: \mathbf {Y} \left. \right| \mathbf {G} \left. \right\rbrack\left. \right. \tag {31}
$$
where  $\mathsf{I}[\cdot, \cdot \mid \cdot]$  is the conditional mutual information. The structure of our generative process in Fig. 1 implies that  $\mathsf{I}[\mathbf{X} : \mathbf{Y} \mid \mathbf{G}] = 0$ , so Eq. (31) boils down to:
$$
\left. \left| [ \mathbf {X}: \mathbf {Y} ] = \right. \right. \left. \left[ \mathbf {Y}: \mathbf {G} \right] - \right. \left. \left| [ \mathbf {Y}: \mathbf {G} \mid \mathbf {X} ] \right. \right. \tag {32}
$$
By A1, we have that:
$$
\begin{array}{l} \mathrm {I} [ \mathbf {Y}: \mathbf {G} \mid \mathbf {X} ] = \mathrm {H} [ \mathbf {G} \mid \mathbf {X} ] - \mathrm {H} [ \mathbf {G} \mid \mathbf {X}, \mathbf {Y} ] (33) \\ = \mathsf {H} \left[ f _ {1: k} ^ {- 1} (\mathbf {X}) \mid \mathbf {X} \right] - \mathsf {H} \left[ f _ {1: k} ^ {- 1} (\mathbf {X}) \mid \mathbf {X}, \mathbf {Y} \right] (34) \\ = \mathsf {H} \left[ f _ {1: k} ^ {- 1} (\mathbf {X}) \mid \mathbf {X} \right] - \mathsf {H} \left[ f _ {1: k} ^ {- 1} (\mathbf {X}) \mid \mathbf {X} \right] = 0 (35) \\ \end{array}
$$
Plugging this into Eq. (32) entails that  $\mathsf{I}[\mathbf{X}:\mathbf{Y}] = \mathsf{I}[\mathbf{Y}:\mathbf{G}]$ , or equivalently that  $\mathsf{H}[\mathbf{Y}:\mathbf{X}] = \mathsf{H}[\mathbf{Y}:\mathbf{G}]$ . This means that the right-hand side of Eq. (29) is indeed zero, which entails that the KL is zero and that therefore  $p_{\theta}(\mathbf{Y}\mid \mathbf{G})$  optimizes the right-hand side of Eq. (5).
(3) We proceed showing that by assuming also A2 whatever optimum  $p_{\theta}(\mathbf{Y} \mid \mathbf{G})$  is identified by  $p_{\theta}(\mathbf{Y} \mid \mathbf{X})$  that is also optimum. First, note that under A2  $p^{*}(\mathbf{Y} \mid \mathbf{G})$  is deterministic, so we have  $\mathsf{H}[\mathbf{Y} \mid \mathbf{G}] = 0$ . With A1 and A2, both  $p^{*}(\mathbf{y} \mid \mathbf{g})$  and  $p^{*}(\mathbf{y} \mid \mathbf{x})$  are deterministic and therefore Eq. (28) can be rewritten as:
$$
\mathbb {E} _ {\mathbf {x} \sim p (\mathbf {X})} \left[ \log p _ {\theta} \left(\mathbf {Y} = \left(\beta_ {\mathrm {K}} \circ f _ {1: k} ^ {- 1}\right) (\mathbf {x}) \mid \mathbf {x}\right) \right] \leq \mathbb {E} _ {\mathbf {g} \sim p ^ {*} (\mathbf {G})} \left[ \log p _ {\theta} \left(\mathbf {Y} = \beta_ {\mathrm {K}} (\mathbf {g}) \mid \mathbf {g}\right) \right] \tag {36}
$$
In particular, for each  $\mathbf{x}$  the maximum of the log-likelihood is 0 and it is attained when the label probability is one. We make use of this observation in the next step.
Next, we show that the bound in Eq. (28) is in fact tight, in the sense that whenever  $p_{\theta}(\mathbf{Y} \mid \mathbf{G})$  maximizes the left-hand side,  $p_{\theta}(\mathbf{Y} \mid \mathbf{X})$  maximizes the right-hand side. We proceed by contradiction. Fix  $\mathbf{g}$  and let  $\mathcal{O}$  be the set of those  $\mathbf{s}$  for which  $p_{\theta}(\mathbf{y} \mid \mathbf{x} = f(\mathbf{g}, \mathbf{s})) < 1$  and assume that it has
non-vanishing measure. Then, the posterior distribution is also strictly less than one:
$$
\begin{array}{l} p _ {\theta} (\mathbf {y} \mid \mathbf {g}) = \mathbb {E} _ {\mathbf {s} \sim p (\mathbf {S})} \mathbb {E} _ {\mathbf {x} \sim p ^ {*} (\mathbf {X} | \mathbf {g}, \mathbf {s})} [ p _ {\theta} (\mathbf {y} \mid \mathbf {x}) ] (37) \\ = \int_ {\mathbb {R} ^ {q}} p (\mathbf {s}) \int_ {\mathbb {R} ^ {d}} p _ {\theta} (\mathbf {y} \mid \mathbf {x}) p ^ {*} (\mathbf {x} \mid \mathbf {g}, \mathbf {s}) d \mathbf {x} d \mathbf {s} (38) \\ = \int_ {\mathbb {R} ^ {s}} p (\mathbf {s}) \int_ {\mathbb {R} ^ {d}} \delta \left\{\mathbf {x} - f (\mathbf {g}, \mathbf {s}) \right\} d \mathbf {x} d \mathbf {s} - \int_ {\mathbb {R} ^ {s}} p (\mathbf {s}) \int_ {\mathbb {R} ^ {d}} [ 1 - p _ {\theta} (\mathbf {y} \mid \mathbf {x}) ] \delta \left\{\mathbf {x} - f (\mathbf {g}, \mathbf {s}) \right\} d \mathbf {x} d \mathbf {s} (39) \\ = 1 - \int_ {\mathcal {O}} \left(1 - p _ {\theta} (\mathbf {y} \mid \mathbf {x} = f (\mathbf {g}, \mathbf {s}))\right) p (\mathbf {s}) d \mathbf {s} (40) \\ <   1 (41) \\ \end{array}
$$
Therefore, there cannot be any optimal solutions  $p_{\theta}(\mathbf{Y} \mid \mathbf{G})$  that are given by non-optimal probabilities  $p_{\theta}(\mathbf{Y} \mid \mathbf{X})$ . This proves the claim.
Point (ii). We consider now which distributions  $p_{\theta}(\mathbf{C} \mid \mathbf{G})$  correspond to a unique distribution  $p_{\theta}(\mathbf{C} \mid \mathbf{X})$ . First, we define as  $\mathcal{P}$  the set of candidate distributions  $p_{\varphi}(\mathbf{C} \mid \mathbf{X})$ , with  $\varphi \in \Theta$ , for which it holds:
$$
\mathbb {E} _ {\mathbf {s} \sim p ^ {*} (\mathbf {S})} \mathbb {E} _ {\mathbf {x} \sim p ^ {*} (\mathbf {X} | \mathbf {g}, \mathbf {s})} [ p _ {\varphi} (\mathbf {C} \mid \mathbf {x}) ] = p _ {\theta} (\mathbf {C} \mid \mathbf {g}) \tag {42}
$$
For all distributions of the form  $p_{\theta}(\mathbf{C}\mid \mathbf{G}) = \mathbb{1}\{\mathbf{C} = \mathbf{c}\}$ , for  $\mathbf{c}\in \mathcal{C}$ ,  $\mathcal{P}$  restricts to a single element, i.e.,  $p_{\varphi}(\mathbf{C}\mid \mathbf{x}) = \mathbb{1}\{\mathbf{C} = \mathbf{c}\}$ . We proceed by contradiction and consider a set  $\mathcal{O}_{\mathbf{X}}$  of non-vanishing measure such that:
$$
p _ {\varphi} (\mathbf {C} \mid \mathbf {x}) \neq \mathbb {1} \left\{\mathbf {C} = \mathbf {c} \right\}, \quad \forall \mathbf {x} \in \mathcal {O} _ {\mathbf {X}} \tag {43}
$$
Let  $p^*(\mathbf{X} \mid \mathbf{G}) = \mathbb{E}_{\mathbf{s} \sim p^*(\mathbf{S})} p^*(\mathbf{X} \mid \mathbf{G}, \mathbf{S})$ . Then, it holds that:
$$
\begin{array}{l} p _ {\varphi} (\mathbf {C} \mid \mathbf {G}) = \int_ {\mathcal {X}} p _ {\varphi} (\mathbf {C} \mid \mathbf {x}) p ^ {*} (\mathbf {x} \mid \mathbf {G}) \mathrm {d} \mathbf {x} \\ = \int_ {\mathcal {X} \backslash \mathcal {O} _ {\mathbf {X}}} \mathbb {1} \left\{\mathbf {C} = \mathbf {c} \right\} p ^ {*} (\mathbf {x} \mid \mathbf {G}) d \mathbf {x} + \int_ {\mathcal {O} _ {\mathbf {X}}} p _ {\varphi} (\mathbf {C} \mid \mathbf {x}) p ^ {*} (\mathbf {x} \mid \mathbf {G}) d \mathbf {x} \\ = (1 - \lambda) \cdot \mathbb {1} \left\{\mathbf {C} = \mathbf {c} \right\} + \lambda \cdot \tilde {p} _ {\varphi} (\mathbf {C} \mid \mathbf {G}) \tag {44} \\ \end{array}
$$