SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our empirical analysis shows that a key issue is scalability. This is a known problem in neuro-symbolic learning and it is partially because the relevant problems in logic are intractable. For instance, semantic loss requires computing the models of Boolean formulas, which is #P-complete. Reasoning over logical theories can be also intractable [2], if not undecidable. Scalability straightforwardly affects accuracy, e.g., NLog obtains better accuracy over DLog-A for  $M = 3$  (see Table 2) due to its ability to explore the whole search space. It is also worth stressing that Scallop could not scale beyond  $M = 4$  for  $k > 1$ . The above challenges bring up new research directions [44, 41]. Closing this discussion, it is worth exploring why ABL and ENT led to low accuracy<sup>4</sup>.
4The authors from [43] also reported low accuracy for ABL for some scenarios.
Table 1: Classifier accuracy for WEIGHTED-SUM for  $M = 2$ . #0, #8 and #16 are # of directly labeled samples used for pre-training. Inside the parentheses are the # of (weak) training samples.
SL(#4K)	SL(#8K)	DLog(#4K)	NLog(#4K)	NASP(#4K)	ABL(#10K)	ENT(#10K)
#0	93% ± 0.01	96% ± 0.008	96% ± 0.004	97% ± 0.05	98% ± 0.01	9% ± 0.05	40% ± 15.2
#8	93% ± 0.01	96% ± 0.008	96% ± 0.005	97% ± 0.005	98% ± 0.01	10% ± 0.01	43% ± 17.11
#16	93% ± 0.01	96% ± 0.008	96% ± 0.001	97% ± 0.01	98% ± 0.01	11% ± 0.1	49% ± 16.4
Table 2: Classifier accuracy for WEIGHTED-SUM for  $M \in  \{ 2,3,4\}$  . #0 and #16 are # of directly labeled samples used for pre-training. Inside the parentheses are the # of (weak) training samples.
M=2,mP=4K	M=3,mP=4K	M=4,mP=10K
#0	#16	#0	#16	#0	#16
DLog-A	34% ± 0.1	80% ± 0.06	24% ± 0.09	27% ± 0.13	18% ± 0.02	26% ± 0.25
NLog	97% ± 0.05	97% ± 0.01	61% ± 0.07	97% ± 0.01	N/A	N/A
NLog(5)	48% ± 0.003	87% ± 0.007	27% ± 0.06	84% ± 0.06	29% ± 0.1	46% ± 0.05
Scallop(1)	97% ± 0.05	97% ± 0.01	21% ± 3.5	97% ± 0.01	36% ± 0.2	54% ± 0.02
Scallop(4)	97% ± 0.05	97% ± 0.05	53% ± 0.15	97% ± 0.07	51% ± 0.09	78% ± 0.05
We provide an overview of the literature that closely relates to our work. A more comprehensive review of PLL, latent structural learning, and neuro-symbolic learning, is presented in Appendix F.
Partial label learning. Learnability of PLL is typically shown under the small ambiguity assumption [11, 28, 5]. While it is technically possible to cast our problem to standard PLL by viewing a target  $s$  as a partial label for the hidden labels  $\pmb{y}$ , the small ambiguity condition is violated in our case since a distracting label can occur because  $\sigma$  is deterministic. Therefore, our proposed conditions relax and generalize the small ambiguity by allowing deterministic transitions, see Appendix E.3. Another line of work in weak supervision exploits the invertibility of transition matrices [46] to compute posterior class probabilities [8, 9, 61]. Our learnability condition,  $M$ -unambiguity, and the invertibility condition do not imply each other, see Appendix E.1 and E.2. There, we also show that small ambiguity and the invertibility condition do not imply each other, which might be of independent interest. Another related topic is learning with label constraints [21, 6, 49], where the label of each instance  $x$  (possibly structured) in the dataset is constrained in a subset  $C \subseteq \mathcal{V}$ . The difference is that the constraint mapping itself is known to the learner and hence can be encoded in the inference algorithm directly, for example, via the CCM [6].
Neuro-symbolic and structured learning. We were motivated by frameworks that employ logic for training neural models [30, 53, 13, 59, 43, 31, 24, 27, 23]. We prove learnability under (unknown) transitions that capture different languages and error bounds under the SL [56] and top- $k$  approximations. Notice that [58] also proves the consistency of a top- $k$  loss. However, they use a zero-one-style loss for standard supervised learning. Our work is closely related to [62], which studies a similar problem of weakly supervised learning with multiple instances. Our results are stronger in the sense that we propose sufficient and necessary conditions to recover the hidden labels, while [62] concerns the likelihood of the observed labels rather than the hidden ones. Our work is also relevant to latent structural prediction [42, 35, 34, 32]. To our knowledge, no formal learning guarantees have been given for that problem. Complementary to ours is the work in [19, 47, 33] that integrates combinatorial solvers into deep models.
We formulated multi-instance PLL and showed its connections with latent structural and neuro-symbolic learning. Our work exhibits a greater level of flexibility compared to (single-instance) PLL, as it allows for multiple instances and deterministic transitions. We introduced minimal assumptions that enabled us to establish ERM-learnability and derive error bounds with the top- $k$  semantics loss. Our findings suggest that the interaction of multiple instances in forming the observed labels can help relax the learnability assumptions in weakly supervised learning. For future work, we will further relax the learning conditions by assuming a certain degree of smoothness for the data distributions. Another direction is to explore the setting where the hidden labels are non-i.i.d. or structured.
This work was partially supported by Contract FA8750-19-2-0201 with the US Defense Advanced Research Projects Agency (DARPA). Approved for Public Release, Distribution Unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
This work was also partially sponsored by the Army Research Office and was accomplished under Grant Number W911NF-20-1-0080. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
This work was also supported by Contract FA8750-19-2-1004 with the US Defense Advanced Research Projects Agency (DARPA). Approved for Public Release, Distribution Unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
This work was also partially funded by ONR Contract N00014-19-1-2620.
[1] S. Abiteboul, R. Hull, and V. Vianu. Foundations of Databases. Addison Wesley, 1995.
[2] Jean-François Baget, Marie-Laure Mugnier, Sebastian Rudolph, and Michael Thomazo. Walking the complexity lines for generalized guarded existential rules. page 712-717, 2011.
[3] Othman El Balghiti, Adam N. Elmachtoub, Paul Grigas, and Ambuj Tewari. Generalization bounds in the predict-then-optimize framework. In NeurIPS, pages 14389–14398, 2019.
[4] Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. The Journal of Machine Learning Research, 3:463-482, 2003.
[5] Vivien Cabannes, Alessandro Rudi, and Francis Bach. Structured prediction with partial labelling through the infimum loss. In ICML, page 1230-1239, 2020.
[6] Ming-Wei Chang, Lev Ratinov, and Dan Roth. Structured learning with constrained conditional models. Machine Learning, 88(3):399–431, 2012.
[7] Mark Chavira and Adnan Darwiche. On probabilistic inference by weighted model counting. Artificial Intelligence, 172(6):772 - 799, 2008.
[8] Jesús Cid-Sueiro. Proper losses for learning from partial labels. In NeurIPS, pages 1574-1582, 2012.
[9] Jesús Cid-Sueiro, Dario García-García, and Raul Santos-Rodríguez. Consistency of losses for learning from weak labels. In ECML PKDD, pages 197-210, 2014.
[10] Corinna Cortes, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. Structured Prediction Theory Based on Factor Graph Complexity. In NeurIPS, 2016.
[11] Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12:1501-1536, 2011.
[12] Wang-Zhou Dai and Stephen Muggleton. Abductive knowledge induction from raw data. In Zhi-Hua Zhou, editor, *IJCAI*, pages 1845–1851. AAAI, 2021.
[13] Wang-Zhou Dai, Qiuling Xu, Yang Yu, and Zhi-Hua Zhou. Bridging Machine Learning and Logical Reasoning by Abductive Learning. In NeurIPS, pages 2815-2826, 2019.
[14] Artur S. d'Avila Garcez, Krysia Broda, and Dov M. Gabbay. Neural-symbolic learning systems: foundations and applications. Perspectives in neural computing. Springer, 2002.
[55] Zhaofeng Wu. Learning with latent structures in natural language processing: A survey. arXiv preprint arXiv:2201.00490, 2022.
[56] Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In ICML, pages 5502-5511, 2018.
[57] Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. In NeurIPS, volume 34, pages 27119-27130, 2021.
[58] Forest Yang and Sanmi Koyejo. On the consistency of top-k surrogate losses. In ICML, volume 119, pages 10727-10735, 2020.
[59] Zhun Yang, Adam Ishay, and Joohyung Lee. NeurASP: Embracing neural networks into answer set programming. In IJCAI, pages 1755-1762, 2020.
[60] Peilin Yu, Tiffany Ding, and Stephen H. Bach. Learning from multiple noisy partial labelers. In PMLR, volume 151, pages 11072-11095, 2022.
[61] Mingyuan Zhang, Jane Lee, and Shivani Agarwal. Learning from noisy labels with no change to the training process. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 12468-12478, 2021.
[62] Yivan Zhang, Nontawat Charoenphakdee, Zheng Wu, and Masashi Sugiyama. Learning from aggregate observations. ArXiv, abs/2004.06316, 2020.
- In Appendix A, we provide detailed definitions for the loss functions and the complexity measures we use in our work.
- In Appendix B, C and D, we provide the proofs to all formal statements.
- In Appendix F, we provide a comprehensive review of the related work in PLL, latent structural learning, constrained learning, and neuro-symbolic learning.
- In Appendix E, we compare common distributional assumptions in the standard PLL literature with ours. In particular:
1. In Appendix E.1, we provide a transition matrix [8, 46] formulation of multi-instance PLL and show that  $M$ -unambiguity does not imply transition matrix invertibility for multi-instance PLL and vice versa.
2. In Appendix E.2, we show a result of independent interest: that transition matrix invertibility does not imply small ambiguity [28] and vice versa.
3. In Appendix E.3, we show that  $M$ -unambiguity, the unambiguity notion proposed in Definition 1, is an extension of the small ambiguity degree [28], by considering a generalized setting where  $\sigma$  is stochastic.
- In Appendix G, we provide implementation details of our experiments.
Table 3: The  $2^{4}$  interpretations of  $\{A_{1,2}, A_{2,0}, A_{1,0}, A_{2,2}\}$  and their corresponding truth probabilities subject to the  $\omega$  and  $\varphi$  from Example 9. Interpretations that are not models of  $\varphi$  have zero probability.
A1,2	A2,0	A1,0	A2,2	Pφ(I,ω)
⊥	⊥	⊥	⊥	0
⊥	⊥	⊥	T	0
⊥	⊥	T	⊥	0
⊥	⊥	T	T	(1 - ω(A1,2)) × (1 - ω(A2,0)) × ω(A1,0) × ω(A2,2)
⊥	T	⊥	⊥	0
⊥	T	⊥	T	0
⊥	T	T	⊥	0
⊥	T	T	T	(1 - ω(A1,2)) × ω(A2,0) × ω(A1,0) × ω(A2,2)
T	⊥	⊥	⊥	0
T	⊥	⊥	T	0
T	⊥	T	⊥	0
T	⊥	T	T	ω(A1,2) × (1 - ω(A2,0)) × ω(A1,0) × ω(A2,2)
T	T	⊥	⊥	ω(A1,2) × ω(A2,0) × (1 - ω(A1,0)) × (1 - ω(A2,2))
T	T	⊥	T	ω(A1,2) × ω(A2,0) × (1 - ω(A1,0)) × ω(A2,2)
T	T	T	⊥	ω(A1,2) × ω(A2,0) × ω(A1,0) × (1 - ω(A2,2))
T	T	T	T	ω(A1,2) × ω(A2,0) × ω(A1,0) × ω(A2,2)
In this section, we introduce in further detail some key notions used in the main body of our paper. We start with the definition of the semantic loss [56].
A.1 Loss functions
Semantic Loss (SL) [56] has been adopted to train classifiers subject to logical theories [30, 43, 24]. SL is the cross entropy of the weighted model counting (WMC) [7] of a formula subject to a softmax output. Below, we provide the necessary notions. Let  $\mathcal{Z}$  denote a set of Boolean variables. An interpretation  $I$  of  $\mathcal{Z}$  is a mapping from each  $A\in \mathcal{Z}$  to true  $(\top)$  or false  $(\bot)$ . Interpretation  $I$  is a model of a Boolean formula  $\varphi$ , if  $\varphi$  evaluates to true in  $I$ . We use the notation  $A\in \varphi$  to denote that Boolean variable occurs in  $\varphi$ . By treating each Boolean variable  $A$  occurring in  $\varphi$  as an independent Bernoulli random variable that becomes true with probability  $\omega(A)$  and false with probability  $1 - \omega(A)$ , the different interpretations of  $\mathcal{Z}$  induce a probability distribution, where the probability  $P_{\varphi}(I,\omega)$  of each interpretation  $I$  subject to  $\omega$  and  $\varphi$  is given by:
$$
P _ {\varphi} (I, \omega) := \left\{ \begin{array}{l l} \prod_ {A \in \varphi | I (A) = \top} \omega (A) \cdot \prod_ {A \in \varphi | I (A) = \bot} (1 - \omega (A)) & I \text {i s a m o d e l o f} \varphi , \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {11}
$$
The WMC  $\mathrm{WMC}(\varphi, \omega)$  of  $\varphi$  under  $\omega$  then denotes the probability of  $\varphi$  being satisfied, i.e., it evaluates to true, under  $\omega^5$ , and is defined as:
$$
\operatorname {W M C} (\varphi , \omega) := \sum_ {\text {I n t e r p r e t a t i o n I o f t h e v a r i a b l e s i n \varphi .}} P _ {\varphi} (I, \omega) \tag {12}
$$
Assuming that each variable in  $\varphi$  is associated with a class from  $\mathcal{V}$ , and treating  $f(x)$  as a mapping from the variables in  $\varphi$  to the  $f$ 's scores on  $x$ , the semantic loss of  $\varphi$  under  $f(x)$  is defined as:
$$
\operatorname {S L} (\varphi , f (x)) := - \log (\operatorname {W M C} (\varphi , f (x))) \tag {13}
$$
We demonstrate the above concepts via Example 4.
Example 9 (Example 4, cont'd). Recall that the Boolean formula induced by the partial label  $s = 2$  is  $\varphi = (A_{1,2} \wedge A_{2,0}) \vee (A_{1,0} \wedge A_{2,2})$ , where  $A_{i,j}$  is a Boolean variable that is true iff the  $i$ -th input digit has label  $j$ . Given a probabilistic prediction  $f(\pmb{x})$ , variable  $A_{i,y}$  is assigned probability  $f^y(x_i)$ , inducing a probability mapping  $\omega := \bigcup_{i \in \{1,2\}, y \in \{0,\dots,9\}} \{A_{i,y} \mapsto f^y(x_i)\}$ . Table 3 shows all
interpretations of  $\{A_{1,2}, A_{2,0}, A_{1,0}, A_{2,2}\}$  and their corresponding probabilities subject to  $\omega$  and  $\varphi$ . The WMC of  $\varphi$  under  $\omega$  is the sum of the probabilities of all the  $2^4$  interpretations.
To establish SL, each  $(x_{i},y)$ , where  $\pmb {x} = (x_{1},\dots ,x_{M})$ ,  $i\in [M]$  and  $y\in \mathcal{V}^{M}$ , is associated with a distinct Boolean variable  $A_{i,y}$ .
The nature of multi-instance PLL implies Boolean formulas in disjunctive normal form (DNF), where a formula is in DNF form, if it is a disjunction over conjunctions of Boolean variables or their negations, see Example 4. In fact, each Boolean formula computed out of the labels associated with a partial label is a positive DNF formula, i.e., a DNF formula where no variable occurs negatively. Using the union bound of standard probability theory and the model-based semantics of formula probabilities [20], the following result directly follows:
Lemma 2. Let  $\Phi_1, \ldots, \Phi_K$  be  $K$  positive conjunctive formulas over the set of Boolean variables  $\mathcal{Z}$  and  $\omega: \mathcal{Z} \mapsto (0,1)$ . Assuming that each variable  $Z \in \mathcal{Z}$  is treated as an independent Bernoulli random variable that becomes true with probability  $\omega(Z)$  and false with probability  $1 - \omega(Z)$ , the following holds:
$$
\operatorname {W M C} \left(\bigvee_ {i = 1} ^ {K} \Phi_ {i}, \omega\right) \leq \sum_ {i = 1} ^ {K} \operatorname {W M C} \left(\Phi_ {i}, \omega\right) \tag {14}
$$
where the equality holds when the  $\Phi_i$ 's are logically inconsistent, i.e., there is no interpretation of  $\mathcal{Z}$  in which two or more conjunctions are simultaneously true.
Let  $\Sigma$  denote exclusiveness constraints among the variables occurring in the  $\Phi_i$ 's. For instance, returning back to Example 9, to align with the semantics of the classifier, only of the Boolean variables  $A_{1,2}$  and  $A_{1,0}$  can be true in each interpretation<sup>6</sup>. Then,  $\mathrm{WMC}(\bigvee_{i=1}^{K} \Phi_i \wedge \Sigma, \omega) \leq \mathrm{WMC}(\bigvee_{i=1}^{K} \Phi_i, \omega)$ . This is because  $\Sigma$  reduces the number of models of  $\varphi$ .
In addition, when the  $\Phi_i$ 's share no Boolean variables, the following holds [20]:
$$
\operatorname {W M C} \left(\bigvee_ {i = 1} ^ {K} \Phi_ {i}, \omega\right) = 1 - \prod_ {i = 1} ^ {K} \left(1 - \operatorname {W M C} \left(\Phi_ {i}, \omega\right)\right) \tag {15}
$$
Due to (15), the top- $k$  partial loss becomes
$$
\ell_ {\sigma} ^ {k} (f (\boldsymbol {x}), s) = - \log \left(1 - \prod_ {i = 1} ^ {k} \left(1 - P _ {f (\boldsymbol {x})} \left(\boldsymbol {y} ^ {(i)}\right)\right)\right) \tag {16}
$$
when the  $\pmb{y}^{(i)}$ 's in Definition 3 share no Boolean variables.
5Notice that WMC also applies to the case where  $\omega$  maps each Boolean variable to a non-negative real number. However, when restricting to (0, 1), the WMC of a Boolean formula equals to its probability [20].
The above can be captured via the integrity constraint  $\neg (A_{1,2}\wedge A_{1,0})$
A.2 VC and Natarajan dimensions
We use the following definitions of VC and Natarajan dimensions, which can be found in [40].
Definition 6 (Shattering: multiclass classifiers). Let  $\mathcal{H}$  be a space of multiclass classifiers of the form  $\mathcal{X} \to \mathcal{Y}$ . A set  $C \subset \mathcal{X}$  is shattered by  $\mathcal{H}$ , if there exist two functions  $h_0, h_1: C \to \mathcal{Y}$ , such that
- For each  $x \in C$ , we have that  $h_0(x) \neq h_1(x)$ .
- For each  $B \subset C$ , there exists a function  $h \in \mathcal{H}$ , such that
$$
\forall x \in B, h (x) = h _ {0} (x) \quad a n d \quad \forall x \in C \backslash B, h (x) = h _ {1} (x)
$$
Definition 7 (Natarajan Dimension). The Natarajan dimension of  $\mathcal{H}$ , denoted by  $\mathrm{Nat}(\mathcal{H})$ , is the maximal size of a shattered set  $C \subset \mathcal{X}$ . If  $\mathcal{H}$  is a space of binary classifiers, then its Natarajan dimension is also called its VC dimension and is denoted by  $\mathrm{VC}(\mathcal{H})$ .
A.3 Rademacher complexity
We use the following definition of the (empirical) Rademacher complexity of a multiclass scoring space  $\mathcal{F}$ , which is adapted from [10].
Definition 8 (Rademacher complexity). Let  $\mathcal{T} = \{x_1, \ldots, x_m\}$  be a set of instances that are i.i.d. drawn from  $\mathcal{D}_X$ . The empirical Rademacher complexity of  $\mathcal{F}$  with respect to  $\mathcal{T}$  is defined by
$$
\hat {\mathfrak {R}} _ {m} (\mathcal {F}; \mathcal {T}) := \frac {1}{m} \mathbb {E} _ {\epsilon} \left[ \sup  _ {f \in \mathcal {F}} \sum_ {i = 1} ^ {m} \sum_ {y \in \mathcal {Y}} \epsilon_ {i, y} f ^ {y} \left(x _ {i}\right) \right] \tag {17}
$$
where the  $\epsilon_{i,y}$ 's are i.i.d. Rademacher random variables, each of which is uniformly distributed over  $\{-1, +1\}$ , and  $\epsilon$  is a vector over all the  $\epsilon_{i,y}$ 's. Furthermore, the Rademacher complexity of scoring class  $\mathcal{F}$  is the expectation of the empirical version:
$$
\Re_ {m} (\mathcal {F}) := \mathbb {E} _ {\mathcal {T} \sim \mathcal {D} _ {X} ^ {m}} [ \widehat {\Re} _ {m} (\mathcal {F}; \mathcal {T}) ] \tag {18}
$$
Before proceeding with the proofs, we will introduce some useful notation.
Definition 9 (Probability of misclassification). Let  $f$  be a scoring function in  $\mathcal{F}$  and  $l_i, l_j$  be two labels in  $\mathcal{Y}$ . Then, the probability that  $f$  misclassifies label  $l_i$  as  $l_j$  subject to samples drawn from  $\mathcal{D}$  is defined as
$$
E _ {l _ {i}, l _ {j}} (f) := \mathbb {P} _ {(x, y) \sim \mathcal {D}} ([ f ] (x) = l _ {j} \cap y = l _ {i}) \tag {19}
$$
For convenience, here we consider the general setting where the transition function  $\sigma$  is unknown to the learner but instead, a transition class  $\mathcal{G}$  is provided. The results for the special case where  $\sigma$  is known to the learner can be derived by setting  $\mathcal{G} = \{\sigma\}$ . Now, fix a scoring function class  $\mathcal{F}$  and a transition class  $\mathcal{G}$ . We define  $\mathcal{H}_{\mathcal{F},\mathcal{G}}$  as the class of binary classifiers that, given samples of the form  $(x,s)$ , for  $x \in \mathcal{X}^M$  and  $s \in S$ , output zero, if the predictions for  $x$  abide by the partial label  $s$  and one, otherwise. Formally:
$$
\mathcal {H} _ {\mathcal {F}, \mathcal {G}} := \left\{\left(\boldsymbol {x}, s\right) \mapsto \mathbb {1} \left\{\sigma^ {\prime} ([ f ] (\boldsymbol {x})) \neq s \right\} \mid f \in \mathcal {F}, \sigma^ {\prime} \in \mathcal {G} \right\} \tag {20}
$$
For the case where the transit is known to the learner, we also denote  $\mathcal{H}_{\mathcal{F},\mathcal{G}} = \mathcal{H}_{\mathcal{F}}$  for simplicity. From (20), it follows that the zero-one binary classification loss of a classifier  $f_{\sigma^{\prime}}\in \mathcal{H}_{\mathcal{F},\mathcal{G}}$ , equals the zero-one partial loss subject to  $\sigma^\prime$  defined as  $\ell_{\sigma '}^{01}(f(\pmb {x}),s)\coloneqq \mathbb{1}\{\sigma '(f(\pmb {x}))\neq s\}$ . We now present the following lemma that bounds the VC dimension of  $\mathcal{H}_{\mathcal{F},\mathcal{G}}$ , which is reminiscent of the arguments of [28, 50].
Lemma 3. Fix a scoring function class  $\mathcal{F}$  and a transition class  $\mathcal{G}$ . Let  $\mathrm{Nat}([\mathcal{F}]) = d_{[\mathcal{F}]} < \infty$  be the Natarajan dimension of  $[\mathcal{F}]$  and  $d_{\mathcal{G}}$  be the VC dimension of the class of classifiers  $\{(y,s)\mapsto \mathbb{1}\{\sigma^{\prime}(\pmb {y})\neq s\} |\sigma^{\prime}\in \mathcal{G}\}$ , where  $\pmb {y}\in \mathcal{Y}^{M}$  and  $s\in S$ . The VC dimension of class  $\mathcal{H}_{\mathcal{F},\mathcal{G}}$  is bounded as:
$$
d \leq 2 \left(\left(d _ {[ \mathcal {F} ]} + d _ {\mathcal {G}}\right) \log \left(6 M \left(d _ {[ \mathcal {F} ]} + d _ {\mathcal {G}}\right)\right) + 2 d _ {[ \mathcal {F} ]} \log c\right) \tag {21}
$$
Proof. Suppose that the VC dimension of  $\mathcal{H}_{\mathcal{F},\mathcal{G}}$  is  $d$ . Let  $N$  be the maximum number of distinct ways to assign labels in  $\mathcal{V}^M$  to  $d$  points in  $\mathcal{X}^M\times S$  using  $[\mathcal{F}]$ . Each possible label assignment leads to a set of  $d$  elements in  $\mathcal{X}^M\times \mathcal{Y}^M\times S$ , and for any  $d$  elements, by Sauer-Shelah lemma (see, for example, [40]'s Lemma 6.10), there are at most
$$
\left(\frac {\mathrm {e} M d}{d _ {\mathcal {G}}}\right) ^ {d _ {\mathcal {G}}}
$$
ways for  $\mathcal{G}$  to decide if  $\mathbb{1}\{\sigma^{\prime}(\pmb {y})\neq s\}$ , where  $\mathrm{e}$  is the base of the natural logarithm. Based on the above, we have:
$$
2 ^ {d} \leq N \left(\frac {\mathrm {e} M d}{d _ {\mathcal {G}}}\right) ^ {d _ {\mathcal {G}}}
$$
Therefore,  $N\geq 2^{d}\left(d_{\mathcal{G}} / \mathrm{e}Md\right)^{d_{\mathcal{G}}}$
By Natarajan's lemma of multiclass classification, we have:
$$
N \leq (M d) ^ {d _ {[ \mathcal {F} ]}} c ^ {2 d _ {[ \mathcal {F} ]}} \tag {22}
$$
Combining (22) with the above equations, it follows that
$$
(M d) ^ {d _ {[ \mathcal {F} ]}} c ^ {2 d _ {[ \mathcal {F} ]}} \geq N \geq 2 ^ {d} \left(\frac {d _ {\mathcal {G}}}{\mathrm {e} M d}\right) ^ {d _ {\mathcal {G}}}
$$
Taking the logarithm on both sides, we have:
$$
d _ {[ \mathcal {F} ]} \log (M d) + 2 d _ {[ \mathcal {F} ]} \log c \geq d \log 2 + d _ {\mathcal {G}} (\log d _ {\mathcal {G}} - \log (M d) - 1)
$$
Rearranging the inequality yields: