SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\gamma := \sup  _ {\mathcal {D} ^ {M} (\boldsymbol {x}, \boldsymbol {y}) > 0, \boldsymbol {y}, \boldsymbol {y} ^ {\prime} \text {a r e d i a g o n a l w i t h} \boldsymbol {y} \neq \boldsymbol {y} ^ {\prime}} \mathbb {P} _ {(\boldsymbol {x}, \boldsymbol {y}) \sim \mathcal {D} ^ {M}} \left(\boldsymbol {y} ^ {\prime} \in s\right) \tag {82}
$$
We say that a PLL instance satisfies the small  $M$ -ambiguity degree condition, if  $\gamma < 1$ .
Firstly, it can be seen that the  $M$ -ambiguity degree, see Definition 14, reduces to the ambiguity degree from Definition 13 when  $M = 1$ , since all label vectors are diagonal in that case. Below, we show that the small  $M$ -ambiguity degree condition guarantees learnability.
Proposition 2. If the small  $M$ -ambiguity degree condition is satisfied, then we have:
$$
\mathcal {R} ^ {0 1} (f) \leq \frac {1}{1 - \gamma} \left(c ^ {2 M - 2} \mathcal {R} _ {\mathrm {P}} ^ {0 1} (f; \sigma)\right) ^ {1 / M} \tag {83}
$$
Proof. Let  $\mathcal{E}$  be the event that for an input vector  $\pmb{x} \in \mathcal{X}^M$  with gold labels  $\pmb{y}$ , there exists a diagonal vector  $\pmb{y}' \in \mathcal{Y}^M$  with  $\pmb{y}' \neq \pmb{y}$ , such that  $\pmb{y}' \in s$ . Then, by definition, we have  $\mathbb{P}(\mathcal{E}) \leq \gamma$ . Also, conditioned on  $\mathcal{E}$ , from the proof of Lemma 1, we know that  $\mathcal{R}_{\mathbb{P}}^{01}(f) \geq \frac{\mathcal{R}^{01}(f)^M}{(c(c - 1))^{M - 1}}$ . The above implies that
$$
\mathcal {R} _ {\mathsf {P}} ^ {0 1} (f; \sigma) \geq (1 - \mathbb {P} (\mathcal {E})) \frac {\mathcal {R} ^ {0 1} (f) ^ {M}}{(c (c - 1)) ^ {M - 1}} \tag {84}
$$
$$
\mathcal {R} ^ {0 1} (f) \leq \frac {\left(\mathcal {R} _ {\mathsf {P}} ^ {0 1} (f ; \sigma) c ^ {2}\right) ^ {1 / M}}{1 - \mathbb {P} (\mathcal {E})} \leq \frac {\left(\mathcal {R} _ {\mathsf {P}} ^ {0 1} (f ; \sigma) c ^ {2}\right) ^ {1 / M}}{1 - \gamma}
$$
concluding the proof of Proposition 2.
The special case of Proposition 2, where  $M = 1$  recovers the classical result for standard PLL (see the discussion at the end of Section 3 in [28]). The partial risk  $\mathcal{R}_{\mathrm{P}}^{01}(f;\sigma)$  can be further bounded by its empirical counterpart, assuming that the Natarajan dimension is finite, as we have previously shown.
Partial Label Learning. Partial label learning (aka superset learning) assumes the learner receives the training data in the form of  $(x,z)$ , where  $z\in 2^{\mathcal{V}}$  is a subset of the label space that contains the gold label. Technically, it is technically possible to cast our problem to standard PLL by identifying  $s$  as its preimages, namely the set of all labels that is consistent with the observation  $s$ . Nevertheless, our work extends the standard PLL in two ways. Firstly, our framework allows multiple instances to appear in one training sample. Secondly, the learnability of PLL is typically shown under the small ambiguity assumption, see, for example, [11, 28, 5]. This condition requires no distracting label to be contained in  $z$  with probability 1, which is violated in our case since a distracting label can always occur because  $\sigma$  is deterministic. Therefore, our proposed conditions relax and generalize small ambiguity by allowing deterministic transitions, see Appendix E.3.
Another line of work in weak supervision exploits the transition matrices [46] to compute posterior class probabilities in different scenarios, e.g., PLL [8, 9] and noisy label learning [61]. The key assumption is that the transition matrix is invertible [8, 46]. Our learnability condition,  $M$ -unambiguity, and the invertibility condition do not imply each other, see Appendix E.1 and E.2 for a self-contained discussion. There, we also show that small ambiguity and the invertibility condition do not imply each other, which might be of independent interest.
Latent structure models. Our work also relates to latent structural learning, where the aim is to learn a latent representation using supervision on a deterministic transition  $\sigma$  over the latent variables [45, 52]. To reduce the cost of computing  $\sigma$ , [42] proposes approximations for a restricted class of decomposable transitions. Both [42, 35] assume latent models in the exponential family. Our work is also related to SPIGOT [34, 32]. There, the aim is to learn both the transition and the latent model, proposing techniques for back-propagating through the argmax layer of the latent model. Complementary to the above research, we focus on rigorous theoretical analysis.
Our work is closely related to [62], which studies a similar problem of weakly supervised learning with multiple instances. However, our results are stronger and closer to the aims of weak supervision in comparison to the results in [62]. This is because our work proposes sufficient and necessary conditions to recover the hidden labels (i.e.,  $Z_{1:K}$  in [62]), while consistency in [62] concerns the likelihood of the observed labels rather than the hidden ones, as defined in Definition 3 and 4 of [62], where two parameters are equivalent if they have the same likelihood on the observed partial label (rather than the hidden labels). Also, we use a different surrogate loss based on semantic loss that allows a top- $k$  approximation. In general, semantic loss is designed to capture Boolean constraints over the hidden labels and is standard to use in neuro-symbolic learning. Furthermore, our work directly extends the theory of PLL, which is thoroughly discussed in Appendix. In particular, our learnability condition from Definition 1 directly extends the classical small ambiguity degree condition, as shown in Appendix E.3. In contrast, the connection to PLL is missing from [62].
Constrained learning. The problem of partial label learning is closely related to constrained learning, in the sense that the prediction for the label (vector) is subject to the constraint  $\sigma(\pmb{y}) = s$  at the training stage. Training classifiers under constraints has been well studied in NLP. The work in [36] proposes a formulation for training under linear constraints; [37] proposes a learning framework that unifies expectation maximization with integer linear programming. Posterior regularization has been proposed for training classifiers under graphical models [21]. The latter two lines of research were adopted by [27] and [23] for neurosymbolic learning.
Neurosymbolic learning. Our work was motivated by frameworks that employ logic for training neural models [30, 53, 13, 59, 43, 31, 24, 27]. We prove learnability under (unknown) transitions that capture different languages: systems of Boolean formulas, non-linear constraints and logical theories. Notice that the key to model logical theories, e.g., Datalog, via  $\sigma$  is through abduction [26], see [43, 13] for a discussion. We also prove error bounds under a common neurosymbolic loss, the SL [56], and top- $k$  approximations. We are the first to provide rigorous results on neurosymbolic learning, closing a gap in the literature.
Notice that [58] also proves consistency of a top- $k$  loss. However, they use a zero-one-style loss. Finally, [3] provides generalization bounds of the smart predict-then-optimize (SPO) loss [15]; it assumes, though, that the gold labels of the inputs are given. Complementary to our research is the work in [19, 47, 33] that aims to integrate combinatorial solvers as differentiable layers in deep neural models. Finally, [48] proposes techniques for learning to solve combinatorial optimization problems using end-to-end neural models—loosing the ability to extract the latent models.
A subtle, yet substantial, difference between standard PLL and multi-instance PLL is that populating  $\sigma$  may incur a substantial computational overhead in the latter case. For instance, let us return back to Example 1. To populate the entries of  $\sigma$  for each target sum  $s$ , we need to compute all solutions of the equation  $X + Y = s$ , where  $X, Y \in \{0, \dots, 9\}$ . In contrast, the set of labels associated with each input instance is part of the input [28, 11].
Learning logical theories. Another relevant strand of research is that of learning logical theories [12, 16, 38]. The work in [12] mixes abduction with induction to jointly learn a theory while training the neural classifiers, while [16] introduces a differentiable formulation of inductive logic programming that relies on the semantics of fuzzy logic for interpreting formulas. Finally, the authors in [38] propose learning rules using the notion of logical neural networks. The techniques in [12, 16, 38] rely on templates for learning logical theories. Instead of relying on pre-specified templates, the authors in [18] aim to learn those templates by mining patterns (aka motifs) from the data, following mathematically rigorous techniques and providing formal guarantees related to the mined motifs. Differently from our work, the above line of research provides no guarantees in terms of learnability or error bounds.
Firstly, recall that in all neurosymbolic frameworks considered in Section 6, we considered weights in  $\{1,\dots ,5\}^{10}$  and used an additional neural classifier to learn the unknown weights. As the weights are fixed, the inputs to the weight classifiers in all frameworks are constant signals, i.e., in each iteration, we provided the classifier with the same signal  $W_{i}$  for the  $i$ -th weight. For each neurosymbolic framework, our implementation built upon the sources made available by the authors.
We now provide information about DeepProbLog, NeurASP and NeuroLog. Listings 1, 2 and 3 show the logic programs used to train the MNIST digit classifiers under the above three frameworks using the WEIGHTED-SUM problem, see Section 6.
DeepProbLog and NeurASP link the classifiers' predictions with the logic program via neural predicates. WEIGHTED-SUM requires two neural predicates: digit and weight. The first one classifies the input MNIST digits into  $\{0, \dots, 9\}$ , while the second one classifies the input weights into  $\{1, \dots, 5\}$ . In NeuroLog, the association between the classifiers' predictions and the logic program is maintained via abducible predicates. In particular, the instances of those predicates associate the probabilistic predictions with the facts from the domain of the logic program. Listing 3 uses two abducible predicates: atdigit and atweight. Atom atdigit(s, j) denotes that the  $j$ -th input to the digit classifier maps to value s. The abducible atom atweight(s, j) is analogously defined.
In terms of heuristics, in DeepProbLog, we used the Geometric Mean [31].
Following ENT's official implementation, we encoded WEIGHTED-SUM(2) using the Z3 SMT solver. In addition, similarly to the handwritten formula evaluation scenario in [27] (Section 5.1 from [27]), we used two projection operators. To generate the initial random labels, we projected out all the digit labels and kept only the weight labels. During random walks, we projected out the second
<sup>10</sup>In NeurASP, we used weights in  $\{1,\dots ,10\}$ , since due to a bug in the most recent NeurASP implementation that was provided by the authors, the output domains of all classifiers should be of the same size.
input digit's labels, keeping the labels of the first digit and the labels of both input weights. The rest of the hyperparameters, as well as the two-stage training process, were set as in the hand-written formula evaluation scenario from [27].
Neural architectures. The layers of the MNIST digit classifier are as follows: Conv2d(1, 6, 5), MaxPool2d(2, 2), ReLU(True), Conv2d(6, 16, 5), MaxPool2d(2, 2), ReLU(True), Linear(16 * 4 * 4, 120), ReLU(), Linear(120, 84), ReLU(), Linear(84, N), Softmax(1).
The neural classifiers for all frameworks but ABL were built using PyTorch 2.0.0 and Python 3.9. For ABL, we aligned with the implementation provided by the authors and used Tensorflow 2.11.0 and Keras 2.11.0. For each framework, we used the hyperparameters proposed by the authors in analogous scenarios.
nn(mnist_net,[X],Y,[0,1,2,3,4,5,6,7,8,9]):: digit(X,Y).  
nn(weight_net,[X],Y,[1,2,3,4,5]):: weight(X,Y).
weighted_sum2(I1, I2, W1, W2, S) :- digit(I1, D1), weight(W1, A1), digit(I2, D2), weight(W2, A2), S is A1*D1 + A2*D2.
weighted_sum3(I1, I2, I3, W1, W2, W3, S) :- digit(I1, D1), weight(W1, A1), digit(I2, D2), weight(W2, A2), digit(I3, D3), weight(W3, A3), S is A1*D1 + A2*D2 + A3*D3.
Listing 1: DeepProbLog program for the WEIGHTED-SUM problem.
weighted_sum4(I1, I2, I3, I4, W1, W2, W3, W4,S) :- digit(I1,D1), weight(W1,A1), digit(I2,D2), weight(W2,A2), digit(I3,D3), weight(W3,A3), digit(I4,D4), weight(W4,A4), S is A1*D1+A2*D2+A3*D3+A4*D4.
nn(digit(1,X), [0,1,2,3,4,5,6,7,8,9]) :- img(X).  
nn(weight(1,X), [1,2,3,4,5]) :- wei(X).  
img(i1). img(i2). img(i3). img(i4).  
wei(o1). wei(o2). wei(o3). wei(o4).
weighted_sum2(X1,X2,X3,X4,S) :- digit(0,X1,N1), weight(0,X3,W1), digit(0,X2,N2), weight(0,X4,W2), S=W1*N1+W2*N2.
weighted_sum3(X1,X2,X3,X4,X5,X6,S) :- digit(0,X1,N1), weight(0,X4,W1), digit(0,X2,N2), weight(0,X5,W2), digit(0,X3,N3), weight(0,X6,W3), S=W1*N1+W2*N2+W3*N3.
Listing 2:NeurASP program for the WEIGHTED-SUM problem.
weighted_sum4(X1,X2,X3,X4,X5,X6,X7,X8,S) :- digit(0,X1,N1), weight(0,X5,W1), digit(0,X2,N2), weight(0,X6,W2), digit(0,X3,N3), weight(0,X7,W3), digit(0,X4,N4), weight(0,X8,W4), S=W1*N1+W2*N2+W3*N3+W4*N4.
abducible(atdigit(_,)).  
abducible(atweight(_,)).  
digit(N) :- N in 0..9.  
weight(N) :- N in 1..5.
weighted_sum2(S) :- digit(D1), atdigit(D1,1), digit(D2), atdigit(D2,2), weight(S1), atweight(S1,1), weight(S2), atweight(S2,2), S#= (D1 * S1) + (D2 * S2).
weighted_sum3(S) :- digit(D1), atdigit(D1,1), digit(D2), atdigit(D2,2), digit(D3), atdigit(D3,3), weight(S1), atweight(S1,1), weight(S2), atweight(S2,2), weight(S3), atweight(S3,3), S #= (D1 * S1) + (D2 * S2) + (D3 * S3).
weighted_sum4(S) :- digit(D1), atdigit(D1,1), digit(D2), atdigit(D2,2), digit(D3), atdigit(D3,3), digit(D4), atdigit(D4,4), weight(S1), atweight(S1,1), weight(S2), atweight(S2,2), weight(S3), atweight(S3,3), weight(S4), atweight(S4,4), S#= (D1 * S1) + (D2 * S2) + (D3 * S3) + (D4 * S4).
Listing 3: NeuroLog program for the WEIGHTED-SUM problem.