Title: On Learning Latent Models with Multi-Instance Weak Supervision
Venue: NeurIPS 2023
Accepted: yes
Keywords: weak supervision

partial label learning

neuro-symbolic learning

latent structural learning
Abstract: We consider a weakly supervised learning scenario where the supervision signal is generated by a transition function $\sigma$ of labels associated with multiple input instances. We formulate this problem as *multi-instance Partial Label Learning (multi-instance PLL)*, which is an extension to the standard PLL problem. Our problem is met in different fields, including latent structural learning and neuro-symbolic integration. Despite the existence of many learning techniques, limited theoretical analysis has been dedicated to this problem. In this paper, we provide the first theoretical study of multi-instance PLL with possibly an unknown transition $\sigma$. Our main contributions are as follows: First, we proposed a necessary and sufficient condition for the learnability of the problem. This condition nontrivially generalizes and relaxes the existing *small ambiguity degree* in PLL literature since we allow the transition to be deterministic. Second, we derived Rademacher-style error bounds based on the top-$k$ surrogate loss that is widely used in the neuro-symbolic literature. Furthermore, we conclude with empirical experiments for learning with an unknown transition. The empirical results align with our theoretical findings; however, they also expose the issue of scalability in the weak supervision literature.

On Learning Latent Models with Multi-Instance Weak Supervision

Abstract

We consider a weakly supervised learning scenario where the supervision signal is generated by a transition function σ of labels associated with multiple input instances. We formulate this problem as multi-instance Partial Label Learning (multi-instance PLL) . Our problem is an extension to the standard PLL problem and is met in different fields, including latent structural learning and neuro-symbolic integration. Despite the existence of many learning techniques, limited theoretical analysis has been dedicated to this problem. In this paper, we provide the first theoretical study of multi-instance PLL with possibly an unknown transition σ . Our main contributions are as follows. First, we propose a necessary and sufficient condition for the learnability of the problem. This condition nontrivially generalizes and relaxes the existing small ambiguity degree in PLL literature since we allow the transition to be deterministic. Second, we derive Rademacher-style error bounds based on a topk surrogate loss that is widely used in the neuro-symbolic literature. Furthermore, we conclude with empirical experiments for learning under unknown transitions. The empirical results align with our theoretical findings, exposing also the issue of scalability in the weak supervision literature.

1 Introduction

Consider the scenario in Figure 1, where a learner aims to learn one or more classifiers f 1 , . . . , f n , each of which maps an instance x to its corresponding label y . The learner is given training examples, each of which consists of a vector of M > 1 instances x = ( x 1 , . . . , x M ) , and each x i is processed by one of the f j 's. Differently from supervised learning, the gold labels y = ( y 1 , . . . , y M ) of x are hidden ; instead, the learner is provided with a weak label s which is produced by applying a transition function 1 σ to the gold labels y . The transition σ itself may be unknown to the learner.

The above weak supervision setting has been a topic of active research in NLP [42, 35, 34, 32, 45, 52, 22]. Recently, it has received renewed attention as it is met in neuro-symbolic frameworks [14], i.e., frameworks that integrate inference and learning over neural models with inference and learning over symbolic models, such as logical theories [30, 53, 13, 59, 43, 31, 24, 27]. The benefits of this setting over architectures that approximate the f i 's and σ via an end-to-end neural model [48] are (i) the ability to reuse the latent models, something particularly useful in NLP [34, 32], (ii) higher accuracy in multiple tasks, e.g., NLP [55] and visual question answering [24], (iii) greater interpretability and (iv) the ability to encode prior knowledge via σ . Below, we present an example of our learning setting in neuro-symbolic learning.

Example 1 (SUM2) . We aim to learn an MNIST classifzer f . Differently from supervised learning, we use training examples of the form ( x 1 , x 2 , s ) , where x 1 and x 2 are MNIST digits and s is

their sum. The above implies a transition function σ : { 0 , . . . , 9 } × { 0 , . . . , 9 } → { 0 , . . . , 18 } , s.t. σ ( y, y ′ ) = y + y ′ . The target sum s restricts the space of labels that can be assigned to the input pair of digits ( x 1 , x 2 ) , e.g., if s = 2 , then the only combinations of labels that abide by the constraint that the sum is 2 are (2,0), (1,1) and (0,2). This problem is referred to as SUM2 in literature [24, 30].

The transition function σ is not necessarily one-toone. For instance, multiple combinations of digits can lead to the same target sum. The above leaves us with two questions to answer: can we provide any learning guarantees for the classifiers f 1 , . . . , f n ? and what if σ is unknown? Despite that multiple techniques for neuro-symbolic and latent structural learning have been recently proposed, those theoretical questions remain open. Practically, for the single input case ( M = 1 ), our setting can reduce to that of partial label learning (PLL), where each input instance is accompanied by a set of labels, including the gold one [25, 11, 5, 29, 39, 54, 57, 60]. However, a key PLL learnability assumption - that the probability a

x

1

x

2

.

.

.

x

M

f

1

f

2

.

.

.

f

n

y

1

y

2

.

.

.

y

M

σ

s

wrong label co-occurs with the gold one is always less than one [28]- is violated in our setting, rendering existing learnability results inapplicable. This is because σ is a deterministic but not necessarily bijective function, as opposed to σ in PLL which is randomized.

Due to its relevance to PLL, we refer to our learning setting as multi-instance PLL . Differently from prior art, e.g., [42, 35], we do not restrict σ to specific types of functions, e.g., linear functions. Hence, we can express constraints in several formal languages, including systems of Boolean equations and Datalog [1], a language widely used in neuro-symbolic techniques [24]. Notice that many neuro-symbolic learning techniques are oblivious to the implementation of the symbolic component: abstracting the symbolic component as a transition σ has been actually proposed as the means to compositionally integrate neural with symbolic models [43]. Furthermore, although we primarily consider deterministic transitions motivated by the neuro-symbolic and NLP literature, our results can be extended to support randomized transitions when σ is known.

We aim to make minimal assumptions on the data distributions by showing learnability even under the 'toughest" distributions. In addition, we provide learning guarantees under the semantic loss [56], a widely-used surrogate loss for training classifiers subject to logical theories [30, 43, 24]. To our knowledge, we are the first to provide this theoretical analysis, closing a gap in the neuro-symbolic and latent structural learning literature.

Contributions. Our contributions can be summarized into the following:

We propose necessary and sufficient learnability conditions of the multi-instance PLL problem assuming all the instances are classified by a single classifier f and the transition is known. We further provide a Rademacher-style error bound using a topk approximation to the semantic loss [56], see Section 3. Our analysis confirms the intuition that a larger k decreases the empirical risk, but increases the number of samples required for learning.

We extend the above results for the more general case where one learns multiple classifiers f = ( f 1 , . . . , f n ) that classify instances from different domains, see Section 4.

We prove learnability with a single classifier and an unknown transition function, see Section 5, and assess the validity of our theoretical results using SOTA neuro-symbolic frameworks, see Section 6.

Proofs, additional backgrounds and details on our empirical analysis are in the appendix.

2 Preliminaries

We use /a80 ( · ) to denote probability mass, ❊ [ · ] to denote expectation and ✶ {·} to denote an indicator function. For a positive integer M , we use [ M ] as a short for { 1 , . . . , M } . For a vector of M elements ( x 1 , . . . , x M ) , a position j is a number in [ M ] used to identify the element x j . A vector is said to be diagonal if its elements in all positions are equal. Suppose f is a function on a domain X and x is a vector of M elements in X , we use the symbol f ( x ) to denote the vector ( f ( x 1 ) , . . . , f ( x M )) .

Classifiers. Let X denote an instance space and Y denote an output space with |Y| = c . Let also D be the joint distribution of two random variables ( X,Y ) ∈ X × Y and D X be the marginal of X . Throughout the text, we use x for ( x 1 , . . . , x M ) and y for the corresponding vector of gold labels. We consider scoring functions of the form f : X → ∆ c , where ∆ c is the space of probability distributions on Y (e.g., f outputs the softmax probabilities of a neural network). We use f j ( x ) to denote the j -th output of f ( x ) . A scoring function f induces a classifier whose prediction on x [ f ]( x ) is defined by [ f ]( x ) := argmax j ∈ [ c ] f j ( x ) . We use F and [ F ] to denote the space of scoring functions and the space of classifiers induced by F , respectively. We use the VC-dimension, the Natarajan dimension and Rademacher complexities to characterize the complexity of F [40].

Loss functions. We define the problem of learning a scoring function using a fixed set of training samples as the one of choosing the scoring function from F that better fits the training samples. We give semantics to the term fits by using the risk R ( f ; ℓ ) of f subject to a sampling distribution and a given loss function ℓ : the lower R ( f ; ℓ ) becomes, the better f fits the data. When there exists an f ∗ ∈ F , such that R ( f ∗ ; ℓ ) = 0 , we say that the space F is realizable under ℓ . In supervised learning , we are provided with samples of the form ( x, y ) drawn from D . We use ̂ R to denote the empirical risk , i.e., the average risk computed over the training samples only. The risk 2 of f subject to a loss function ℓ : Y × Y → R + is given by R ( f ; ℓ ) · · = ❊ ( X,Y ) ∼D [ ℓ ([ f ]( X ) , Y )] .

̸

A commonly used loss function is the zero-one loss defined by ℓ 01 ( y, y ′ ) · · = ✶ { y ′ = y } for any pair y, y ′ ∈ Y . We use R 01 ( f ) as a short for R ( f ; ℓ 01 ) and refer to it as the zero-one risk of f . We will also consider Semantic Loss (SL) [56], which has been adopted to train classifiers subject to Boolean formulas [30, 43, 24]. Let φ be a Boolean formula where each variable in φ is associated with a class from Y . Then, the SL of φ subject to f ( x ) is the negative logarithm of the weighted model counting (WMC) [7] of φ under f ( x ) , namely SL( φ, f ( x )) · · = -log(WMC( φ, f ( x ))) , where WMC( φ, f ( x )) takes values in (0 , 1) and denotes the probability that φ is logically satisfied when its variables become true with probabilities specified by f ( x ) .

3 Learning a Single Classifier Under a Known Transition

We begin with the setting where the goal is to learn a single classifier f ∈ F under a known transition σ . Firstly, we show ERM-learnability (Theorem 1). Then, inspired by neuro-symbolic learning, we provide a Rademacher-style error bound with a topk surrogate loss based on WMC (Theorem 2). Below, we formally introduce the learning setting.

̸

Problem setting. Let σ : Y M →S be a transition function, where S = { 1 , . . . , d } with |S| = d ≥ 1 . Let also T P be a set of m P partially labeled samples of the form ( x , s ) = ( x 1 , . . . , x M , s ) . Each training sample is formed by, firstly drawing M i.i.d. samples ( x i , y i ) from D and then setting s = σ ( y 1 , . . . , y M ) . We use D P to denote the distribution followed by the training samples ( x , s ) . In analogy to the zero-one classification loss ℓ 01 , we define the zero-one partial loss subject to σ as ℓ 01 σ ( y , s ) · · = ✶ { σ ( y ) = s } , for any y ∈ Y M and s ∈ S . The learner aims to find the classifier f with the minimal zero-one risk R 01 ( f ) . As the gold labels are hidden, the learner uses the dataset T P to estimate and minimize the zero-one partial risk of f subject to σ defined as R 01 P ( f ; σ ) · · = ❊ ( X 1 ,...,X M ,S ) ∼D P [ ℓ 01 σ (([ f ]( X 1 ) , . . . , [ f ]( X M )) , S )] .

We demonstrate these notions via Example 1. There, f is an MNIST classifier, X is the space of MNIST images and Y = { 0 , . . . , 9 } . The training samples are of the form ( x 1 , x 2 , s ) where s = σ ( y 1 , y 2 ) = y 1 + y 2 . The partial risk of f is then the probability of predicting the wrong sum.

Learnability. A partial learning algorithm A takes a partially labeled dataset T P with |T P | = m P as input and outputs a function A ( T P ) ∈ F . Similar to standard PAC learning (see, for example, [40]), we say a multi-instance problem is learnable , if there exists a partial learning algorithm A , such that for any data distribution D over X × Y and any δ, ϵ ∈ (0 , 1) , there is an integer m ϵ,δ , such that m P ≥ m ϵ,δ implies R 01 ( A ( T P )) ≤ ϵ with probability at least 1 -δ with respect to D P . Weconsider A to be an Empirical Risk Minimizer (ERM) algorithm that finds the classifier f ∈ F which minimizes the empirical zero-one partial risk ̂ R 01 P ( f ; σ ; T P ) := ∑ ( x ,s ) ∈T P ℓ 01 σ (([ f ]( x 1 ) , . . . , [ f ]( x M )) , s ) /m P .

3.1 ERMLearnability

To prove learnability under the partial zero-one loss, we aim to bound R 01 ( f ) with the partial risk R 01 P ( f ; σ ) . Firstly, we propose a sufficient and necessary condition called M -unambiguity .

̸

Recall that learnability requires handling all possible distributions of ( x , y ) . Therefore, to propose a necessary condition to guarantee learnability, we consider a special case: the one in which D concentrates its mass on a single item x 0 , i.e., /a80 D X ( x 0 ) = 1 . In that case, the only sample observed during learning includes only the instance x 0 . Now, let l 0 ∈ Y be the gold label of x 0 . If f mispredicts the class of x 0 , i.e., f ( x 0 ) = l 0 , and this misprediction leads to the same observed partial label, i.e., σ ( f ( x 0 ) , . . . , f ( x 0 )) = σ ( l 0 , . . . , l 0 ) , then the problem will be not learnable, as we will never be able to correct this error. Motivated by this special case, we propose the following condition:

̸

Definition 1 ( M -unambiguity) . Transition σ is M -unambiguous if for any two diagonal label vectors y and y ′ ∈ Y M such that y = y ′ , we have that σ ( y ′ ) = σ ( y ) .

̸

Recall that a vector is diagonal if all its elements are equal. Below, we provide examples of transitions that satisfy (or not) the M -unambiguity condition.

Example 2. Consider the following learning problems.

(SUMM ) : Consider a variant of Example 1 where s is the sum of M digits, i.e., σ ( y 1 , . . . , y M ) = ∑ i M =1 y i . The transition is M -unambigous, since σ ( y, . . . , y ) = σ ( y ′ , . . . , y ′ ) holds for any y = y ′ .

̸

̸

̸

(PRODUCTM ) : Consider a variant of Example 1 where s is given by σ ∗ ( y 1 , . . . , y M ) = ∏ i M =1 y i . Then, σ ∗ is M -unambiguous, since σ ∗ ( y, . . . , y ) = y M = ( y ′ ) M holds for any y = y ′ .

̸

̸

(XOR) : Consider Boolean labels. We take the XOR of the two digits as the partial label, i.e., σ XOR ( y 1 , y 2 ) = ✶ { y 1 = y 2 } . Then σ XOR is not M -unambiguous, since σ XOR (1 , 1) = σ XOR (0 , 0) .

Although M -unambiguity is proposed as a necessary condition under special types of data distributions, we now show that it is also sufficient for proving ERM-learnability. Firstly, we prove that the classification risk can be bounded by the partial risk.

Lemma 1. If σ is M -unambigous, then we have:

Moreover, if σ is not M -unambiguous, then learning from partial labels is arbitrarily difficult, in the sense that a classifier f with partial risk R 01 P ( f ; σ ) = 0 can have a risk of R 01 ( f ) = 1 .

To show learnability, we bound the partial risk with its empirical counterpart in the realizable case.

Theorem 1 (ERM learnability under M -unambiguity) . Suppose F is realizable under ℓ 01 P and [ F ] has a finite Natarajan dimension d [ F ] . Then for any ϵ, δ ∈ (0 , 1) , there exists a universal 3 constant C 0 > 0 , such that with probability at least 1 -δ , the empirical partial risk minimizer with ̂ R 01 P ( f ; σ ; T P ) = 0 has a classification risk R 01 ( f ) < ϵ , if

Beyond Lemma 1, Theorem 1 builds upon several non-trivial intermediate results: (i) A bound of the VC dimension for the partial label predictor (Lemma 3 in the Appendix) and (ii) Construction of counter-examples for arguing the necessity of M -unambiguity (see the proof of Theorem 1).

Toward a faster convergence rate. Theorem 1 presents a rather slow convergence rate of ( c 2 /ϵ ) M . In the following, we show that a better convergence rate is achievable by forcing stricter ambiguity conditions. In the following, we introduce the concept of 1-unambiguity , which requires the transition to be sensitive to 1-position perturbations:

̸

Definition 2 (1-unambiguity) . Transition σ is 1-unambiguous , if there exists an index 1 ≤ i ≤ M , such that flipping the i -th label of any label vector y ∈ Y M , results in a vector y ′ with σ ( y ′ ) = σ ( y ) .

In the following, we provide examples of transitions that satisfy (or not) the 1 -unambiguity condition.

Example 3. Let us continue with Example 2. The transition σ from SUMM is 1-unambiguous since σ ( y ) = ∑ i M =1 y i always change when replacing any of the labels y i . However, the transition σ ∗ from PRODUCTM is not 1-unambiguous, since for the label vector y = (0 , 1 , 1) (with product zero), the flipped vector y ′ = (0 , 1 , 2) also leads to product zero.

We show that a better convergence rate can be achieved if σ is both 1- and M -unambigous.

Proposition 1 (ERM learnability under 1- and M -unambiguity) . If σ is both 1- and M -unambigous, then we have:

Furthermore, if F is realizable under ℓ 01 P and [ F ] has a finite Natarajan dimension d [ F ] , then for any δ ∈ (0 , 1) and ϵ ∈ (0 , 1) that is sufficiently close to 0, there exists a universal constant C 1 , such that with probability at least 1 -δ , the empirical partial risk minimizer with ̂ R 01 P ( f ; σ ) = 0 has a classification risk R 01 ( f ) < ϵ , if

Remark 1. The supervision power of a partial label s depends on the 'instability" of σ : if σ returns different results under certain perturbations in I positions for an I ∈ [ M ] , then the classification risk is bounded by R 01 ( f ) ≤ O ( R 01 P ( f ; σ ) 1 /I ) . To formalize this intuition, in the appendix, we provide a generalized definition of both 1- and M -unambiguity, called I -unambiguity, and show learnability with intermediate convergence rates. Notice that M -unambiguity is closely related to the small ambiguity degree condition in the PLL literature [28]. Appendix E.3 provides a relevant discussion and an extension of our results for randomized transitions.

Remark 2. We use the concentrated distribution described at the beginning of Section 3.1 as a pivot to design a necessary and sufficient learnability condition. However, the results in Lemma 1, Theorem 1 and Proposition 1 do not apply only to this distribution, but to any distribution D P of partial data. The same applies to all learnability results that follow.

3.2 Error Bounds with the Topk Semantic Loss

We now study the error bounds with the semantic loss (see Section 2 and Appendix A) under topk approximations (Theorem 2). We are interested in this loss for two reasons. Firstly, risk minimization under the zero-one loss is intractable even for linear classifiers [17] and hence in practice, learning is performed by minimizing differentiable surrogate losses such as the semantic loss. Secondly, computing a surrogate loss typically introduces another source of intractability: that of enumerating the entries of σ [24, 43, 31]. For example, in SUMM , we have to enumerate 10 M digit combinations to populate σ , which is not scalable.

A popular way to reduce the second source of inefficiency is to consider only the k most probable label combinations during training, where the probability of a label vector y given x and classifier f is defined by P f ( x ) ( y ) · · = ∏ i M =1 f y i ( x i ) . Then, training can proceed by taking the semantic loss over the topk label vectors [24]. The above is possible, as the topk label vectors can be equivalently viewed as a formula that is true iff one or more of the topk label vectors is the gold one.

Example 4. Let us return back to Example 1. Instead of considering all three combinations (2 , 0) , (1 , 1) and (0 , 2) assuming a target s = 2 , we can consider only the first and the last combination, if the probabilities (predicted by f ) of the two images being 1 are way lower than those of being 0 or 2. Then, the top-2 label vectors (2 , 0) , (0 , 2) can be seen as the formula ( A 1 , 2 ∧ A 2 , 0 ) ∨ ( A 1 , 0 ∧ A 2 , 2 ) , where A i,j is a Boolean variable that is true iff the i -th input digit is assigned label j .

We use ∨ k i =1 y i to denote the Boolean formula computed out of the k label vectors y 1 , . . . , y k , where each y i is the conjunction of Boolean variables of the form A i,y , as in Example 4. Given a softmax score f ( x ) , variable A i,y is assigned probability f y ( x i ) . We now define the topk partial loss :

Definition 3 (Topk partial loss) . The topk partial loss for an integer k ≥ 1 subject to a classifier f , transition σ and sample ( x , s ) is defined as

where y (1) , . . . , y ( k ) ∈ Y M are the topk maximizers of P f ( x ) in the preimage of s , i.e., { y ∈ Y M : σ ( y ) = s } . The topk partial classification risk of f subject to σ is then given by

Remark. The WMC is upper bounded by the sum of probabilities, so we can approximate the SL as ℓ k σ ( f ( x ) , s ) ≥ -log( ∑ k i =1 P f ( x ) ( y ( i ) )) . Furthermore, the special case where k = 1 reduces to the infimum loss [5] and minimal loss [29] in the PLL literature as we show in Appendix B.2.

Let R m ( F ) denote the Rademacher complexity [4, 10] of F with m samples as defined in Appendix A. We are now ready to bound the zero-one classification risk with the empirical topk partial risk.

Theorem 2 (Error bound under unambiguity) . Let an integer k ≥ 1 and δ ∈ (0 , 1) . If σ is both 1and M -unambiguous, then with probability at least 1 -δ , we have:

where ̂ R k P ( f ; σ ; T P ) = ∑ ( x ,s ) ∈T P ℓ k σ ( f ( x ) , s ) /m P is the empirical counterpart of (6) and Φ is an increasing function that satisfies lim t → 0 Φ( t ) /t = 1 .

Proof sketch. Theorem 2 builds upon several results. Firstly, we derived an inequality that bounds the topk loss with the zero-one loss (Lemma 5), which requires the construction of an intermediate ℓ 1 loss (Definition 11). Secondly, we show the Lipschtness of the semantic loss (Lemma 7). This result is further combined with a contraction lemma that is proposed in [10] (Lemma 6) to bound the Rademacher complexity of the model.

Remark. Similarly to Theorem 1 and Proposition 1, Theorem 2 suggests that the learning difficulty increases as M increases. This is intuitive, since it gets harder to disambiguate the gold labels when M increases, e.g., for 2-SUM, there are 10 2 possible label vectors need to be considered, while for 4-SUM, there are 10 4 ones. Our bound also suggests a tradeoff with the choice of k : a larger k decreases the risk, but tends to increase the complexity term.

4 Learning Multiple Classifiers Under a Known Transition

In this section, we extend the learning problem from Section 3 to jointly learn n ≥ 1 different classifiers. Similarly to Section 3, we aim to propose minimal assumptions for the data distribution to show both results on ERM-learnability (Theorem 3) and a Rademacher-style error bound with the topk loss (Theorem 4). We formally define the learning setting below.

Problem setting. We aim to learn n ≥ 1 classifiers, each of which maps instances from a space X i to the corresponding label space Y i with |Y i | = c i . For each i ∈ [ n ] , we define a scoring space F i , which contains mappings of the form f i : X i → ∆ c i . Each f i is used to classify M i ≥ 1 instances ( x i 1 , . . . , x iM i ) = x i ∈ X M i i with (hidden) gold labels y i = ( y i 1 , . . . , y iM i ) . We denote the vector of scoring functions as f = ( f 1 , . . . , f n ) . Let D i be the joint distribution over elements from X i ×Y i . Each training sample given to the learner ( x 1 , . . . , x n , s ) is formed by (i) drawing M i i.i.d. samples ( x i , y i ) from D i , for each i ∈ [ n ] and then (ii) obtaining its partial label as s = σ ( y 1 , . . . , y n ) . We override the notation D P to denote the distribution of the training samples in this learning setting. The partially labeled dataset T P then contains m P i.i.d. samples drawn from D P .

Similarly to Section 3, the learner aims to find the classifiers f 1 , . . . , f n with the minimal zeroone risk defined as R 01 ( f ) · · = ∑ n i =1 R 01 ( f i ) . As the gold labels are hidden, the learner uses the dataset T P to estimate and minimize the zero-one partial risk of f 1 , . . . , f n , which is defined as R 01 P ( f 1 , . . . , f n ; σ ) := ❊ ( X 1 ,..., X n ,S ) ∼D P [ ℓ 01 σ (([ f 1 ]( X 1 ) , . . . , [ f n ]( X n )) , S )] , where [ f i ]( X i ) is a short for ([ f i ]( X i 1 ) , . . . , [ f i ]( X iM i )) , for 1 ≤ i ≤ M i .

Example 5 (Learning binary operators) . Consider a setting where we aim to train a classifier f 1 for recognizing MNIST digits and a classifier for recognizing images of addition and multiplication taken from some space X 2 . The training samples are of the form ( x 11 , x 12 , x 21 , s ) , where x 11 and x 12 are non-zero MNIST digits with Y 1 = { 1 , . . . , 9 } , x 21 is an image of an operator with Y 2 = { + , ×} , and s is the result of applying the operator in x 2 on the digits in x 11 and x 12 .

4.1 ERMLearnability

Similarly to Section 3, to propose a necessary condition for learnability, we consider the most challenging distribution where each D i is concentrated on a single instance x ∗ i ∈ X i , for 1 ≤ i ≤ n . Then, the only sample that will be observed during learning is ( x ∗ 1 , . . . , x ∗ n ) where x i = ( x ∗ i , . . . , x ∗ i ) , for 1 ≤ i ≤ n . Assuming that l ∗ i is the gold label of x ∗ i , detecting a classification error is possible only if a misclassification of any x ∗ i , i.e., [ f i ]( x ∗ i ) = l ∗ i , will lead to a prediction vector y ′ with a different image under σ from the gold label y , i.e., σ ( y ′ ) = σ ( y ) . The above intuition is captured via the notion of multi-unambiguity , which generalizes the notion of M -unambiguity. Below, we use y M i i , for i ∈ [ n ] , to denote the M i -ary diagonal vector including the y i element only.

̸

̸

Definition 4 (Multi-unambiguity) . Transition σ is multi-unambiguous if for any vector y = ( y M 1 1 , . . . , y M n n ) , and any position i ∈ [ n ] , such that the vector y ′ that results after flipping the labels in y M i i to some diagonal vector ( y ′ i ) M i = y M i i , has a different image under σ , i.e., σ ( y ) = σ ( y ′ ) .

̸

Multi-unambiguity reduces to M -unambiguity for n = 1 . An example of multi-unambiguity is below.

Example 6 (Learning binary operator, cont'd) . In Example 5, the multi-unambiguity condition is violated since 2 + 2 = 2 × 2 . Namely, if a distribution assigns all its weight to the digit 2 , then it is difficult for the model to distinguish the two operators. On the other hand, if instead, the label space is Y 1 = { 3 , . . . , 9 } , then the multi-unambiguity condition is satisfied.

Bounded risk assumption. Differently from Section 3, multi-unambiguity alone is not sufficient to ensure learnability. To see this, consider a variant of SUM2 where the digits are classified by two independent classifiers, f 1 and f 2 . If the distributions of the first and the second image are concentrated on 1 and 7 respectively, but f 1 ( x 1 ) = 7 and f 2 ( x 2 ) = 1 , then these errors cannot be detected. Therefore, we assume there is a constant R < 1 , such that R 01 ( f i ) ≤ R holds for any i ∈ [ n ] and any f ∈ F i . We refer to such f i 's as zero-one risk R -bounded . This assumption is mild since it only requires the classifiers to be slightly better than being totally wrong. Also, when a small directly labeled dataset is available, one can bound the classification risks away from 1 with high probability in a uniform manner using standard learning theory. See Appendix C for more details. Now, we are ready to state the ERM-learnability result assuming ∏ n i =1 F i is realizable under ℓ 01 P .

Theorem 3 (ERM learnability under multi-unambiguity) . Assume that there is a constant R < 1 , such that for each i ∈ [ n ] , each f ∈ F i is zero-one risk R -bounded. Assume also that there exist positive integers M ∗ and c ∗ , such that M i ≤ M ∗ and c i ≤ c 0 hold for any i ∈ [ n ] . Then, if σ is multi-unambiguous, we have:

Furthermore, for any ϵ, δ ∈ (0 , 1) , there is a universal constant C 3 , such that with probability at least 1 -δ , the empirical partial risk minimizer with ̂ R 01 P ( f ; σ ) = 0 has a classification risk R 01 ( f ) < ϵ if

Remark. Theorem 3 suggests that the rate of convergence is controlled by M ∗ . A smaller M ∗ leads to a more strict unambiguity condition. For example, the case M ∗ = 1 requires σ to be unstable to any 1-position perturbations, while the case M ∗ = M reduces to M -unambiguity. This observation confirms the intuition that the supervision power of partial labels depends on the 'instability" of σ .

The topk partial loss ℓ k σ ( f ( x ) , s ) for multiple classifiers subject to f ∈ F n , transition σ and s ∈ S straightforwardly extends the topk partial loss for the single classifier case, see Definition 3. Similarly, the (empirical) topk partial classification risk of f subject to σ straightforwardly extends the one from Section 3.2. Below, we provide a Rademacher-style error bound with the topk loss.

Theorem 4 (Error bound under multi-unambiguity with multiple classifiers) . Suppose σ is multiunambiguous and each f i is zero-one risk R -bounded, for R ∈ (0 , 1) . Then, for any integer k ≥ 1 and any δ ∈ (0 , 1) , with probability at least 1 -δ , we have:

̸

5 Learning Under an Unknown Transition

We now explore another direction by dropping the assumption that σ is known. Instead, we assume the learner has access to a transition space G that contains mappings of the form Y M →S including the 'true' transition σ . Notice that learning now becomes particularly challenging, since G can be expressive enough to lead to correct predictions for s from many wrong classifications.

Problem setting. To illustrate the core idea, we consider a simple setting where we only learn a single scoring function f ∈ F as in Section 3. However, σ is an unknown mapping in G . Given partially labeled samples, the learner aims to minimize the zero-one partial risk of f subject to G R 01 P ( f ; G ) that is defined as the minimal partial risk to predict s with a candidate transition in G , i.e., R 01 P ( f ; G ) · · = inf σ ∗ ∈G R 01 P ( f ; σ ∗ ) . The risk is empirically estimated with a partially labeled dataset T P as inf σ ∗ ∈G ̂ R 01 P ( f ; σ ∗ ; T P ) . We demonstrate this learning setting with the following example:

Example 7. Let G = { ( y 1 , y 2 ) ↦→ αy 1 + βy 2 | ( α, β ) ∈ ❘ 2 -{ (0 , 0) }} , namely all the weighted sums with at least one non-zero weight. We aim to learn an MNIST classifier f , given training samples of the form ( x 1 , x 2 , s ) , where the x i 's are MNIST digits and s is the result of applying some σ from G on ( y 1 , y 2 ) , where y i is the prediction of f on x i . The gold σ , i.e., the exact α and β , is unknown.

5.1 ERMLearnability

Similarly to the known transition case, learnability requires us to learn under any data distribution D . Slightly differently from Sections 3 and 4, we start with the adversarial distribution where the mass in D is concentrated to a single label, i.e., all instances x i have the same gold label l i , and hence, the training samples are all associated with the gold label vector y = ( l i , . . . , l i ) . Suppose there is a certain probability that f misclassifies l i as l j . Then, the predicted label vectors will be in { l i , l j } M . In this case, it will be impossible to detect the classification errors if there is a candidate transition σ ′ ∈ G which maps all the vectors in { l i , l j } M to the gold observation of s . Since the actual σ is unknown, we require this does not happen for any pair of σ, σ ′ ∈ G to ensure learnability. Below, we formalize our learnability condition:

̸

Definition 5 (Unambiguous transition space) . Transition space G is unambigous , if for each σ ′ ∈ G , each diagonal label vector y = ( l i , . . . , l i ) , where l i ∈ Y , and each l j = l i , where l j ∈ Y , there exists a vector y ′ ∈ { l i , l j } M , such that σ ′ ( y ′ ) = σ ( y ) .

̸

̸

In pratice, one can examine whether a transition space is unambiguous by checking if for each σ ′ ∈ G and any two different labels l i = l j , set { σ ′ ( y ) | y ∈ { l i , l j } M } is not a singleton. The above ensures that when given a fixed diagonal label vector, and when the classifier makes mistakes, the predicted partial labels are not unique, and hence cannot all agree with the ground truth label.

̸

Class G in Example 7 is unambiguous: for each transition ( y 1 , y 2 ) ↦→ α ′ y 1 + β ′ y 2 and each two labels l ′ = l , the set { α ′ y 1 + β ′ y 2 | ( y 1 , y 2 ) ∈ { l, l ′ } 2 } is not a singleton, so there must exist a label vector with a different weighted sum that is different than the true sum. A counterexample is below:

̸

Example 8. Let G = { ( y 1 , y 2 ) ↦→ w 1 y 1 + w 2 y 2 1 + w 3 y 2 2 + w 4 y 2 | w i = 0 ∀ i ∈ [4] } . Consider the true transition σ : ( y 1 , y 2 ) ↦→ y 1 -y 2 1 + y 2 -y 2 2 from G and a candidate transition σ ′ : ( y 1 , y 2 ) ↦→ y 1 -y 2 1 -y 2 + y 2 2 . Then, G is not unambiguous, since σ ′ ( y ) = σ ( y ) = 0 for any y ∈ { 0 , 1 } 2 . Namely, the partially labeled data are not informative enough to distinguish the label 0 from 1 .

Bounded risk assumption. Similarly to Section 4, this unambiguity condition is necessary but not sufficient to show learnability. To show learnability, we additionally assume there is a constant r > 0 such that /a80 ([ f ]( X ) = y | Y = y ) ≥ r for each y ∈ Y and f ∈ F . We refer to such an f as r -bounded . This assumption is slightly stronger than that of Section 4 as it additionally requires the classifiers to be not fully wrong for every possible label. The main learnability result of this section is below:

Theorem 5. If G is unambigous and any f ∈ F is r -bounded, then we have:

Furthermore, suppose [ F ] has a finite Natarajan dimension d [ F ] and the function class { ( y , s ) ↦→ ✶ { σ ′ ( y ) = s }| σ ′ ∈ G} has a finite VC-dimension d G . Then, for any ϵ, δ ∈ (0 , 1) , there is a universal constant C 4 such that with probability at least 1 -δ , the empirical partial risk minimizer with

̸

̂ R 01 P ( f ; σ ) = 0 has a classification risk R 01 ( f ) < ϵ , if

6 Experiments

We further explore the learning scenario in Section 5 with empirical evaluations. We aim to learn an MNIST classifier using the weighted sum of 2, 3 and 4 MNIST digits and assuming that the weights are unknown, as in Example 7. The weighted sum function is very expressive as it can model Boolean formulas including conjunction and disjunction [36]. Notice that we do not aim to exhaustively assess the weak supervision literature ([24, 43, 59, 13]), but to validate the results of our theoretical analysis.

Baselines. We considered state-of-the-art neuro-symbolic frameworks, namely DeepProbLog (DLog) [30], DeepProbLog with approximations (DLog-A) [31], NeuroLog (NLog) [43], NeurASP (NASP) [59], ABL [13], ENT [27] and Scallop [24]. Only the target sum is used during training under those frameworks. DLog and NLog employ the semantic loss without any approximations (see Section 2). NLog( k ) denotes an NLog variant where only the topk predictions are considered during training, while Scallop( k ) denotes Scallop using the topk semantic loss, see Section 3.2. The approximations in DLog-A are different from the ones in NLog and Scallop; however, both DLog-A and NLog employ the semantic loss on the chosen subset of proofs. As the above frameworks learn classifiers under fixed theories only, we considered weights in { 1 , 2 , 3 , 4 , 5 } , and used an additional neural classifier to learn the unknown weights. Finally, we considered standard supervised learning (SL). To contrast learning under unknown to learning under known transitions, we also ran experiments assuming that the weights are known. Our results show that the classification accuracy exceeds the 98% even for M = 10 . More details on this experiment are in the arXiv version of this paper [51].

Results. Tables 1 and 2 report the accuracy of the learned MNIST classifiers over ten runs. #0, #8 and #16 denote the number of directly labeled samples used for pretraining the MNIST classifiers: #0 means no pretraining; for #8 and #16, the accuracy of the pre-trained classifiers is 37% and 58%. Table 1 presents results for M = 2 , while Table 2 presents results for M ∈ { 2 , 3 , 4 } for Scallop and the two best performing frameworks in Table 1, DLog and NLog- NASP could not scale for M ≥ 3 . We used DLog-A for M ∈ { 3 , 4 } , due to scalability reasons. Similarly, NLog without approximations does not scale for M = 4 (N/A indication). In Table 1, the number of weak samples is in parentheses, e.g., DLog(#4K). For ABL and ENT, we used 10K samples, as they rely on sampling, and used the hyperparameters suggested by the authors.

Discussion. The results confirm our theory: (i) we can learn classifiers under unknown transitions as per Theorem 5 (see the accuracy for DLog and NLog in Table 1); and (ii) learning gets harder when M increases as per Eq. (5) (see the accuracy for different M 's in Table 2). For M = 2 , weak supervision leads to better accuracy than SL even without pre-training and with fewer training samples: DLog(#4K) and NLog(#4K) lead to mean accuracy 96% and 97%, while SL(#4K) leads to mean accuracy 93%, see Table 1. This result suggests that the r -bounded assumption in Theorem 5 could be potentially relaxed. Another observation is that the approximations in DLog-A are less effective than the topk one of NLog. We leave the theoretical analysis of the latter approximation as part of future research. Table 2 also suggests that the topk semantic loss constitutes the most scalable and effective approach to training: in contrast to DLog-A and NLog, the accuracy of the digit classifier reaches 78% for M = 4 , when k = 4 for Scallop.

Our empirical analysis shows that a key issue is scalability . This is a known problem in neurosymbolic learning and it is partially because the relevant problems in logic are intractable. For instance, semantic loss requires computing the models of Boolean formulas, which is #P-complete. Reasoning over logical theories can be also intractable [2], if not undecidable. Scalability straightforwardly affects accuracy, e.g., NLog obtains better accuracy over DLog-A for M = 3 (see Table 2) due to its ability to explore the whole search space. It is also worth stressing that Scallop could not scale beyond M = 4 for k > 1 . The above challenges bring up new research directions [44, 41]. Closing this discussion, it is worth exploring why ABL and ENT led to low accuracy 4 .

7 Related Work

We provide an overview of the literature that closely relates to our work. A more comprehensive review of PLL, latent structural learning, and neuro-symbolic learning, is presented in Appendix F.

Partial label learning. Learnability of PLL is typically shown under the small ambiguity assumption [11, 28, 5]. While it is technically possible to cast our problem to standard PLL by viewing a target s as a partial label for the hidden labels y , the small ambiguity condition is violated in our case since a distracting label can occur because σ is deterministic. Therefore, our proposed conditions relax and generalize the small ambiguity by allowing deterministic transitions, see Appendix E.3. Another line of work in weak supervision exploits the invertibility of transition matrices [46] to compute posterior class probabilities [8, 9, 61]. Our learnability condition, M -unambiguity, and the invertibility condition do not imply each other, see Appendix E.1 and E.2. There, we also show that small ambiguity and the invertibility condition do not imply each other, which might be of independent interest. Another related topic is learning with label constraints [21, 6, 49], where the label of each instance x (possibly structured) in the dataset is constrained in a subset C ⊆ Y . The difference is that the constraint mapping itself is known to the learner and hence can be encoded in the inference algorithm directly, for example, via the CCM [6].

Neuro-symbolic and structured learning. We were motivated by frameworks that employ logic for training neural models [30, 53, 13, 59, 43, 31, 24, 27, 23]. We prove learnability under (unknown) transitions that capture different languages and error bounds under the SL [56] and topk approximations. Notice that [58] also proves the consistency of a topk loss. However, they use a zero-one-style loss for standard supervised learning. Our work is closely related to [62], which studies a similar problem of weakly supervised learning with multiple instances. Our results are stronger in the sense that we propose sufficient and necessary conditions to recover the hidden labels, while [62] concerns the likelihood of the observed labels rather than the hidden ones. Our work is also relevant to latent structural prediction [42, 35, 34, 32]. To our knowledge, no formal learning guarantees have been given for that problem. Complementary to ours is the work in [19, 47, 33] that integrates combinatorial solvers into deep models.

8 Conclusions

We formulated multi-instance PLL and showed its connections with latent structural and neurosymbolic learning. Our work exhibits a greater level of flexibility compared to (single-instance) PLL, as it allows for multiple instances and deterministic transitions. We introduced minimal assumptions that enabled us to establish ERM-learnability and derive error bounds with the topk semantics loss. Our findings suggest that the interaction of multiple instances in forming the observed labels can help relax the learnability assumptions in weakly supervised learning. For future work, we will further relax the learning conditions by assuming a certain degree of smoothness for the data distributions. Another direction is to explore the setting where the hidden labels are non-i.i.d. or structured.

References

S. Abiteboul, R. Hull, and V. Vianu. Foundations of Databases . Addison Wesley, 1995.

Jean-François Baget, Marie-Laure Mugnier, Sebastian Rudolph, and Michaël Thomazo. Walking the complexity lines for generalized guarded existential rules. page 712-717, 2011.

Othman El Balghiti, Adam N. Elmachtoub, Paul Grigas, and Ambuj Tewari. Generalization bounds in the predict-then-optimize framework. In NeurIPS , pages 14389-14398, 2019.

Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. The Journal of Machine Learning Research , 3:463-482, 2003.

Vivien Cabannes, Alessandro Rudi, and Francis Bach. Structured prediction with partial labelling through the infimum loss. In ICML , page 1230-1239, 2020.

Ming-Wei Chang, Lev Ratinov, and Dan Roth. Structured learning with constrained conditional models. Machine Learning , 88(3):399-431, 2012.

Mark Chavira and Adnan Darwiche. On probabilistic inference by weighted model counting. Artificial Intelligence , 172(6):772 - 799, 2008.

Jesús Cid-Sueiro. Proper losses for learning from partial labels. In NeurIPS , pages 1574-1582, 2012.

Jesús Cid-Sueiro, Darío García-García, and Raúl Santos-Rodríguez. Consistency of losses for learning from weak labels. In ECML PKDD , pages 197-210, 2014.

Corinna Cortes, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. Structured Prediction Theory Based on Factor Graph Complexity. In NeurIPS , 2016.

Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research , 12:1501-1536, 2011.

Wang-Zhou Dai and Stephen Muggleton. Abductive knowledge induction from raw data. In Zhi-Hua Zhou, editor, IJCAI , pages 1845-1851. AAAI, 2021.

Wang-Zhou Dai, Qiuling Xu, Yang Yu, and Zhi-Hua Zhou. Bridging Machine Learning and Logical Reasoning by Abductive Learning. In NeurIPS , pages 2815-2826, 2019.

Artur S. d'Avila Garcez, Krysia Broda, and Dov M. Gabbay. Neural-symbolic learning systems: foundations and applications . Perspectives in neural computing. Springer, 2002.

Adam N. Elmachtoub and Paul Grigas. Smart "predict, then optimize". CoRR , abs/1710.08005, 2017.

Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research , 61(1):1-64, 2018.

Vitaly Feldman, Venkatesan Guruswami, Prasad Raghavendra, and Yi Wu. Agnostic learning of monomials by halfspaces is hard. SIAM Journal on Computing , 41(6):1558-1590, 2012.

Jonathan Feldstein, Dominic Phillips, and Efthymia Tsamoura. Principled and efficient motif finding for structure learning of lifted graphical models. CoRR , abs/2302.04599, 2023.

Aaron M. Ferber, Bryan Wilder, Bistra Dilkina, and Milind Tambe. Mipaal: Mixed integer program as a layer. CoRR , abs/1907.05912, 2019.

Robert Fink, Jiewen Huang, and Dan Olteanu. Anytime approximation in probabilistic databases. VLDB Journal , 22(6):823-848, 2013.

Kuzman Ganchev, João Graça, Jennifer Gillenwater, and Ben Taskar. Posterior regularization for structured latent variable models. Journal of Machine Learning Research , 11:2001-2049, 2010.

Nitish Gupta, Sameer Singh, Matt Gardner, and Dan Roth. Paired examples as indirect supervision in latent decision models. In EMNLP , pages 5774-5785, 2021.

Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. In ACL , pages 2410-2420, 2016.

Jiani Huang, Ziyang Li, Binghong Chen, Karan Samel, Mayur Naik, Le Song, and Xujie Si. Scallop: From probabilistic deductive databases to scalable differentiable reasoning. In NeurIPS , pages 25134-25145, 2021.

Rong Jin and Zoubin Ghahramani. Learning with multiple labels. In NeurIPS , pages 897-904, 2002.

Antonis C. Kakas. Abduction. In Claude Sammut and Geoffrey I. Webb, editors, Encyclopedia of Machine Learning and Data Mining , pages 1-8. Springer US, Boston, MA, 2017.

Zenan Li, Yuan Yao, Taolue Chen, Jingwei Xu, Chun Cao, Xiaoxing Ma, and Jian Lu. Softened symbol grounding for neurosymbolic systems. In ICLR , 2023.

Li-Ping Liu and Thomas G. Dietterich. Learnability of the superset label learning problem. In ICML , pages 1629--1637, 2014.

Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In ICML , page 6500-6510, 2020.

Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. In NeurIPS , pages 37493759, 2018.

Robin Manhaeve, Giuseppe Marra, and Luc De Raedt. Approximate Inference for Neural Probabilistic Logic Programming. In KR , pages 475-486, 2021.

Tsvetomila Mihaylova, Vlad Niculae, and André F. T. Martins. Understanding the mechanics of SPIGOT: Surrogate gradients for latent structure learning. In EMNLP , pages 2186-2202, 2020.

Anselm Paulus, Michal Rolínek, Vít Musil, Brandon Amos, and Georg Martius. Comboptnet: Fit the right np-hard problem by learning integer programming constraints, 2021.

Hao Peng, Sam Thomson, and Noah A. Smith. Backpropagating through structured argmax using a SPIGOT. In ACL , pages 1863-1873, 2018.

Aditi Raghunathan, Roy Frostig, John Duchi, and Percy Liang. Estimation from indirect supervision with linear moments. In ICML , volume 48, pages 2568-2577, 2016.

Dan Roth and Wen-tau Yih. Global Inference for Entity and Relation Identification via a Linear Programming Formulation . MIT Press, Introduction to Statistical Relational Learning edition, 2007.

Rajhans Samdani, Ming-Wei Chang, and Dan Roth. Unified expectation maximization. In ACL , pages 688-698, 2012.

Prithviraj Sen, Breno W. S. R. de Carvalho, Ryan Riegel, and Alexander Gray. Neuro-symbolic inductive logic programming with logical neural networks. AAAI , 36(8):8212-8219, 2022.

Junghoon Seo and Joon Suk Huh. On the power of deep but naive partial label learning. In ICASSP , pages 3820-3824, 2021.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, USA, 2014.

Mate Soos, Divesh Aggarwal, Sourav Chakraborty, Kuldeep S. Meel, and Maciej Obremski. Engineering an efficient approximate dnf-counter. In IJCAI , pages 2031-2038, 2023.

Jacob Steinhardt and Percy S Liang. Learning with relaxed supervision. In NeurIPS , volume 28, 2015.

Efthymia Tsamoura, Timothy Hospedales, and Loizos Michael. Neural-symbolic integration: A compositional perspective. In AAAI , pages 5051-5060, 2021.

Efthymia Tsamoura, Jaehun Lee, and Jacopo Urbani. Probabilistic reasoning at scale: Trigger graphs to the rescue. Proceedings of the ACM on Management of Data , 1(1), 2023.

Shyam Upadhyay, Ming-Wei Chang, Kai-Wei Chang, and Wen-tau Yih. Learning from explicit and implicit supervision jointly for algebra word problems. In EMNLP , pages 297-306, 2016.

Brendan van Rooyen and Robert C. Williamson. A theory of learning with corrupted labels. Journal of Machine Learning Research , 18(228):1-50, 2018.

Marin Vlastelica, Anselm Paulus, Vít Musil, Georg Martius, and Michal Rolínek. Differentiation of blackbox combinatorial solvers. CoRR , abs/1912.02175, 2019.

Haoyu Peter Wang, Nan Wu, Hang Yang, Cong Hao, and Pan Li. Unsupervised learning for combinatorial optimization with principled objective relaxation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, NeurIPS , volume 35, pages 31444-31458, 2022.

Kaifu Wang, Hangfeng He, Tin D. Nguyen, Piyush Kumar, and Dan Roth. On regularization and inference with label constraints. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 35740-35762. PMLR, 23-29 Jul 2023.

Kaifu Wang, Qiang Ning, and Dan Roth. Learnability with indirect supervision signals. In NeurIPS , volume 33, pages 9145-9155, 2020.

Kaifu Wang, Efi Tsamoura, and Dan Roth. On learning latent models with multi-instance weak supervision, 2023.

Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. Template-based math word problem solvers with recursive neural networks. In AAAI , pages 7144-7151, 2019.

Po-Wei Wang, Priya L. Donti, Bryan Wilder, and J. Zico Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In ICML , 2019.

Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. Leveraged weighted loss for partial label learning. CoRR , abs/2106.05731, 2021.

Zhaofeng Wu. Learning with latent structures in natural language processing: A survey. arXiv preprint arXiv:2201.00490 , 2022.

Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In ICML , pages 5502-5511, 2018.

Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. In NeurIPS , volume 34, pages 27119-27130, 2021.

Forest Yang and Sanmi Koyejo. On the consistency of top-k surrogate losses. In ICML , volume 119, pages 10727-10735, 2020.

Zhun Yang, Adam Ishay, and Joohyung Lee. NeurASP: Embracing neural networks into answer set programming. In IJCAI , pages 1755-1762, 2020.

Peilin Yu, Tiffany Ding, and Stephen H. Bach. Learning from multiple noisy partial labelers. In PMLR , volume 151, pages 11072-11095, 2022.

Mingyuan Zhang, Jane Lee, and Shivani Agarwal. Learning from noisy labels with no change to the training process. In ICML , volume 139 of Proceedings of Machine Learning Research , pages 12468-12478, 2021.

Yivan Zhang, Nontawat Charoenphakdee, Zheng Wu, and Masashi Sugiyama. Learning from aggregate observations. ArXiv , abs/2004.06316, 2020.

Appendix

This appendix is organized as follows:

In Appendix A, we provide detailed definitions for the loss functions and the complexity measures we use in our work.

In Appendix B, C and D, we provide the proofs to all formal statements.

In Appendix F, we provide a comprehensive review of the related work in PLL, latent structural learning, contrained learning, and neuro-symbolic learning.

In Appendix E, we compare common distributional assumptions in the standard PLL literature with ours. In particular:

In Appendix E.1, we provide a transition matrix [8, 46] formulation of multi-instance PLL and show that M -unambiguity does not imply transition matrix invertibility for multi-instance PLL and vice versa.

In Appendix E.2, we show a result of independent interest: that transition matrix invertibility does not imply small ambiguity [28] and vice versa.

In Appendix E.3, we show that M -unambiguity, the unambiguity notion proposed in Definition 1, is an extension of the small ambiguity degree [28], by considering a generalized setting where σ is stochastic.

In Appendix G, we provide implementation details of our experiments.

A Preliminaries

In this section, we introduce in further detail some key notions used in the main body of our paper. We start with the definition of the semantic loss [56].

A.1 Loss functions

Semantic Loss (SL) [56] has been adopted to train classifiers subject to logical theories [30, 43, 24]. SL is the cross entropy of the weighted model counting (WMC) [7] of a formula subject to a softmax output. Below, we provide the necessary notions. Let Z denote a set of Boolean variables. An interpretation I of Z is a mapping from each A ∈ Z to true ( ⊤ ) or false ( ⊥ ). Interpretation I is a model of a Boolean formula φ , if φ evaluates to true in I . We use the notation A ∈ φ to denote that Boolean variable occurs in φ . By treating each Boolean variable A occurring in φ as an independent Bernoulli random variable that becomes true with probability ω ( A ) and false with probability 1 -ω ( A ) , the different interpretations of Z induce a probability distribution, where the probability P φ ( I, ω ) of each interpretation I subject to ω and φ is given by:

The WMC WMC( φ, ω ) of φ under ω then denotes the probability of φ being satisfied , i.e., it evaluates to true, under ω 5 , and is defined as:

Assuming that each variable in φ is associated with a class from Y , and treating f ( x ) as a mapping from the variables in φ to the f 's scores on x , the semantic loss of φ under f ( x ) is defined as:

We demonstrate the above concepts via Example 4.

Example 9 (Example 4, cont'd) . Recall that the Boolean formula induced by the partial label s = 2 is φ = ( A 1 , 2 ∧ A 2 , 0 ) ∨ ( A 1 , 0 ∧ A 2 , 2 ) , where A i,j is a Boolean variable that is true iff the i -th input digit has label j . Given a probabilistic prediction f ( x ) , variable A i,y is assigned probability f y ( x i ) , inducing a probability mapping ω · · = ⋃ i ∈{ 1 , 2 } ,y ∈{ 0 ,..., 9 } { A i,y ↦→ f y ( x i ) } . Table 3 shows all interpretations of { A 1 , 2 , A 2 , 0 , A 1 , 0 , A 2 , 2 } and their corresponding probabilities subject to ω and φ . The WMC of φ under ω is the sum of the probabilities of all the 2 4 interpretations.

To establish SL, each ( x i , y )) , where x = ( x 1 , . . . , x M ) , i ∈ [ M ] and y ∈ Y M , is associated with a distinct Boolean variable A i,y .

The nature of multi-instance PLL implies Boolean formulas in disjunctive normal form (DNF), where a formula is in DNF form, if it is a disjunction over conjunctions of Boolean variables or their negations, see Example 4. In fact, each Boolean formula computed out of the labels associated with a partial label is a positive DNF formula, i.e., a DNF formula where no variable occurs negatively. Using the union bound of standard probability theory and the model-based semantics of formula probabilities [20], the following result directly follows:

Lemma 2. Let Φ 1 , . . . , Φ K be K positive conjunctive formulas over the set of Boolean variables Z and ω : Z ↦→ (0 , 1) . Assuming that each variable Z ∈ Z is treated as an independent Bernoulli random variable that becomes true with probability ω ( Z ) and false with probability 1 -ω ( Z ) , the following holds:

where the equality holds when the Φ i 's are logically inconsistent, i.e., there is no interpretation of Z in which two or more conjunctions are simultaneously true.

Let Σ denote exclusiveness constraints among the variables occurring in the Φ i 's. For instance, returning back to Example 9, to align with the semantics of the classifier, only of the Boolean variables A 1 , 2 and A 1 , 0 can be true in each interpretation 6 . Then, WMC( ∨ K i =1 Φ i ∧ Σ , ω ) ≤ WMC( ∨ K i =1 Φ i , ω ) . This is because Σ reduces the number of models of φ .

In addition, when the Φ i 's share no Boolean variables, the following holds [20]:

Due to (15), the topk partial loss becomes

when the y ( i ) 's in Definition 3 share no Boolean variables.

A.2 VC and Natarajan dimensions

We use the following definitions of VC and Natarajan dimensions, which can be found in [40].

Definition 6 (Shattering: multiclass classifiers) . Let H be a space of multiclass classifiers of the form X → Y . A set C ⊂ X is shattered by H , if there exist two functions h 0 , h 1 : C →Y , such that

̸

For each x ∈ C , we have that h 0 ( x ) = h 1 ( x ) .

For each B ⊂ C , there exists a function h ∈ H , such that

Definition 7 (Natarajan Dimension) . The Natarajan dimension of H , denoted by Nat( H ) , is the maximal size of a shattered set C ⊂ X . If H is a space of binary classifiers, then its Natarajan dimension is also called its VC dimension and is denoted by VC( H ) .

A.3 Rademacher complexity

We use the following definition of the (empirical) Rademacher complexity of a multiclass scoring space F , which is adapted from [10].

Definition 8 (Rademacher complexity) . Let T = { x 1 , . . . , x m } be a set of instances that are i.i.d. drawn from D X . The empirical Rademacher complexity of F with respect to T is defined by

where the ϵ i,y 's are i.i.d. Rademacher random variables, each of which is uniformly distributed over {-1 , +1 } , and ϵ is a vector over all the ϵ i,y 's. Furthermore, the Rademacher complexity of scoring class F is the expectation of the empirical version:

B Proofs for Section 3

Before proceeding with the proofs, we will introduce some useful notation.

Definition 9 (Probability of misclassification) . Let f be a scoring function in F and l i , l j be two labels in Y . Then, the probability that f misclassifies label l i as l j subject to samples drawn from D is defined as

For convenience, here we consider the general setting where the transition function σ is unknown to the learner but instead, a transition class G is provided. The results for the special case where σ is known to the learner can be derived by setting G = { σ } . Now, fix a scoring function class F and a transition class G . We define H F , G as the class of binary classifiers that, given samples of the form ( x , s ) , for x ∈ X M and s ∈ S , output zero, if the predictions for x abide by the partial label s and one, otherwise. Formally:

̸

̸

For the case where the transiton is known to the learner, we also denote H F , G = H F for simplicity. From (20), it follows that the zero-one binary classification loss of a classifier f σ ′ ∈ H F , G , equals the zero-one partial loss subject to σ ′ defined as ℓ 01 σ ′ ( f ( x ) , s ) · · = ✶ { σ ′ ( f ( x )) = s } . We now present the following lemma that bounds the VC dimension of H F , G , which is reminiscent of the arguments of [28, 50].

̸

Lemma 3. Fix a scoring function class F and a transition class G . Let Nat([ F ]) = d [ F ] < ∞ be the Natarajan dimension of [ F ] and d G be the VC dimension of the class of classifiers { ( y , s ) ↦→ ✶ { σ ′ ( y ) = s }| σ ′ ∈ G} , where y ∈ Y M and s ∈ S . The VC dimension of class H F , G is bounded as:

Proof. Suppose that the VC dimension of H F , G is d . Let N be the maximum number of distinct ways to assign labels in Y M to d points in X M ×S using [ F ] . Each possible label assignment leads to a set of d elements in X M ×Y M ×S , and for any d elements, by Sauer-Shelah lemma (see, for example, [40]'s Lemma 6.10), there are at most

ways for G to decide if ✶ { σ ′ ( y ) = s } , where e is the base of the natural logarithm. Based on the above, we have:

̸

Therefore, N ≥ 2 d ( d G / e Md ) d G .

By Natarajan's lemma of multiclass classification, we have:

Combining (22) with the above equations, it follows that

Taking the logarithm on both sides, we have:

Rearranging the inequality yields:

where the second step follows from the first-order Taylor series expansion of the logarithm function at the point 6( d [ F ] + d G ) . Therefore,

where the last step follows from the fact that log 2 -1 / 6 > 1 / 2 . The above concludes the proof of Lemma 3.

Notice that if σ is known to the learner (i.e., G is a singleton) and hence, d G = 0 7 , then (21) becomes

B.1 Proofs for Section 3.1

Lemma 1. If σ is M -unambigous, then we have:

Moreover, if σ is not M -unambiguous, then learning from partial labels is arbitrarily difficult, in the sense that a classifier f with partial risk R 01 P ( f ; σ ) = 0 can have a risk of R 01 ( f ) = 1 .

Proof. Given (19), the zero-one risk of f can be rewritten as

̸

Since σ is M -unambigous, we know that if all the M input instances have the same label and are wrongly classified as another label, then the predicted partial label will be wrong. Therefore, the zero-one partial risk is lower bounded by the probability of such events, namely the sum of the same type of classification mistake being repeated by M times. We have that:

̸

̸

where |Y| = c . Rearranging the inequality yields:

The above concludes the first part of the proof of Lemma 1.

̸

We now focus on the second part of the proof. If σ is not M -unambiguous, then from Definition 1, we know that there exists a pair of labels y, y ′ ∈ Y with y ′ = y , such that σ ( y, . . . , y ) = σ ( y ′ , . . . , y ′ ) holds. Consider now a sample ( x 0 , y ) drawn from D . If D concentrates all its mass on x 0 ∈ X , i.e., /a80 D ( x 0 ) = 1 , and classifier f misclassifies x 0 as y ′ = y , then R 01 P ( f ; σ ) = 0 but R 01 ( f ) = 1 . The above concludes the proof of Lemma 1.

̸

Theorem 6 (ERM learnability under M -unambiguity) . Suppose F is realizable under ℓ 01 P and [ F ] has a finite Natarajan dimension d [ F ] . Then for any ϵ, δ ∈ (0 , 1) , there exists a universal 8 constant C 0 > 0 , such that with probability at least 1 -δ , the empirical partial risk minimizer with ̂ R 01 P ( f ; σ ; T P ) = 0 has a classification risk R 01 ( f ) < ϵ , if

Proof. Let f be the empirical partial risk minimizer, such that ̂ R 01 P ( f ; σ ; T P ) = 0 holds. For any ϵ ∈ (0 , 1) , we know by the proof of Lemma 1 that R 01 ( f ) ≤ ( c 2 M -2 R 01 P ( f ; σ )) 1 /M holds. Hence, by bounding R 01 P ( f ; σ ) ≤ ϵ M /c 2 M -2 , the following would follow: R 01 ( f ) ≤ ϵ . By the standard bound for sample complexity based on VC-dimension (see, for example, Theorem 6.7 in [40]), we know that for any confidence parameter δ ∈ (0 , 1) , R 01 ( f ) ≤ ( c 2 M -2 R 01 P ( f ; σ )) 1 /M holds with probability at least 1 -δ , if

where where C is a universal constant. Combining (23) with (28) and (29) yields the desired result, concluding the proof of Theorem 6.

Before proving Proposition 1, we will introduce a more general definition of unambiguity:

̸

Definition 10 ( I -unambiguity) . Mapping σ is I -unambiguous if there exists a set of distinct I indices I = { i 1 , . . . , i I } ⊆ [ M ] , such that for any y ∈ Y M having the same label l in all positions in I , the vector y ′ that results after flipping l to l ′ = l in those positions satisfies σ ( y ′ ) = σ ( y ) .

̸

̸

We now prove the following lemma based on I -unambiguity:

Lemma 4. If σ is both I - and M -unambigous, then we have:

̸

Proof. Let I be the disambiguation set of σ . Then, for each pair of labels l i , l j ∈ Y with l i = l j , the following holds with probability E l i ,l j ( f ) I : the gold labels in I are all l i , but they are all misclassified by f as j . Since σ is I -unambiguous, this type of error implies a partial label error. Then, we have the following:

̸

̸

Since σ is also M -unambigous by assumption, we have from (26) that R 01 ( f ) ≤ c 2 ( R 01 P ( f ; σ )) 1 /M . Hence, 1 -R 01 ( f ) ≥ 1 -c 2 ( R 01 P ( f ; σ )) 1 /M . Combining the above with (31), yields:

Therefore,

Based on the above, we conclude that

where as R P ( f ; ℓ 01 P ) → 0 . We can see that Φ I ( t ) = O ( t 1 /I ) as t → 0 . The above concludes the proof of Lemma 4.

We are now ready to prove Proposition 1.

Proposition 1 (ERM learnability under 1- and M -unambiguity) . If σ is both 1- and M -unambigous, then we have:

Furthermore, if F is realizable under ℓ 01 P and [ F ] has a finite Natarajan dimension d [ F ] , then for any δ ∈ (0 , 1) and ϵ ∈ (0 , 1) that is sufficiently close to 0, there exists a universal constant C 1 , such that with probability at least 1 -δ , the empirical partial risk minimizer with ̂ R 01 P ( f ; σ ) = 0 has a classification risk R 01 ( f ) < ϵ , if

Proof. The first claim can be derived from Lemma 4, by setting I = 1 . For the second claim, using inequality (33) and letting I = 1 , we know that

Suppose ϵ is small enough so that

Now, if we could bound R 01 P ( f ; σ ) ≤ ϵ/ 2 , we will have that

Now, our goal is to bound the sample complexity so that R 01 P ( f ; σ ) ≤ ϵ/ 2 . Similarly to the proof of Theorem 6, by the standard bound for sample complexity based on VC-dimension, for any δ ∈ (0 , 1) , R 01 P ( f ; σ ) ≤ ϵ/ 2 holds with probability at least 1 -δ , if

where where C is a universal constant. Combining (23) with (35) and (36) yields the desired result. The above concludes the proof of Proposition 1.

B.2 Proofs for Section 3.2

B.2.1 Relating the topk semantic loss to the infimum/minimum loss

In this section, we show that the minimal loss can be viewed as a special case of the topk partial loss from Definition 3.

We start by recapitulating the definition of the minimal loss [29]. Given a scoring function f : X → R M , a loss function ℓ and a partial labelset s ∈ 2 Y , the minimal loss [29] for standard PLL [28] is defined as:

Below, we show that the minimal loss coincides with the top1 partial loss.

For multi-instance PLL, we can extend (37) in a straightforward fashion as follows:

If ℓ is the cross-entropy loss, then the minimal loss finds the most likely label vector in the preimage of s . Now, let us focus on Definition 3. If we set k = 1 in ℓ k σ , then the only model for the formula φ = y (1) · · = A 1 ,y (1) 1 ∧ · · · ∧ A M,y (1) M assigns to each Boolean variable occurring in y (1) value ⊤ . The WMC of φ is given by

which is also the maximum likelihood that can be obtained in the preimage of s . This implies that the top1 semantic loss finds the minimum negative log-likelihood. Therefore, the topk partial loss equals to the minimal loss for k = 1 .

Furthermore, for a partial label s ∈ 2 Y and a label prediction z , the infimum loss [5] is defined as

Treating the probabilistic prediction f ( x ) as a generalized label, the above definition becomes equivalent to that of the minimal loss (notice that Y is finite, so infimum = minimum).

The above discussion shows that the minimal loss is a special case of the topk partial loss.

B.2.2 Proofs

To prove the main result of this section, we will need the following definitions and lemmas.

Definition 11 (Topk partial ℓ 1 loss) . For an integer k ≥ 1 , the topk partial ℓ 1 loss under scoring function f , input x ∈ X M and partial label s ∈ S is given by

where y (1) , . . . , y ( k ) are as in Definition 3.

We now prove the following lemma:

Lemma 5. For any classifier f ∈ F , x ∈ X M , s ∈ S and integer k ≥ 1 , we have:

̸

Proof. Let us denote by y f = ([ f ]( x 1 ) , . . . , [ f ]( x M )) the most likely label assignment to x by f . If ℓ 01 P ( y f , s ) = 1 holds, then σ ( y f ) = s . Therefore y f is different from the topk y ( i ) vectors. Hence, the probabilities of the y ( i ) 's, for i ∈ [ k ] , and y f sum to at most 1, i.e.,

The second inequality holds because the assignment y f has a larger score than any other label assignments in the preimage of s . Furthermore, by Lemma 2, we know that

Since the zero-one loss ℓ 01 σ ( y f , s ) is by definition either 0 or 1, we have:

Finally, from the inequality -log(1 -t ) ≥ t ∀ t > 0 , we know that ˜ ℓ k P ( f ( x ) , s ) lower bounds the cross-entropy topk loss:

The above concludes the proof of Lemma 5.

The following two lemmas concern the Lipschitzness of the topk loss.

Lemma 6 (Contraction lemma (Lemma 5 from [10])) . Let N and m be two positive integers. Let also H be a set of functions that map X to ❘ N . Suppose that for each i ∈ [ m ] , function Ψ i : ❘ N → ❘ is µ i -Lipschitz with the 2-norm, i.e.,

Then, for any set of m points x 1 , . . . , x m ∈ X , the following holds:

where the σ i 's and the ϵ ij 's are independent Rademacher variables uniformly distributed over {-1 , +1 } .

Lemma 7 (Lipschitzness) . For a given scoring function f : X × Y → ❘ and a vector of instances x ∈ X M , let f ( x ) = [ f y ( x i )] i ∈ [ M ] ,y ∈Y ∈ ❘ M ×|Y| denote the vector of scores for each label. Then, for any two scoring functions f, f ′ ∈ F and any sample ( x , s ) ∈ D P , we have:

Proof. Given ( x , s ) , the derivative of the WMC with respect to the score f y ( x i ) for a label y ∈ Y is 0 if A i,y does not appear in formula φ . Otherwise,

where where

and φ i,y is the logic formula that deletes all occurrences of A i,y in φ . Therefore, the 2-norm of the gradient of the mapping f ( x ) ↦→ ˜ ℓ k P ( f ( x ) , s ) is upper bounded by

The boundedness of the gradient implies that the function f ( x ) ↦→ WMC( ∨ k i =1 y ( i ) , f ( x )) is Lipschitz with a Lipschitz constant √ Mk , concluding the proof of Lemma 7.

We are now ready to present the proof of the main theorem of this section.

Theorem 2 (Error bound under unambiguity) . Let an integer k ≥ 1 and δ ∈ (0 , 1) . If σ is both 1and M -unambiguous, then with probability at least 1 -δ , we have:

where ̂ R k P ( f ; σ ; T P ) = ∑ ( x ,s ) ∈T P ℓ k σ ( f ( x ) , s ) /m P is the empirical counterpart of (6) and Φ is an increasing function that satisfies lim t → 0 Φ( t ) /t = 1 .

Proof. Bound with the partial risk. From the proof of Lemma 4 and I = 1 , we know that

We can see that Φ is monotone increasing and lim t → 0 Φ( t ) /t = 1 .

Bound the zero-one partial risk with the topk loss. From Lemma 5, we know that ℓ 01 P ( f ( x ) , s ) ≤ ( k +1) ˜ ℓ k P ( f ( x ) , s ) . Taking expectation yields:

Bound with the empirical risk. Given a partially labelled dataset T P = { ( x i , s i ) } m P i =1 , by the standard Rademacher complexity bounds, we know that for each δ ∈ (0 , 1) and each f ∈ F , the following inequality holds with probability at least 1 -δ :

where A F ,k is a class of functions that take as inputs elements of the form ( x , s ) with x ∈ X M and s ∈ S , and is defined as follows:

By Lemma 6 with Ψ i : f ( x i ) → ˜ ℓ k P ( f ( x i ) , s i ) , and Lipschitzness (Lemma 7) we have:

Bound with cross-entropy. Finally, from the inequality -log(1 -t ) ≥ t ∀ t > 0 , we know that

Putting the above inequalities together yields the proof of Theorem 2.

C Proofs for Section 4

C.1 On the bounded risk assumption

In Section 4, we introduced the bounded risk assumption. Recall that the assumption requires a constant R < 1 , such that for each i ∈ [ n ] and each f ∈ F i , R 01 ( f ) ≤ R holds. Below, we discuss how this assumption is achieved with a small amount of directly labeled data T L = { ( x i , y i ) } m L i =1 i.i.d. drawn from D .

Given T L and a trade-off parameter λ > 1 , consider the following combined learning objective:

Suppose that there is a classifier f 0 ∈ F that can achieve a small empirical classification risk r 0 given by 1 m L ∑ ( x,y ) ∈T L ℓ 01 ([ f 0 ]( x ) , y ) 9 . Then, any classifier f 1 with an empirical classification risk greater than r 0 +1 /λ will be rejected by this learning objective. This is because

Therefore, all the possibly learned classifiers must have an empirical risk less than r 0 +1 /λ . Furthermore, under standard learning theory assumptions (e.g., finite VC dimension or vanishing Rademacher complexity), for any δ 0 ∈ (0 , 1) we can typically bound the expected classification risk with probability at least 1 -δ 0 by

where B 1 , B 2 is a constant depending on the learning problem. As long as we can have m L that is large enough to make (55) less than 1 (i.e., non-vacuous ), we can guarantee that our assumption holds with probability at least 1 -δ 0 . The above justified the bounded risk assumption.

C.2 Proofs

Below, we introduce the analog of Lemma 3 for the multi-classifier case. Let

Given a training sample ( x , s ) , a binary classifier from H F 1 ,..., F n predicts whether the labels predictions for x are consistent with the partial label s . Suppose ( x , s ) are drawn from D P . Then, 0 is always the gold label of this binary classification problem. The zero-one binary classification error of { ( x , s ) ↦→ ✶ { σ ([ f ]( x )) / ∈ s } equals the zero-one partial loss defined as ℓ 01 σ ( f ( x ) , s ) .

Lemma 8. Let Nat([ F i ]) = d [ F i ] < ∞ be the Natarajan dimension of each [ F i ] . Then, we can bound the VC dimension of the function class H F 1 ,..., F n as

Proof. By Natarajan's lemma of multiclass classification, we have:

where c i = |Y i | . Taking the logarithm on both sides, we have

Rearranging the inequality yields:

where the second step follows from the first-order Taylor series expansion of the logarithm function log( M i t ) at the point t = n e d [ F i ] . Using the fact that (log 2 -1 / e) -1 < 4 , yields (56), concluding the proof of Lemma 8.

Theorem 3 (ERM learnability under multi-unambiguity) . Assume that there is a constant R < 1 , such that for each i ∈ [ n ] , each f ∈ F i is zero-one risk R -bounded. Assume also that there exist positive integers M ∗ and c ∗ , such that M i ≤ M ∗ and c i ≤ c 0 hold for any i ∈ [ n ] . Then, if σ is multi-unambiguous, we have:

Furthermore, for any ϵ, δ ∈ (0 , 1) , there is a universal constant C 3 , such that with probability at least 1 -δ , the empirical partial risk minimizer with ̂ R 01 P ( f ; σ ) = 0 has a classification risk R 01 ( f ) < ϵ if

Proof. Recall that c i = |Y i | . By the multi-unambiguity assumption, we can lower bound the partial label risk as

̸

Therefore, we have:

̸

The above concludes the first part of Theorem 3.

To show the second part, from (59), we know that if the following holds:

then, R 01 ( f ) ≤ ϵ holds. By the standard bound for sample complexity based on VC-dimension, we know that for any confidence parameter δ ∈ (0 , 1) , inequality (60) holds with probability no less than 1 -δ , if where

and C is a universal constant. Combining (56) with (61) and (62), concludes the second part of Theorem 3.

Theorem 4 (Error bound under multi-unambiguity with multiple classifiers) . Suppose σ is multiunambiguous and each f i is zero-one risk R -bounded, for R ∈ (0 , 1) . Then, for any integer k ≥ 1 and any δ ∈ (0 , 1) , with probability at least 1 -δ , we have:

Proof. Let

be the vector that encodes all the probabilities that f assign to a pair ( x, y ) where x is from x and y is a possible label for x . Similarly to the single-variable case, we define

Bound with the partial risk. From the proof of Theorem 3, we know that

Bound with the topk loss. Similarly to the single-classifier case, it can be shown that

Bound with the empirical risk. By standard Rademacher complexity bounds, given a partially labeled dataset T P of size m P , for each δ ∈ (0 , 1) and each f ∈ F , the following holds with probability at least 1 -δ :

where H F ,k is a class of functions that take as input elements of the form ( x , s ) , with x ∈ X M and s ∈ S , and is defined as follows:

Denote F = ∏ i M =1 F i . Using exactly the same argument in Lemma 7, we can show the mapping f ( x ) ↦→ ˜ ℓ k P ( f ( x ) , s ) is Lipschitz with Lipschitz constant √ kM , By the contraction lemma (Lemma 6), we have that

The above implies

Bound with cross-entropy. Finally, from the inequality -log(1 -t ) ≥ t ∀ t > 0 , we know that

Putting the above inequalities together, the proof of Theorem 4 follows.

D Proofs for Section 5

Theorem 5. If G is unambigous and any f ∈ F is r -bounded, then we have:

Furthermore, suppose [ F ] has a finite Natarajan dimension d [ F ] and the function class { ( y , s ) ↦→ ✶ { σ ′ ( y ) = s }| σ ′ ∈ G} has a finite VC-dimension d G . Then, for any ϵ, δ ∈ (0 , 1) , there is a universal constant C 4 such that with probability at least 1 -δ , the empirical partial risk minimizer with ̂ R 01 P ( f ; σ ) = 0 has a classification risk R 01 ( f ) < ϵ , if

̸

̸

Proof. Fix a classifier f ∈ F . Since G is unambiguous, it follows that for each pair of labels ( l i , l j ) such that E l i ,l j ( f ) > 0 and each σ ′ ∈ G , there exists a vector y ∈ { l i , l j } M , such that σ ′ ( y ) = σ ( l i , l i , . . . , l i ) holds. Therefore, we have:

̸

̸

The above implies that R 01 ( f ) ≤ O ( R 01 P ( f ; σ ) 1 /M ) holds.

To prove the second part of Theorem 5, from the above discussion we know that R 01 ( f ) ≤ ( c 2 M -2 R 01 P ( f ; σ ) /r M ) 1 /M . Therefore, if we could control R 01 P ( f ; σ ) ≤ ϵ M r M /c 2 M -2 , we would have that R 01 ( f ) ≤ ϵ . By the standard bound for sample complexity based on VC-dimension, then for any δ ∈ (0 , 1) , R 01 P ( f ; σ ) ≤ ϵ M r M /c 2 M -2 can happen with probability at least 1 -δ if

where and C is a universal constant. Now, recall (21) from Lemma 3. Combining (21) with (70) and (71) yields the proof of the second part of Theorem 5.

E Results for standard PLL

E.1 Counterexamples with transition matrices

Definition 12 (Transition matrix [8, 46]) . A transition matrix T for a learning problem with hidden label Y ∈ Y and observation S ∈ S is a stochastic matrix of dimension |Y| × |S| , where the element in its i th column and j th row is the conditional probability /a80 ( S = j | Y = i ) . Transition matrix T is invertible if it is left invertible, i.e., there is a matrix P of dimension |Y| × |S| such that PT = I |Y| .

As shown in [8, 46], if the transition is invertible, then it is possible to construct an unbiased estimator for the classification loss using the samples of the partial label s . Therefore, the invertibility of the transition is desirable since it implies one can minimize the classification risk with a partially labeled dataset alone.

̸

̸

A transition matrix formulation for multi-instance PLL. Consider the multi-instance PLL with a single classifier and a deterministic transition σ . Consider a naive formulation of the transition matrix where we view Y M as the label space. Then, if there are two different label assignments y = y ′ such that σ ( y ) = σ ( y ′ ) , their corresponding column vectors in T will be the same, resulting in a non-invertible transition. For example, in the MNIST with SUM2 problem, the column vectors corresponding to the label pairs (0 , 1) and (1 , 0) will be:

This fact implies that the transition σ must be injective to ensure invertibility, which is too strong for the partial labels in practice.

̸

To overcome this issue, we consider an alternative formulation, where we view s as a randomized partial label for each of the M instances, where the randomness comes from the distribution of the other instances. Suppose the marginal distribution of Y is known (or can be estimated in some ways). Then, we construct a transition matrix T k for each k ∈ [ M ] by defining its i th column and j th row to be /a80 ( s = j | Y i = i ) = /a80 ( y | σ ( y ) = j, y k = i ) . For example, the transition matrices for the MNIST with 2sum problem are

We say the standard PLL problem is invertible if there exists an k ∈ [ M ] such that T k is left invertible.

̸

M -unambiguity ̸⇒ invertibility for multi-instance PLL. Consider the following example, where M -unambiguity holds but the transition matrix is not invertible for no input position. Consider Y = [3] and M = 3 . Assume σ (1 , 1 , 1) = σ (2 , 2 , 2) = σ (3 , 3 , 3) so that σ is M -unambiguous. Now, suppose

̸

so that S = { σ (1 , 1 , 1) , σ (2 , 2 , 2) , σ (3 , 3 , 3) } . Suppose that /a80 ( Y = 1) = /a80 ( Y = 2) = /a80 ( Y = 3) = 1 / 3 . Then, it can be verified that

This means that all the transition matrices are non-invertible.

Inveritbility ̸⇒ M -unambiguity for multi-instance PLL. Suppose Y = [2] and σ ( y 1 , y 2 ) = ✶ { y 1 = y 2 } . Then M -unambiguity does not hold since σ (1 , 1) = σ (2 , 2) . But if /a80 ( Y = 1) = 0 . 1 = 1 -/a80 ( Y = 2) , then

and and

is an invertible transition.

E.2 Counterexamples for standard PLL

We show that for the standard PLL, the small ambiguity degree condition [28] and the invertibility of the transition matrix, see Definition 12, do not imply each other. Below, we recapitulate the small ambiguity degree condition.

Definition 13 (Ambiguity degree [28]) . The ambiguity degree for standard PLL is defined as

̸

where D ( x, y ) is the density function at ( x, y ) . We say that a PLL instance satisfies the small ambiguity degree condition , if γ < 1 .

Small ambiguity ̸⇒ invertibility. Consider the following transition matrix (where all blank elements are zero) with Y = [5] and S = 2 Y :

It can be verified that the small ambiguity degree is 1 / 2 but this matrix is of rank 4 < 5 and the invertibility condition does not hold.

Invertibility ̸⇒ small ambiguity. Consider the following transition matrix with Y = [2] and S = 2 Y :

We can see that the small ambiguity degree is 1 but the above matrix is of full rank.

E.3 M -ambiguity and small ambiguity

We show that our proposed M -unambiguity, see Definition 1, is an extension of the small ambiguity degree, see Definition 13. To do this, we consider a generalized setting, where s is a randomized function of y . In particular, similarly to standard PLL (aka superset learning problem) [28, 8], we define σ to be a random function Y M → 2 Y M so that s ∈ 2 Y M is a random set of vectors in Y M . Below, let D M be a distribution followed by M i.i.d. random variables ( X i , Y i ) , where ( X i , Y i ) ∼ D , for each i ∈ [ M ] . We use D M ( x , y ) to denote the density at ( x , y ) , for x ∈ X M and y ∈ Y M .

Definition 14 ( M -Ambiguity degree) . The ambiguity degree for standard PLL subject to D M is given by

̸

We say that a PLL instance satisfies the small M -ambiguity degree condition , if γ < 1 .

Firstly, it can be seen that the M -ambiguity degree, see Definition 14, reduces to the ambiguity degree from Definition 13 when M = 1 , since all label vectors are diagonal in that case. Below, we show that the small M -ambiguity degree condition guarantees learnability.

Proposition 2. If the small M -ambiguity degree condition is satisfied, then we have:

Proof. Let E be the event that for an input vector x ∈ X M with gold labels y , there exists a diagonal vector y ′ ∈ Y M with y ′ = y , such that y ′ ∈ s . Then, by definition, we have /a80 ( E ) ≤ γ . Also, conditioned on E , from the proof of Lemma 1, we know that R 01 P ( f ) ≥ R 01 ( f ) M ( c ( c -1)) M -1 . The above implies that

̸

Therefore, we have:

concluding the proof of Proposition 2.

The special case of Proposition 2, where M = 1 recovers the classical result for standard PLL (see the discussion at the end of Section 3 in [28]). The partial risk R 01 P ( f ; σ ) can be further bounded by its empirical counterpart, assuming that the Natarajan dimension is finite, as we have previously shown.

F Related Work

Partial Label Learning. Partial label learning (aka superset learning ) assumes the learner receives the training data in the form of ( x, z ) , where z ∈ 2 Y is a subset of the label space that contains the gold label. Technically, it is technically possible to cast our problem to standard PLL by identifying s as its preimages, namely the set of all labels that is consistent with the observation s . Nevertheless, our work extends the standard PLL in two ways. Firstly, our framework allows multiple instances to appear in one training sample. Secondly, the learnability of PLL is typically shown under the small ambiguity assumption, see, for example, [11, 28, 5]. This condition requires no distracting label to be contained in z with probability 1, which is violated in our case since a distracting label can always occur because σ is deterministic. Therefore, our proposed conditions relax and generalize small ambiguity by allowing deterministic transitions, see Appendix E.3.

Another line of work in weak supervision exploits the transition matrices [46] to compute posterior class probabilities in different scenarios, e.g., PLL [8, 9] and noisy label learning [61]. The key assumption is that the transition matrix is invertible [8, 46]. Our learnability condition, M -unambiguity, and the invertibility condition do not imply each other, see Appendix E.1 and E.2 for a self-contained discussion. There, we also show that small ambiguity and the invertibility condition do not imply each other, which might be of independent interest.

Latent structure models. Our work also relates to latent structural learning , where the aim is to learn a latent representation using supervision on a deterministic transition σ over the latent variables [45, 52]. To reduce the cost of computing σ , [42] proposes approximations for a restricted class of decomposable transitions. Both [42, 35] assume latent models in the exponential family. Our work is also related to SPIGOT [34, 32]. There, the aim is to learn both the transition and the latent model, proposing techniques for back-propagating through the argmax layer of the latent model. Complementary to the above research, we focus on rigorous theoretical analysis.

Our work is closely related to [62], which studies a similar problem of weakly supervised learning with multiple instances. However, our results are stronger and closer to the aims of weak supervision in comparison to the results in [62]. This is because our work proposes sufficient and necessary conditions to recover the hidden labels (i.e., Z 1: K in [62]), while consistency in [62] concerns the likelihood of the observed labels rather than the hidden ones, as defined in Definition 3 and 4 of [62], where two parameters are equivalent if they have the same likelihood on the observed partial label (rather than the hidden labels). Also, we use a different surrogate loss based on semantic loss that allows a topk approximation. In general, semantic loss is designed to capture Boolean constraints over the hidden labels and is standard to use in neuro-symbolic learning. Furthermore, our work directly extends the theory of PLL, which is thoroughly discussed in Appendix. In particular, our learnability condition from Definition 1 directly extends the classical small ambiguity degree condition, as shown in Appendix E.3. In contrast, the connection to PLL is missing from [62].

Constrained learning. The problem of partial label learning is closely related to constrained learning, in the sense that the prediction for the label (vector) is subject to the constraint σ ( y ) = s at the training stage. Training classifiers under constraints has been well studied in NLP. The work in [36] proposes a formulation for training under linear constraints; [37] proposes a learning framework that unifies expectation maximization with integer linear programming. Posterior regularization has been proposed for training classifiers under graphical models [21]. The latter two lines of research were adopted by [27] and [23] for neurosymbolic learning.

Neurosymbolic learning. Our work was motivated by frameworks that employ logic for training neural models [30, 53, 13, 59, 43, 31, 24, 27]. We prove learnability under (unknown) transitions that capture different languages: systems of Boolean formulas, non-linear constraints and logical theories. Notice that the key to model logical theories, e.g., Datalog, via σ is through abduction [26], see [43, 13] for a discussion. We also prove error bounds under a common neurosymbolic loss, the SL [56], and topk approximations. We are the first to provide rigorous results on neurosymbolic learning, closing a gap in the literature.

Notice that [58] also proves consistency of a topk loss. However, they use a zero-one-style loss. Finally, [3] provides generalization bounds of the smart predict-then-optimize (SPO) loss [15]; it assumes, though, that the gold labels of the inputs are given. Complementary to our research is the work in [19, 47, 33] that aims to integrate combinatorial solvers as differentiable layers in deep neural models. Finally, [48] proposes techniques for learning to solve combinatorial optimization problems using end-to-end neural models-loosing the ability to extract the latent models.

A subtle, yet substantial, difference between standard PLL and multi-instance PLL is that populating σ may incur a substantial computational overhead in the latter case. For instance, let us return back to Example 1. To populate the entries of σ for each target sum s , we need to compute all solutions of the equation X + Y = s , where X,Y ∈ { 0 , . . . , 9 } . In contrast, the set of labels associated with each input instance is part of the input [28, 11].

Learning logical theories. Another relevant strand of research is that of learning logical theories [12, 16, 38]. The work in [12] mixes abduction with induction to jointly learn a theory while training the neural classifiers, while [16] introduces a differentiable formulation of inductive logic programming that relies on the semantics of fuzzy logic for interpreting formulas. Finally, the authors in [38] propose learning rules using the notion of logical neural networks. The techniques in [12, 16, 38] rely on templates for learning logical theories. Instead of relying on pre-specified templates, the authors in [18] aim to learn those templates by mining patterns (aka motifs) from the data, following mathematically rigorous techniques and providing formal guarantees related to the mined motifs. Differently from our work, the above line of research provides no guarantees in terms of learnability or error bounds.

G Experiment Details

Firstly, recall that in all neurosymbolic frameworks considered in Section 6, we considered weights in { 1 , . . . , 5 } 10 and used an additional neural classifier to learn the unknown weights. As the weights are fixed, the inputs to the weight classifiers in all frameworks are constant signals, i.e., in each iteration, we provided the classifier with the same signal W i for the i -th weight. For each neurosymbolic framework, our implementation built upon the sources made available by the authors.

We now provide information about DeepProbLog, NeurASP and NeuroLog. Listings 1, 2 and 3 show the logic programs used to train the MNIST digit classifiers under the above three frameworks using the WEIGHTED-SUM problem, see Section 6.

DeepProbLog and NeurASP link the classifiers' predictions with the logic program via neural predicates . WEIGHTED-SUM requires two neural predicates: digit and weight . The first one classifies the input MNIST digits into { 0 , . . . , 9 } , while the second one classifies the input weights into { 1 , . . . , 5 } . In NeuroLog, the association between the classifiers' predictions and the logic program is maintained via abducible predicates. In particular, the instances of those predicates associate the probabilistic predictions with the facts from the domain of the logic program. Listing 3 uses two abducible predicates: atdigit and atweight . Atom atdigit ( s, j ) denotes that the j -th input to the digit classifier maps to value s . The abducible atom atweight ( s, j ) is analogously defined.

In terms of heuristics, in DeepProbLog, we used the Geometric Mean [31].

Following ENT's official implementation, we encoded WEIGHTED-SUM(2) using the Z3 SMT solver. In addition, similarly to the handwritten formula evaluation scenario in [27] (Section 5.1 from [27]), we used two projection operators. To generate the initial random labels, we projected out all the digit labels and kept only the weight labels. During random walks, we projected out the second

input digit's labels, keeping the labels of the first digit and the labels of both input weights. The rest of the hyperparameters, as well as the two-stage training process, were set as in the hand-written formula evaluation scenario from [27].

Neural architectures. The layers of the MNIST digit classifier are as follows: Conv2d(1, 6, 5) , MaxPool2d(2, 2) , ReLU(True) , Conv2d(6, 16, 5) , MaxPool2d(2, 2) , ReLU(True) , Linear(16 * 4 * 4, 120) , ReLU() , Linear(120, 84) , ReLU() , Linear(84, N) , Softmax(1) .

The neural classifiers for all frameworks but ABL were built using PyTorch 2.0.0 and Python 3.9. For ABL, we aligned with the implementation provided by the authors and used Tensorflow 2.11.0 and Keras 2.11.0. For each framework, we used the hyperparameters proposed by the authors in analogous scenarios.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors define and study the problem of multi-instance partial label learning (PLL), where weak supervision is given in the form of a (potentially) unknown transition function $\\sigma$, which maps the ground truth labels onto some label set $S$. \n\nUnder this problem setting, the paper goes on to show multiple theoretical contributions\n* 1) Under a known transition function (i.e. SUM2 in the MNIST task), a sufficient and necessary condition for learnability is M-ambiguity.\nTo get faster rates (removing the exponential term of 1/M), they show a stronger condition of 1-ambiguity (or that the output of the transition function given one label perturbation will also be changed).\nThey generalize this to consider a top-k loss that is a more efficient surrogate\n* 2) They study the setting of learning multiple classifiers with shared label spaces, and provide similar learnability results under analogies of M-ambiguity for multiple classifiers, with an additional assumption on the boundedness of the risk function.\n* 3) Finally, they provide a similar result for learning a single classifier under an unknown transition function, which requires a new assumption about the transition space being unambiguous and the boundedness of the risk function.\n\nThe paper also provides experiments to evaluate the quality of existing weakly supervised/neuro-symbolic architectures, which support their theoretical findings and takeaways.\n\nSTRENGTHS:\nThe paper provides comprehensive theoretical results for the PLL setting. They provide results for multiple different problem setups, considering known/unknown transition functions and learning a single/multiple classifiers. \n\nThey provide rigorous theoretical analysis, which requires (in my opinion) rather intuitive assumptions. \n\nRelevent experiments are nicely descriptive and verify their theoretical analysis.\n\nWEAKNESSES:\nAs mentioned in the paper, scalability seems to be an issue with the experimental setting. On what seems to be a not overly complex task (a weighted sum of 4 MNIST digits), performance indeed significantly drops, which calls into question the widespread potential of the PLL setting (and not this paper in particular). \n\nWhile the examples are given that illustrate when the various ambiguity assumptions are violated or not violated, a more in depth discussion in the case of Definition 5 would be appreciated, as this is the most interesting setting of an unknown transition function.\n\nQUESTIONS:\nAre the scalability issues as observed in Table 2 primarily due to the chosen neuro-symbolic methods, the PLL setting, or a combination of both factors?\n\nLIMITATIONS:\nLimitations are sufficiently addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the questions of learnability and generalization under an interesting form of supervision feedback: namely, when the true label of interest is not observed, but the learner instead has access to the output of a \"transition function\" $\\sigma(y_1, \\ldots, y_M)$ computed on the labels of an $M$-tuple of examples.\nThe paper gives necessary and sufficient conditions on the transition function for learnability from two \"partial\" (evaluated on the aggregate/partial labels) loss functions: partial 0/1 loss and a surrogate for the \"semantic loss\" studied in other related literature. These results are extended in multiple directions and various details such as convergence rates are also derived.\n\nSTRENGTHS:\n- The results are well presented and the key condition (M-unambiguity) is shown in Appendix E.3 to be a natural extention of the \"small ambiguity degree\" assumption from Partial Label Learning (PLL) literature.\n\n- The authors connect the convergence rate (in terms of sample complexity) of learning to the degree of unambiguity of the transition function.\n\n- The results provide a theoretical basis for learning with the top-$k$ approximate _semantic loss_, which was introduced by other work.\n\n- The core results are extended in multiple directions, including to the case where there are multiple different classifiers and to the case where the transition function is unknown (under additional assumptions).\n\n- Studies and provides a nice set of learnability results for an interesting case of weak supervision that could be practically relevant.\n\nWEAKNESSES:\n- Core technical contributions / difficulties are not explained enough (e.g., Lemma 1 seems like key result for Thm 1, and the rest follows from standard tools?) What was the key technical challenge to proving the generalization results, and why is it novel / original?\n\n- It's not clear what practical insights are gained from the results. The experiments section seems like a proof of concept---no new method other than the \"baselines\" is proposed or evaluated based on the theory. The main insight seems to be the qualitative relationship between $M$ and the bound, but $M$ is not really a controllable parameter in practice.\n\n- Related to the previous point about the experiments, this setup is missing a compelling application. What's a more realistic practical scenario where learning from this type of supervision is relevant? I.e., where observing $s$ for an $M$-tuple of examples is much more practical than observing $y$ for each example?\n\nQUESTIONS:\nIn the Remark starting at L172 the authors suggest a possible connection between the perturbation stability of the transition function and learnability / convergence rate. Could this be connected in any way to the algorithmic stability results (e.g., Bousquet and Ellisseef)? Such a connection might lead to better convergence rates?\n\n- Am I correct that the PAC failiure probabilities $\\delta$ are w.r.t. the sampling of $\\mathcal{D}_P$ (not $\\mathcal{D}$)? It might be good to clarify this.\n\nA few questions I had while reading the intro/setup, some of which were cleared up later by the main text, but suggest that clarity/flow could be improved:\n\nL117 Learnability. What is a \"partial learning algorithm\"?\n\nL138 should be $\\sigma_(y,\\ldots,y) \\ne \\sigma(y',\\ldots,y')$ ?\n\nL182 why do we need to \"populate $\\sigma$\" to compute the zero-one loss? I don't fully follow. In the sum-M case, can't we just sum the outputs predicted by $f(x_i)$ and then compare to $s$? Or is the point that this is problematic if we wanted to use a *surrogate loss* such as cross entropy, so we'd need to form a distribution over the image of $\\sigma$?\n\nLIMITATIONS:\nThe authors describe one of the main limitations of their work (the somewhat strict assumptions required on the transition function $\\sigma$ due to the worst-case distributions they consider). However, the discussion could be expanded to mention broader limitations, such as the current lack of practical insights to glean from the theoretical results.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a weakly supervised learning scenario where supervision signals are given to sets of instances (instead of individual instances), while the goal is still to predict labels of unseen individuals. For example, the learner is provided with a dataset in which each training example comprises a set of instances $(x_1, x_2)$ (for which the gold labels $y_1 = 1$ and $y_2 = 2$ are unobservable) and an aggregate signal $s=3$ (which is calculated by a known function $s=\\sigma(y_1,y_2)=y_1 + y_2$).\n\nThe authors present learnability results for this scenario under certain assumptions. One main result is that if the function $\\sigma$ has *M-unambiguity*, a perfect classifier can be learned as the number of training data approaches infinity. The authors further extend the results to situations where the function $\\sigma$ is unknown.\n\nSTRENGTHS:\n- This paper concerns an important problem setting called *multi-instance Partial Label Learning* (multi-instance PLL), which is an extension of the standard Partial Label Learning (PLL) problem.\n- The authors introduce the notion of \"M-unambiguity\" and show its utility in proving the learnability of the considered problem scenario under specific assumptions.\n- The authors further propose the notions of \"multi-unambiguity\" and \"unambiguous transition space\" to tackle the scenarios of learning multiple classifiers and unknown transition functions.\n\nWEAKNESSES:\n- The theoretical results presented in this submission have limited significance. Generally, learnability means that a hypothesis class is learnable under any data distribution. However, the authors state that they prove learnability under distributions that concentrate mass on a single instance or label. This assumption can be invalidated in the real world. This submission primarily considers a “toughest\" distribution that concentrates mass on a single instance, implying all the instances in the training data are the same. Such a distribution is unrealistic in real-world applications. Consequently, positive results under this assumption are not so meaningful. The authors may need to explore more realistic data distributions.\n\n- Theoretical aspects of the problem setting, which is called multi-instance Partial Label Learning in this submission, have previously been examined in the literature of weakly supervised learning, as seen in [1]. This setting is known as *learning from aggregate observations*. In [1], the consistency of learners is investigated, and the expected log-likelihood in [1] is close to the semantic loss studied in this submission. To avoid confusion and potential overlap, it is essential for the authors to discuss and elucidate the distinctions between these studies.\n\n- The terminology \"Multi-Instance Partial-Label Learning\" has been used in prior research, such as in [2]. Although [2] examines a different problem setting than this paper, to avoid confusion, it would be beneficial for the authors to adopt a distinct term when referring to the problem setting considered in this submission.\n\n- The proof of Lemma 1 appears to be convoluted. Specifically, the first inequality in Equation (26) warrants further clarification. It is not immediately evident why the assertion $R_{P}^{01}(f;\\sigma) \\ge \\sum_{l_i\\neq l_j} E_{l_i,l_j}(f)^M$ holds true. In a similar vein, could you explain the first inequality of Eq. (32) in the proof of Lemma 4? Do these Lemma rely only on the M-unambiguous condition?\n\n\n\n[1] Zhang, Y., Charoenphakdee, N., Wu, Z., & Sugiyama, M. (2020). Learning from aggregate observations. Advances in Neural Information Processing Systems, 33, 7993-8005. \n\n[2] Tang, W., Zhang, W., & Zhang, M. L. (2022). Multi-instance partial-label learning: Towards exploiting dual inexact supervision. arXiv preprint arXiv:2212.08997.\n\nQUESTIONS:\n- Is the expected log-likelihood discussed in [1] equivalent to the semantic loss (or the top-k surrogate loss) examined in this submission?\n\n- Could you provide a clearer explanation for the proof of Lemma 1, particularly regarding the first inequality in Equation (26)?  Could you also elaborate on the first inequality in Equation (32) in the proof of Lemma 4? Is it correct to state that the proofs of these lemmas solely rely on the M-unambiguous condition?\n\nLIMITATIONS:\nThe limitation of the submission is that it relies on assumptions and conditions, such as a concentration of mass on a single instance or label, which may not align with realistic data distributions and weaken the applicability and persuasiveness of the results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper connects latent structural learning and neuro-symbolic integration and provides the first theoretical study of multi-instance PLL with possibly an unknown transition. Under such weakly supervision scenario where the transition is deterministic, it defines the necessary and sufficient condition, the minimal distribution assumptions, for the ERM-learnability of the problem. They also derive the Rademacher-style error bounds using the top-k semantics loss. The empirical results support their theory.

STRENGTHS:
1. The study provides the first theoretical analysis of multi-instance PLL with possibly an unknown transition, which is a significant contribution to the learning theory community. 
2. The theory holds for both known and unknown transition cases, and the assumptions are necessary and sufficient. The experiment results also support the learnability of classifiers and the conclusion that when $M$ gets larger, the learning process gets harder.

WEAKNESSES:
1. The author better move some discussion of the necessity to let the transition function be deterministic to the introduction part to prevent confusion since it seems too strict compared with the commonly used assumption.
2. The empirical results are monotone for the weighted sum transition function, and the authors should add some other empirical results for more known transition function cases.

QUESTIONS:
1. The scalability in this scenario is poor as Table 2, and it seems to be worse than the error bound given by the theory. Will this happen for the known transition function case? What may be the possible reasons?

LIMITATIONS:
The author mentions that scalability may be an important issue. No further negative societal impact raises.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06