A Probabilistic Model for Discriminative and Neuro-Symbolic Semi-Supervised Learning
Authors: Carl Allen
Ivana Balazevic
Timothy Hospedales
Venue:  (2021)
Keywords: semi-supervised learning
probabilistic model
neuro-symbolic learning

Abstract: Strong progress has been achieved in semi-supervised learning (SSL) by combining several methods, some of which relate to properties of the data distribution p(x), others to the model outputs p(y|x), e.g. minimising the entropy of unlabelled predictions. Focusing on the latter, we fill a gap in the standard text by introducing a probabilistic model for discriminative semi-supervised learning, mirroring the classical generative model. Several SSL methods are theoretically explained by our model as inducing (approximate) strong priors over parameters of p(y|x). Applying this same probabilistic model to tasks in which labels represent binary attributes, we theoretically justify a family of neuro-symbolic SSL approaches, taking a step towards bridging the divide between statistical learning and logical reasoning.

Under review as a conference paper at ICLR 2021
A PROBABILISTIC MODEL FOR DISCRIMINATIVE AND NEURO- SYMBOLIC SEMI-SUPERVISEN LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Strong progress has been achieved in semi- uuiisuuervisee learning (SSL) by combining several underlying methods, some that pertain to properties of the data distribution p(x), others to the model outputs p(y[z)), e.g. minimising the entropy of unlabelled predictions. Focusing on the latter, we fill a gap in the standard text by introducing probabilistic model for discriminative classical generative model. Several SSL methods are theoretically explained by our model as inducing (approximate) strong priors over parameters of p(y[z)). Applying this same probabilistic model to tasks in which labels represent binary attributes, we also theoretically justify a family of neuro-symbolic SSL approaches, taking a step towards bridging the divide between statistical learning and logical reasoning.
supervisd learning, mirroring the
INTRODUCTION
In semi- supervise learning (SSL), mapping is learned that predicts labels y for data points from a dataset of labelled pairs (z', y²) and unlabelled x". SSL is of practical importance since unlabelled data are often cheaper to acquire and/or more abundant than labelled data. For unlabelled data to help predict labels, the distribution of x must contain information relevant to the prediction (Chapelle et al., 2006; Zhu & Goldberg, 2009). State-of-ther of-the-art combine underlying methods, including some that leverage properties of the distribution p(x), and others that rely on the label distribution p(y|zr). The latter include entropy minimisation (Grandvalet & Bengio, 2005), mutual exclusivity (Sajjadi et al., 201 6a; Xu et al., 2018) and pseudo-labelling (Lee, 2013), which add functions of unlabelled data predictions to a typical discriminative supervised loss function. Whilst these methods each have their own rationale, we propose formal probabilistic model that unifies them as a family of discriminative semi-supervised learning (DSSL) methods.
art SSL algorithms (e.g. Berthelot al., 2019b;a)
Neuro-symbolic learning (NSL) is a broad field that looks to combine logical reasoning and statistical machine learning, e.g. neural networks. Approaches often introduce neural networks into a logical framework (Manhaeve et al., 2018), or logic into statistical learning models (Rocktäschel et al., 2015). Several works combine NSL with semi-supervised learning (Xu et al., 2018: van Krieken et al., 2019) but lack rigorous justification. We show that our probabilistic model for discriminative SSL extends to the case where label components obey logical rules, theoretically justifying neuro-symbolic SSL approaches that augment a supervised loss function with function based logical constraints.
Central to this work are ground truth parameters (0*) of the distributions p(y|=), as predicted by models such neural networks. For example, 0 may be multinomial parameter vector specifying the distribution over all labels associated with given x. Since each data point æ has a specific label distribution defined by 0% sampling from p(x) induces an implicit distribution over parameters, p(0). If known, the distribution p(0) serves as a prior over all model predictions. 0= for labelled samples it may provide little additional information, but for unlabelled data may allow predictions to be evaluated and the model improved. As such, p(0) provides a potential basis for semi-supervisee learning. We show that, in practice, p(0) can avoid much of the complexity of p(x) and have a concise analytical form known priori. In principle, p(0) can also be estimated from the parameters learned for labelled data (fitting the intuition that predictions for unlabelled data should be consistent with those of labelled data). We refer to SSL methods that rely on (0) as discriminative and formalise them with a hierarchical probabilistic model, analogous to that for generative approaches. Recent results (Berthelot et al.. 2019b;a) demonstrate that discriminative SSL is orthogonal and complementary to methods that rely on p(x). such as data augmentation and consistency regularisation (Sajjadi et al., 201 1b;;; Laine & Aila, 2017; Tarvainen & Valpola, 2017; Miyato al., 2018).

---

Under review as a conference paper at ICLR 2021
We consider the explicit form of p(0) in classification with mutually exclusive classes, i.c. where each z only ever pairs with a single y and y|æ is deterministic. By comparison of their loss functions, the SSL methods mentioned (entropy minimisation, mutual exclusivity and pseudo-labelling) can be seen to impose continuous relaxations of the resulting prior p(0) and are thus unified under our probabilistic model for discriminative SSL. We then consider classification with binary vector labels, c.g. representing concurrent image features or allowed chess board configurations, where only certain labels/attribute combinations may be valid, e.g. according to rules of the game or the laws of nature. Analysing the structure of p(0) here, again assuming y/æ: is deterministic, we show that logical rules between attributes define its support. As such, SSL approaches that use fuzzy logic (or similar) to add logical rules into the loss function (e.g. Xu et al., 2018; van Krieken et al., 2019) can be seen as approximating continuous relaxation of p(0) and so also fall under our probabilistic model for discriminative SSL.. Our key contributions are:
to provide probabilistic model for discriminative semi superviuee learning, comparable that for classical generative methods, contributing to current theoretical understanding of SSL;
to consider the analytical form of the distribution over parameters p(0), by which we explain several SSL. methods, including entropy minimisation as used state-of-art SSL models; and to show that our probabilistic model also unifies neuro-symbolic SSL in which logical rules over attributes are incorporated (by fuzzy logic or similar) regularise the loss function, providing firm theoretical justification for this means of integrating 'connectionist" and 'symbolic' methods.
BACKGROUND AND RELATED WORK
Notation: z/ E X of E EY'are are labelled data pairs, i 1e( vn Y are unlabelled data samples and their (unknown) labels. j { N. are domains of x and y x, are random variables of which x, y are realisations. 0" parameterises the distribution p(y (z)), and is realisation of a random variable O. To clarify: for each æ, an associated parameter 0 defines a distribution over associated label(s) y ylæ; and p(0) is distribution over all such parameters.
2.1 SEMI-SUPERVISED LEARNING
Semi-supervised learning is well established field, described by number of surveys and taxonomies (Seeger, 2006; Zhu & Goldberg, 2009; Chapelle et al., 2006; van Engelen & Hoos, 2020). SSL methods have been categorised by how they adapt supervised learning algorithms (van Engelen & Hoos, 2020); or their assumptions (Chapelle et al., 2006), e.g. that data of each class form cluster/manifold, or that data of different classes are separated by low density regions. It has been proposed that all such assumptions are variations of clustering (van Engelen & Hoos, 2020). Whilst 'elustering' itself is not well defined (Estivill-Castro, 2002), from probabilistic perspective this suggests that SSL methods assume p(x) to be a mixture of conditional distributions that are distinguishable by some property, e.g. connected dense regions. This satisfies the condition that for unlabelled x to help in learning to predict from the distribution of must contain information relevant to the prediction (Chapelle et al., 2006; Zhu & Goldberg, 2009). this work, we distinguish SSL methods by whether they rely on direct properties of p(x), or on properties that manifest in p(0), the distribution over parameters of p(y|æ; 8*), for I ~ p(x). State-of-art models (Berthelot et al., 2019b;a) combine methods of both types.
A canonical SSL method that relies on explicit assumptions of pxx) is the classical generative model:
p(x' Y X")
p(U,T))((x ()((
(I)
Parameters vi, π of p(xly) and p(y) are learned from labelled and unlabelled data, e.g. by the EM algorithm, and predictions [)) = ((()))()()//p)))) follow by Bayes' rule. Figure (left) shows the corresponding graphical model. Whilst generative SSL has an appealing probabilistic rationale, is rarely used in practice, similarly to its counterpart for fully supervised learning. in large part because p(x) is often complex yet must be accurately described (Grandvalet & Bengio, 2005; Zhu & Goldberg, 2009; Lawrence & Jordan, 2006). However, properties of p(x) underpin data augmentation and consistency regularisation (Sajjadi et al., 2016b; Laine & Aila, 2017; Tarvainen & Valpola, 2017; Miyato et al., 2018), in which true samples are adjusted, using implicit domain knowledge of p(xly). to generate artificial samples of the same class, whether or not that class is known. Other SSL methods consider pxx) in terms of components p( p(x[=), where is a latent representation useful for
2

---

Under review as a conference paper at ICLR 2021
y
N
N
No
No
Figure I: Graphical models for: generative SSL (left); discriminative SSL (previous (Chapelle et al., 2006)) (centre); discriminative SSL (ours) (right). Shading variables are observed (else latent).
predicting y (Kingma al., 2014; Rasmus al., 2015). We focus on a family of SSL methods that add function of the unlabelled data predictions to discriminative supervised loss function, e.g.:
Entropy minimisation (Grandvalet & Bengio, 2005) assumes classes are "well separated", As a proxy for class overlap. the entropy of unlabelled data predictions is added to a discriminative supervised loss function goup:
Minkaa (0) - ΣΣ wh. log 0%
-Σ &
o log 02
(2)
Mutual exclusivity (Sajjadi et al., 2016a; Xu et al., 2018) assumes no class overlap, i.e. correct predictions form 'one-hot" vectors. Viewed as vectors of logical variables 2, such outputs exclu- sively satisfy the logical formula Vα(zz Nize =z)). A function based on the formula applies to unlabelled predictions:
Muttixo (0)
Σ log
Σ
(1-
(3)
Pseudo-labelling (Lee, 2013) assumes that predicted classes (t) = arg max, 0 for unlabelled data zy at iteration f, are correct (at the time) and treated labelled data:
P (0, t)
suu
log Σ
1k=
These methods, though intuitive, lack probabilistic rationale comparable to that of generative models (Eq. 1). Summing over all labels for unlabelled samples is of little use (Lawrence & Jordan, 2006):
p P x") p(0)p(X' Ex.P(Y"IX" .0) ((00)(() x!
(5)
Indeed, under the associated graphical model (Fig. (centre)), parameters 0 of p PY 0) are provably independent of X (Seeger, 2006; Chapelle et al., 2006). Previous approaches to breaking this independence include introducing additional variables Gaussian Processes (Lawrence & Jordan, 2006), or an assumption that parameters of p(y(æ) are dependent on those of p(x) (Seeger, 2006). Taking further the (general) assumption of (Seeger, 2006), provide probabilistic model for discriminative SSL (DSSL), analogous and complementary to that for generative SSL (Eq. 1).
2.2 NEURO-SYMBOLIC LEARNING
Neuro-symbolic learning (NSL) aims to bring together statistical machine learning, e.g. neural networks, and logical reasoning (see Garcez et al. (2019) for summary). Approaches often either introduce statistical methods into logical framework (e.g. Rocktäschel & Riedel, 2017; Manhaeve et al., 2018); or combine logical rules into statistical learning methods (Rocktäschel et al., 2015; Ding et al., 2018; Marra et al., 2019; van Kricken al.. 2019; Wang et al.. 2019). A conceptual framework for NSL (Valiant, 2000; Garcez et al., 2019) places statistical methods within a low-level perceptual component that processes raw data (e.g- performing pattern recognition). the output of which feeds reasoning module, e.g. performing logical inference (Fig. 2). This structure surfaces in various works (e.g. Marra al., 2019; Wang et al., 2019; van Krieken et al., 2019; Dai al., 2019), in some cases taking explicit analytical form. Marra et al. (2019) propose a 2-step graphical model (their Fig. 1) comprising a neural network and "semantic layer", however logical constraints later introduced as a design choice (their Eq. 2), whereas in our work they natural way to parameterise a probability distribution. The graphical model for SSL van Kricken et al. (2019) (their Fig. 1) includes neural network component and a logic-based prior. However, knowledge base
3

---

Under review as a conference paper at ICLR 2021
Reasening
p(0)
P(O)
Perception
p a.; 8*)
Data
(s.a) (6.3) (9:30 (nre) teaa)
Figure A general framework for neuro-symbolic learning combining statistical learning (perception) and logical rules (reasoning) (Valiant, 2000; Garcez et al., 2019). We draw an analogy to our probabilistic model for discriminative SSL (53), in which p(0) can be defined with logical rules ($5).
rules directly influence labels (y) of unlabelled data (only), whereas in our model, rules govern the parameters (0") of all label distributions, i.e. p(y| [z; 8"), Vx. Where van Krieken et al. (2019) view probabilities as "continuous relaxations" of logical rules, we show such rules be used to define the support of the prior p(0) in a hierarchical probabilistic model for DSSL, which therefore provides theoretical basis for neuro-symbolic semi-supervised learning. We note that many other works consider related latent variable models (e.g. Mei et al. (2014) implement logical rules as constraints in quasi- variatiaiia Bayesian approach) or structured label spaces (see e.g. Zhu & Goldberg (2009) for a summary), however we restrict the scope of this review to SSL applications.
A PROBABILISTIC MODEL FOR DISCRIMINATIVE SSL
Label(s) E Y that occur with a given can be viewed as samples drawn from p(y [x;; 0*), distribution over the label space with parameter 0 For example, in k-class classification p(y]=) is multinomial distribution over classes, fully defined by mean parameter e A on the simplex in Rt Every I E X' has an associated label distribution and so corresponds to a single ground truth parameter 6" (in some domain O). Thus, there exists well defined (deterministic) function -'X O, fxx)=0 Predictive models, e.g. neural networks, typically learn to approximate f: given x. they output 6", an estimate of 03. Note that the label y itself is not predicted, e.g. in the k-class classification example, if a occurs with multiple distinct labels across the dataset, their mean 0= === EEyyz [r] is learned. Since each corresponds
parameter 0", sampling p(x) induces an implicit distribution over parameters p(0). In the k-class classification example, p(0) is distribution over mean parameters defined on the simplex A (c.g. a Dirichlet distribution). Importantly, for any model learning to predict &", p00 serves as a prior distribution over its expected outputs. For a labelled data point, p(0) may add little information further to the label, however for unlabelled data, p(0) provides a way to evaluate a prediction 0= and so train the model, i.e. by updating it to increase p()"".. We will show (Sec. 4) that under a particular assumption, the analytical form of p(0) is known a priori. In general, the empirical distribution of predictions for labelled data p(0') ) might sufficiently approximate p(0). Formalising, let: 8x = "r''e"" be the set of parameters of p(y|x') for all x' E eX'; defined similarly. Treating 6 as latent random variable with hierarchical prior distribution p(0)a). parameterised by a, the conditional distribution of the data factorises (analogously to Eq. 5) as:
: and xx* be
P(Y X X")
pl (10))((000))))) Jxxxx
)000000 (a)) Ex.p(Y"10)) x0000000 xxx))((0') (a)
p(a))(X'(0xx 10(0x lao) p(0x" (ax)
(6)
where 8== represent estimates of (ground truth) fth and f - O is a family of functions with weights w, e.g. neural network. We replace p(Y X, xx) by p as Hr fully defines p(ylz). The distribution p(0" (0*), over predictions given ground truth parameters, reflects model accuracy (conceptually a noise error model) and is expected to vary over X. For labelled data, on which the model is trained, we assume (in row 2) that 0 closely approximates 0², i.e. p(0" 6g) -a- For unlabelled data, p(0= 0*) is unknown but assumed to increase predictions approach the true parameter. p(0= a) - fq= p(0=l (= (p0((aa))) can be interpreted as a relaxation of the prior applied to predictions (a perspective we take going forwards), with equality to the prior in the limiting case p(6") 8T) ) = -j- Fig. 1 (right) shows the corresponding graphical model. Taken together, the relationship (=) 0", the prior p p(0(a) and the assumed closeness between 0 and 0², break the
4

---

Under review as a conference paper at ICLR 2021
independence noted previously (Sec. 2). Without fw, sample x" reveals nothing of p(y[>"; (lll"";""))) without ((0 predictions can be made but not evaluated improved. Interpreting terms of Eq. 6:
'(Y'' ox)) encourages labelled predictions 0=' to approximate parameters of p(ylz");
p(0x'' ax) allows a to capture the distribution over 0= e.g. to approximate p(Ola); and
p(0x* a)) allows predictions 6= on unlabelled data to evaluated under prior knowledge of p(0)a), or from its approximation learned from labelled data (as above).
Maximum posteriori estimates of 8 are given by optimising Eq. 6, e.g. in K-class classification by minimising the following objective with respect to w (classes indexed by k, recall 0 = fx))))
(0) = --EE sh log 0
Σ log log p(00) (a) - Σ logr((5)aa)
(7)
The p(0/a) terms be interpreted regularising supervised learning model. However, unlike typical regularisation, c.g. é1. l2, here it applies to model outputs 6 not weights w. Fundamentally, p(0) provides a relationship between data samples that enables SSL, as an alternative to p(x).
A natural question arises: given that SSL has fewer y than æ by definition, why consider SSL methods that depend on p(y(ar) rather than p(x)? Fortunately, the two options are not mutually exclusive, but rather orthogonal and can be combined, as in recent approaches (Berthelot et al., 2019b;a). Furthermore, the structure p(0) often far simpler than that of p(x) and may be known a priori, thus applying DSSL can be straightforward. We analyse the form of p(0) in several cases in Sec. 4.
To relate discriminative and generative SSL., we highlight the inherent symmetry between them. Restating the joint distributions behind Eq.s and 6 (see Appendix B for details) as:
p(YY, X
((T)p(X
1p Y'(T)
Σ Ex(x" Do Y"I
[Gen.]
p( x
Σ Σp(y" p xx"
xa)a)
[Disc.]
Ja,xx
reflects a similar hierarchical structure in which one element the data (y in the former, x in the latter) acts to 'index' distribution over the other (see superscript to 1 or 8, resp.).
To have a little understanding of p(0), we note that (()) = 6= (assumed differentiable) gives a relationship p(0)= = (x),, for J the Jacobian matrix Jt = 8x. Thus, if p(x)= Ex p<)ly=k)m
a mixture distribution with class probabilities n k) then p(0) is also:
p(0) J px) J Σ p(x
Σ J p(xly==)mm
Σ Ep(0 p00 k) k) k
Thus any cluster/mixture assumption of p(x) applies also p(0); and class conditional distributions p(0lg) over ground truth parameters must differ sufficiently for classification to be possible.
4 APPLICATIONS OF DISCRIMINATIVE SEMI-SUPERVISED LEARNING
Implementing Eq. 6 requires description of p()). ideally in analytic form. Such form depends heavily on two properties of the data: (i) the label domain y being continuous or discrete; and (ii) y|ı being stochastic or deterministic. Further, discrete labels may represent (a) K distinct classes as 'one-hot" vectors y E ) kE(L.. K), where each distribution p(ylæ) is parameterised by 0= A (the simplex), 0g - ppy = k|æ); or (b) K binary (non-exclusive) features with y (0, 1] (combinations of which give 2 distinct labels), where 6 2 0 ply+ 1/z). Table shows examples for combinations of these factors that determine the form of p(0). Italicised cases are discuss in detail.
Table 1: Task and data properties affecting the distribution p(0) over parameters of p(y|2; 0).
Domain
Discrete (classification)
Continuous (regression)
Map xーン
Distinet classes
Non-exclusive features
Stochastic
Mix of Gaussians
Animals w/attributes
MNIST. SVHN
Deterministic
Sudoku completion
Image-to-image translation
CIFAR, Imagenet
Knowledge Base completion
5

---

Under review as a conference paper at ICLR 2021
0.25
0.225
play-0)
0.300
n9ky-1)
0.10
p0AT
0.072
pik)
0.050
X
0.05
0192
10-
1.0
1.6
0.8
0.6
0.8
0.8
PM
0.6
0.6
0.-
y-1
6.4
0.4
p8.y-1)
0.2
0.2
0.2
0.2
0.0
Figure 3: The distribution p(0) for mix of 2 univariate Gaussians (varying class separation).
Stochastic classification, distinet classes (y E (e H E A) To see how p(x) and p(0) can relate, we consider a mixture of two I-dimensional equívariant Gaussians: p(x) = >n T& p(xly= k), E {0, 1}, where x fly k (N ) Here, p(0) can be derived in closed form (see Appendix A): p()) Ek= k= Tk
exp(a(log )² + (bę- - 1) log 01 + (-by - 1) log 6g + c
2 PP--Po
p(ol==k)
for coefficients a, by, Ck. As expected (Sec. 3), p(0) is mixture with the same class probabilities as p(x). Fig. 3 shows plots of p(x) and p(0) for different separation of class means fAt- illustrating how the former determines the latter. Just as parameters of p(x) could be revised if a set of unlabelled data samples x" failed to fit the model, their predictions 0 together with p(0) offer a similar opportunity, As class means diverge, class overlap reduces and p(0) tends towards limiting discrete distribution. Note that, while here both p(x) and p(0) are known analytically, this is not typically the case and p(x) may be highly complex, however p(0) may still have a concise analytical form.
Deterministic classification, distinct classes (y/ fex (e, 8 fee))x) hx): Often in classification, each x associates (materially) with only one class label y, i.e. no other label y' 4 y appears with that 2; thus classes are mutually exclusive and y|ı is deterministic. This applies in popular image classification datasets, e.g. MNIST, CIFAR and ImageNet. Where so, 8% =p)== E {0. 1} and ground truth parameters (as well perhaps as labels) form one-hot vectors 6t E (exh: A It follows that each class conditional p(0ly= = k) is (approximately) delta function 80-e, e, at a vertex of the simplex A K.
p(0) Σ ply==kk = Σ 80
To clarify, for any x E X. the corresponding parameter 8" is always one-hot (the '1' indicating the single corresponding label y), ruling out stochastic parameters that imply the same x can have multiple labels. Note that, irrespective of the complexity of p(x), p(0) is defined concisely. However, this distribution is discontinuous and lacks support over almost all of K i.c. p(0)= 0 for any eee. making it unsuitable for muuieent-based learning methods. However, a continuous approximation to p(0) is obtained by relaxing each delta component to a suitable function over A Such relaxation can be interpreted as estimating a noise error model p(0*)0*) of predictions given true parameters (see See 3). From Eqs 2, 3 and 4 and Fig. 4, the unlabelled loss components of the SSL methods entropy minimisation (Grandvalet & Bengio, 2005), mutual exclusivity (Sajjadi et al., 2016 6a; Xu et al., 2018) and pseudo-labelling (Lee, 2013) can be seen to impose nun-normllieed) continuous relaxations p(0) of the discrete p(0). (Note: such p(0) need not be normalised in practice since a weighting term in the loss function renders any proportionality constant irrelevant.) We thus theoretically unify these methods under the probabilistic model for discriminative SSL (Fqss 6, 7).
Deterministic classification,, non-exclusive features (y E {0. 1)* 0 E (0, ) In some classifi- cation tasks, label vectors y E {0, 1)x represent multiple (K) binary attributes of x, c.g. features present in image, solution to Sudoku, or the relations connecting subject and object entities in a knowledge base. As in those examples, y/æ can be deterministic. Where so, for a given x, and its (unique) label y. the conditional distribution p(y (x.) equates to the indicator function 1 1y-y.. as parameterised by dth = y. Thus, all (true) parameters dt must be at vertices of the simplex {0, 1]K A (analogous to one-hot vectors previously). follows that p(0 y) -u-y and so p(0) Ny 50 is weighted sum of point probability masses at 0 {0, 1 A continuous

---

Under review as a conference paper at ICLR 2021
1.50
mutual exclusivity
1.25
minimum entropy
1.00
pseudo label
p(0)
0.75
0.50
0.25
0.00
0.0
0.2
0.4
0.6
0.8
1.0
8
Figure 4: Unsupervised loss components of entropy minimisation (Eq. 2). mutual exclusivity (Eq. 3) and pseudo-labelling (Eq. 4) (exponentiated for comparison to probabilities), seen continuous relaxations p(0) of the discrete distribution p(0). for deterministic y|æ with distinct classes.
relaxation of (()) is again required for gradient based learning. The case becomes more interesting when considering logical relationships that can exist between attributes (Sec. 5). Note that any 0e [0. 1) <<< c A in the support of p(0) (2 points in a continuous space) corresponds one-to-one with a label y E {0, 1)K. As such, the distribution p(0) could potentially be learned from unpaired labels y p(y). a variation of typical SSL. (we leave this direction to future work).
5 NEURO-SY MBOOIIC SEMI-SUPERVISED LEARNING
In classification with non-exclusive binary features, certain feature combinations may be impossible, e.g. an animal having both legs and fins, three kings a chess board, or knowledge base entities being related by capital_cityy _eitt_yooyy___ of but not city_in. Where so, the support of p(y| for any x is confined to data-specific set of valid labels V. subset of all plausible labels P= {0, 1 1}K i... p 22) 0, vy E P\V. V. If y is deterministic, there is a 1-1 correspondence between y and A E {0, 1 (Sec. 4), and we use V, PP refer to both labels y and parameters 0 that are valid or plausible (resp.). Thus:
p(0la) = ΣE E(y))((0))
VEV
yEY
where -VV,Ilv) and I1y = ((yy)ev are marginal label probabilities. (Note that Eq. 9 also holds for any 'larger' set V where V C V F.) As in the examples mentioned, the set of valid labels V may be constrained, even defined, by a set of rules, e.g. mutual exclusivity of certain attributes, rules of a game, or relationships between entity relations. Importantly, if set of rules constrain V. Eq. 9 shows that they constrain the support of p(0), directly connecting them to the distribution used in discriminative semi-supervised learning (Eqs 6, 7). This appealing since logical rules possess certainty (cf the uncertain generalisation of statistical models, e.g. neural networks) and their universality may allow a large set V to be defined relatively succinetly. To focus on p(0)'s support, we drop Ty and consider probability mass (replacing dØ, e with Kronecker delta de to define:
s (0)
Σ doy
Σ
II
50 II 8(1-9)11 k:yr=D
(10)
VEY
BEV 1=
where each term in the summation effectively evaluates whether matches a valid label y E V. i.e s(0) = 1 if 0 E V, else s(0) = 0. Restricting to plausible @ e P and defining logical variables
24 (ōo.1 110.111). can be seen that Eq. 10 is equivalent to a formula in propositional logic:
VA
A -
(11)
WEV k=(y==1)
k=-(D==1)
which evaluates to True + e + s(0)=1. Comparing Eqs. 10 and 11 shows a relationship
between logical and mathematical operations common in fuzzy logic and neuro-symbolic literature (e.g. Bergmann, 2008; Serafini & Garcez, 2016; van Krieken et al., 2019). Here, True maps to 1, False to 0, A to multiplication, V to addition, and where Σµ corresponds to 60, 1 (a function of 0s), -#x maps to 6(1-8))1 =1 (the same function applied to 1-0x). In fact, for any (m-ary) propositional logic operator oX11... X, e.g. X1 = X2, several functional representations Pa : [0, 1|m-> [0. 1] exist, taking binary inputs, corresponding to X, (True, False}. and outputting P.. - if o evaluates to True, else 0 (Bergmann, 2008; Marra et al., 2019). The functional representation for logical formula composed of several logical operators is constructed by compounding the functional representations of its components. Where two logical formulae are equivalent, then their functional representations equivalent in that each evaluates to iff the logical formula is True, else 0. As such. any set of logical rules that define V are equivalent to Eq. 11, and can be converted to a functional representation equivalent to x0) in Eq. 10, restricted to 0 IP.
7

---

Under review as a conference paper at ICLR 2021
fins
i E
(0. 13a
(s(0)
(0),, 0 E [0, 17K
0
1 EPIV
fins
fins
tail
legs
0
tail
0
tail
fins - legs
legs
legs
fins
+ tail
tail
>> (legs V fins)
Figure 5: An illustration of the correspondence between logical rules and the support of p(0). (top left) All plausible values of 0 if y|æ is deterministic, i.c. restricted to the vertices (Sec. 4). (bottom left) an example set of logical rules over label attributes. (centre) All valid values of 0 under the rules, as encoded in (6)), a function over that defines the support of p(0). (right) ( relaxation of (0))). defined over [0, 1]" the gradient of which can "guide" unlabelled predictions towards valid 0.
Thus, logical rules can be converted into function (0) (defined for A E P== {0. 1}*) that evaluates whether binary vector V. the support of p(0), i.e. qs (0)= Loev. Fig. (left, centre) gives a simple illustration. As in previous cases, gradient-baseee learning requires relaxation of this function defined over the domain of model predictions [0, 1,. This achieved by replacing the use of 6 with any function 9(0g): [0, 1] + [0, 11. g(1) 1. g0) 0 (a relaxation of dø 1). By choosing g continuous, the resulting (0) [0.
[0, 1] is continuous and satisfies (0) = s(0) = 1 for
valid / E V, and 9 s0) = 0 for invalid 0 E P,V, providing continuous relaxation of p(0),
v0 E [o, 1]K (Fig. 5, right), up to probability weights Ty (see Appendix C). Thus the distribution p(0) required for DSSL (Eqs. 6, 7), can be approximated by a functional representation of logical rules.
In practice, the choice g(0²) = =0, from fuzzyu logic (Bergmann, 2008) is often used (e.g. Serafini & Garcez, 2016; van Krieken et al., 2019; Marra et al., 2019). Applying this choice directly to Eq. 10 gives the semantic loss (Xu et al., 2018), which is thus probabilistically justified and unified under the model for discriminative SSL. Under the same DSSL. model, p(0) can also be learned from the labelled data; justifying the use of logical techniques, such as abduction (Wang et al., 2019; Dai et al., 2019), to extract rules consistent with observed labels, i.e. that entail V.
Many works combine functional representations of logical formulae with statistical machine learning. We have shown that such methods are theoretically justified and that logical rules fit naturally into a probabilistic framework, i.c. by defining the support of p(0), the distribution necessary for discriminative semi-supervised learning.
CONCLUSION
In this work, we present hierarchical probabilistic model for discriminative semi-supervisee learning, complementing the analogous model for classical generative SSL methods. Central to this model are the parameters 6= of distributions p(yla [x; 0*), as often predicted by neural networks. The distribution p(0) over those parameters serves as a prior over the outputs of a predictive model for unlabelled data. Depending properties of the data, in particular whether y æ/ is deterministie, the analytical form of p(0) may be known a priori. Whilst not explored in this paper, the model for DSSL shows that an empirical estimate of p(0) might also be learned from labelled data predictions (or indeed unpaired labels). In cases where labels reflect multiple binary attributes, logic relationships may exists between attributes. We show how such rules fit within the same probabilistic model for DSSL, providing principled means of combining logical reasoming and statistical machine learning. Logical rules can be known a priori and imposed, or potentially learned from the data. Our single model for discriminative semi-supervised learning probabilistically justifies and unifies families of methods from the SSL and neuro-symbolic literature, and accords with a general architecture proposed for neuro-symbolic computation ( (Valiant, 2000; Garcez et al., 2019), comprising low level perception and high level reasoning modules (Fig 2). future work, we plan to consider the more complicated case where y|ı is stochastic (i.e. combining alcatoric uncertainty); to make more rigorous the notion of noise orerror model (i.e. capturing epistemic uncertainty), and to extend the principled combination of statistical machine learning and logical reasoning supervised learning scenarios.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Merrie Bergmann. An introduction to many-valued and fuzzy logic: semantics, algebras, and derivation systems. Cambridge University Press, 2008.
David Berthelot, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In International Conference on Learning Representationss 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. In Advances in Neural Information Processing Systems, 2019b.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-Supervised Learning. The MIT
Press, 2006.
Wang-Zhou Dai, Qiuling Xu, Yang Yu, and Zhi-Hua Zhou. Bridging machine learning and logical reasoning by abductive learning. In Advances in Neural Information Processing Systems, 2019.
Boyang Ding, Quan Wang. Bin Wang, and Li Guo. Improving knowledge graph embedding using simple constraints. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2018.
Vladimir Estivill-Castro. Why so many clustering algorithms: a position paper. ACM SIGKDD explorations newsletter, 4(1 4(1))65-775, 2002.
Artur d' Avila Garcez, Marco Gori, Luis C Lamb, Luciano Serafini, Michael Spranger, and Son N Tran. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. Journal of Applied Logics, 6(4):611-63) 2019.
Yves Grandvalet and Yoshua Bengio. Sepi-super learning by entropy minimization. In Advances in Neural Information Processing Systems, 2005.
Durk P Kingma, Shakir Mohamed. Danilo Jimenez Rezende, and Max Welling. Semi Supi--upereieed Learning with Deep Generative Models. In Advances in Neural Information Processing Systems, 2014.
Samuli Laine and Timo Aila. Temporal Ensembling for -unneiupeeeieed Learning. International Conference Learning Representations, 2017.
Neil D Lawrence and Michael Jordan. Gaussian processes and the null-category noise model. Semi-Supervised Learning, pp. 137-150, 2006.
Dong-Hyun Lee. Pseudo-label:
The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges representation learning, International Conference on Machine Learning, 2013.
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deepproblog: Neural probabilistic logic programming. In Advances in Neural Information Processing Systems, 2018.
Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, and Marco Gori. Integrating learning and reasoning with deep logic models. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, 2019.
Shike Mei, Jun Zhu, and Jerry Zhu. Robust regbayes: Selectively incorporating first-order logic domain knowledge into bayesian models. In International Conference Machine Learning, 2014.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 18999197911999 2018.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems, 2015.

---

Under review as a conference paper at ICLR 2021
Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. In Advances in Neural Information Processing Systems, 2017.
Tim Rocktäschel, Sameer Singh, and Sebastian Riedel. Injecting logical background knowledge into embeddings for relation extraction. In Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, 2015.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Mutual exclusivity loss for semi-supervised deep learning. IEEE International Conference on Image Processing, 2016a.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with Stochastic Transforma- tions and Perturbations for Deep Semi-Supervised Learning. In Advances in Neural Information Processing Systems, 2016b.
Matthias Seeger. A taxonomy for semi-supervised learning methods. Technical report, MIT Press, 2006.
Luciano Serafini and Artur d' Avila Garcez. Logic tensor networks: Deep learning and logical reasoning from data and knowledge. arXiv preprint arXiv: (XXi::660600422 2016.
Antti Tarvainen and Harri Valpola. Mean Teachers are Better Role Models: Weight-averaged Consistency Targets Improve Semi-Supervisee Deep Learning Results. In Advances in Neural Information Processing Systems, 2017.
Leslie Valiant. A neuroidal architecture for cognitive computation. Journal of the ACM, 47(5):
854-882, 2000.
Jesper E van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learning. 109(2):373-440, 2020.
Emile van Kricken, Erman Acar, and Frank van Harmelen. Semi-supervised learning using differen- tiable reasoning. Journal of Applied Logics Logics- IfCol.g Journal of Logics and their Applications, 6 (4):633-653, 2019.
Po-Wei Wang, Priya L Donti, Bryan Wilder, and Zico Kolter. SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning, 2019.
Jingyi Xu, Zilu Zhang. Tal Friedman, Yitao Liang, and Guy van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In International Conference on Muchine Learning, 2018.
Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning, 3(1):1-130, 2009.
10

---

Under review as a conference paper at ICLR 2021
APPENDIX A
DERIVATION OF p(0) FOR A MIXTURE OF GAUSSIANS
For a general mixture distribution:
0% ply=k(æ) o( log d0
p( WTTT
ENEeR P( ke )TK
(1 (1) 0 ( logp(zly-
d ) yTa de
log k'))
dx
k2 En p(o
kttk
which, in our particular case, become:
0. log #1 no + M1 "2 x 2 (()
d0g
dr 0% 0 (2)).
Rearranging the former gives x in terms of Substituting into p(0) J p(() gives:
p(0) Tk
exp(a(log )² + (bp log 01 + (-by- 1) log Oo ck
p(0ly=k)
with coefficients:
o² 3(1iPP)3 bk (en -oo o²
log =,), Ck
(11 3M
2
APPENDIX B RECONCILING EQUATIONS (1) AND [GEN.]
p(X', Y X
p(U, p(U)))((X (o)("" () m) Σ "XX"""",))0(X""
(I)
p(T)p(U> B P (
p(X" (I)
Y
((T) 1p(
p X Y vB )p("" y
p(T))(XX11b) D 1') Σ
D )p( Y"IT)
[Gen.]
Explanation: Each term Y parameterises a distribution of the form p( X Y x)) Those distribu- tions conditional the labels Y. hence attach that lahel to the respective parameter identify the correspondence. Such parameters are referred to collectively as 3 in line 1. Line 2 separates them and identifies where each occurs elsewhere. Since labels Y and their associated parameters Ri go hand in hand, they are probabilistically interchangeable: we could think of drawing each label y from pool of le labels (and the parameter v comes with it). or draw parameter e!"" from pool of k parameters. This explains the last 2 lines. For clarity, note that in p( " (Y")), Y can be considered redundant, given parameter of the distribution, identity of the label that distribution is irrelevant.
APPENDIX C JUSTIFICATION FOR CONSIDERING ONLY THE SUPPORT OF p(0)
In section 5, we focus on the support of p(0) defined by the set of valid binary vectors V, ignoring the corresponding class probabilities p(y) = Ty E Ty.. The discriminative SSL methods analysed (see Eqs 2, 3, 4) ignore class weights also. Practical reasons for doing are (i) that they may not be known, and (ii) that unless attributes are independent (p(y) = II p(ys)) )), class probabilities do not factorise across dimensions equivalently to the support (Eq. 10). We briefly consider the validity and implications considering only the support p(0).
Validity: Regardless of whether capturing all aspects of p(0) is preferable, considering only its support valid since it is equivalent to assuming a uniform distribution over the support. The resulting approximation p00) might be considered "partially-uminforraatvv"""" prior.
11

---

Under review as a conference paper at ICLR 2021
Implications: Considering only the support of p(0) may be sufficient for the purposes of DSSL, since where p(0) provides prior over unlabelled predictions, is not used alone. we had the full (discrete) p(0) and it alone were used to predict labels for unlabelled data, maximum likelihood approach would simply assign the most common label to all unlabelled data points. However, p(0) used in conjunction with supervised model that learns approximates f()) 8" to gives predictions 6" taking class probabilities into account. To the extent the model generalises, its predictions on unlabelled data should correlate with (i.e. be close to) their true values 0", Therefore function gg (f), that is continuous relaxation of the support of p(0), helps by guiding predictions 6 to nearby valid values of f, which should to some extent reflect the correct labels. Intuitively, the prior class weights may be useful for data samples where the model is highly uncertain, where the best option may again to choose the most popular class, For well balanced classes, ignoring "y should have little impact.
12


Reviews:
[{"review_id": "tY6jOWxVsgx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a discriminative model for semi-supervised learning for which several existing methods are special cases. In their model, for each data value, there is a distribution from which the label is sampled. Although this distribution is unknown, in their framework the sampling distribution's parameters are approximately produced by a discriminative model such as a neural network trained on the labeled data.\n\nWhen the labels are vectors of features, the prior distribution of this parameter distribution can be defined so as to enforce logical constraints about which combinations of features are valid. This is a neat direction and in keeping with recent trends to integrate statistical and logical reasoning, although the paper would be strengthened if the authors gave concrete examples of a real-world dataset for which their innovation would be helpful. Likewise for their overall model, it's nice to help unify previous work, but additional discussion of impact or potential applications beyond what's been done would strengthen the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jsJLJPLJZA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims at proposing a theoretical rationale for discriminative semi-supervised learning that is comparable with that of generative models. Moreover,\nthe paper aims at theoretically justifying a family of neuro-symbolic SSL approaches.\nFor the first task, the paper states that the proposal justifies entropy minimisation (Grandvalet &\nBengio, 2005), mutual exclusivity (Sajjadi et al., 2016a; Xu et al., 2018) and pseudo-labelling (Lee,\n2013).\nFor the second task, the paper states that the proposal justifies  (Serafini & Garcez, 2016; van Krieken et al., 2019, Marra et al., 2019, Xu et al., 2018).\n\nWith respect to the first task, while I agree the the paper provides a justification for mutual exclusivity (Sajjadi et al., 2016a) \nand pseudo-labelling (Lee, 2013), I have doubts on entropy minimisation (Grandvalet & Bengio, 2005):\nhow do you model an entropy with terms of the form $-\\sum\\log p(\\theta|\\alpha)$?\nThe authors also say that previous discriminative SSL lack theoretical justification but (Xu et al., 2018) proves that their loss is a consequence of a \nnumber of assumptions, thus theoretically justifying it.\n\nThe paper misunderstood the scope of neuro-symbolic integration (NeSy): it is not the integration of statistics and logic but the integration of \nneural networks with (possibly probabilistic) logic. The integration of statistics and logic is well established in the field of\nStatistical Relational Learning.  For example, (De Raedt et al., 2007) is not NeSy while (Manhaeve et al., 2018) is.\nAs such, the paper does not provide \"a theoretically principled understanding of integrating ‘connectionist’ and ‘symbolic’ methods.\" in general, only\nfor some specific methods (not for example for (Manhaeve et al., 2018)). In fact, the justification is only for NeSy that are based on fuzzy logic, \nwhich is much simpler to manipulate than probabilistic logic as in (Manhaeve et al., 2018). Fuzzy logic is not a probabilistic logic because the\nfirst is truth functional (the value of A and B depends only on the values of A and B) while the latter is not (A and B depends on whether A and B are\nindependent, mutually exclusive,...). Moreover, (Xu et al., 2018) is more than simply impositive mutual exclusivity as it  also allow a form of probabilistic reasoning by \nrepresenting the loss formula as an arithmetic circuit obtained with automated reasoning. So overall I think that the paper claims more than it effectively provides.\n\nThe presentation of the paper makes following the exposition unnecessarily difficult. \nFirst, the notation is confusing, with a nonuniform representation\nof vectors, which are sometimes bold and sometimes not ($\\mathbb{z}$ and $\\theta$), while sometimes also scalars are bold ($\\mathbb{z}_k$).\nMoreover, random variables and their values are also not typographically distinct.\nAll integrals have the integrating variables as subscripts of the integral symbol instead of as $dxdy$...\nIn Figure 1 the left and center subfigures are equal, I guess in the left one x and y should be exchanged.\nFigure 1 left should represent the model of eq 1 but $\\psi$ and $\\pi$ are absent from the figure.\n\nIn eq. 6, shouldn't the integrating variable be $\\tilde{\\theta}^X$?\nIt is not clear how you derive the formulas [Gen.] from eq 1: since Figure 1 does not show parameters\n$\\psi$ and $\\pi$, it is difficult to judge the conditional independences among variables.\nThe symbol $\\Delta^{2^K}$ is confusing since it can also be interpreted as the simplex over a set with $2^K$ elements. Better\n$(\\Delta^2)^K$.\nYou say that Figure 3 represents a mix of two univariate Gaussians but the variance of the two Gaussians do not seem equal: if only\nthe mean changes, then the Gaussians should have the same height.\nThe paper mentions Table 4 which is absent from the paper.\n\nMinor comments:\n\"proportionality constant obsolete\"->\"proportionality constant irrelevant\"\nA the bottom of page 6, last formula: the subscript of the sum should be $\\mathbb{y}$\nPage 7 \"Note that each $\\theta\\in\\{0,1\\}^K\\subseteq \\Delta^{2^K}$ (from a continuous space): $\\theta$ does not seem from a continuous space.\nBibliography: reference Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. IEEE\nTransactions on Neural Networks, 20(3):542–542, 2009. is a book review, you want to cite the book, not the review\nIn Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. A semantic loss function for\ndeep learning with symbolic knowledge. In International Conference on Machine Learning, 2018.\nthe last author is Guy van den Broeck\n\n\n---After reading the other reviews and the authors' comments, I sill think that the paper promises more than it delivers, even if the paper was extensively rewritten as a consequence of many problems in the original version, so I will keep my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bjrSuO8tCpn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors discuss several ideas aimed at improved semi-supervised learning by adopting an appropriate \"plate model\" with probabilistic content, and then examining various techniques and variants. The theme is relevant but the whole effort seems a bit preliminary: there are many keywords and many discussed techniques, but the whole picture is not clear in terms of concrete contributions (and the provided testing does not clarify whether gains are realized). \n\nThe paper contains a somewhat long introduction that in a sense includes Sections 1 and 2, then quickly goes through the proposed Expression (6), and derives consequences that are not always clearly articulated; for instance what is the point of Table 1? Also the connection with neuro-symbolic learning is interesting but it feels a bit too much; why exactly is it needed in this framework? Or is it just an optional add-on? (Besides, for the proposed approach to work I believe more testing is needed.)\n\nA problem: as far as I can tell, Figure 1 left and center are identical. What is the difference?\n\nA problem: is the citation (Van Engelen and Hoos 2020) indeed in the references? I could not find it.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ri6de9pFn9a", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThis paper proposes a probabilistic model to describe semi-supervised/unsupervised learning, which is further applied to model neuro-symbolic learning. Comparing to traditional unsupervised/semi-supervised learning formulations, the proposed model imposes a prior on the label distribution instead of input features. When applying this formulation to neuro-symbolic learning, the symbolic part can be regarded as a prior on label space to constrain the learning process. Finally, the authors propose three methods to calculate the loss of violating the symbolic prior constraints on label space.\n  \n#### Pros\n\n+ This paper is motivated by a very important problem. Combining statistical learning and symbolic reasoning becomes a trend recently, many algorithms and models have been proposed. However, there lacks a unified theoretical framework to understand this combination fundamentally.\n+ Modelling neuro-symbolic learning as a traditional semi-supervised statistical learning problem is an interesting idea. The resulted model integrates symbolic constraints as a prior on label distribution, which is very natural.\n\n#### Cons\n- The writing of this paper could be improved. For example, the notation in this paper is sometimes confusing. The meaning of parameter $\\theta^x$ is not clearly explained: In section 2, it is explained as a parameter of the conditional distribution $p(\\mathrm{y}|x)$, so I took it as the parameters of a classifier $y=h_\\theta^x(x)$. But in Section 4 and 5, the authors state that $\\mathbf{y}$ and $\\theta$ has a 1-1 corresponds in the space of $\\{0,1\\}^K$, so it seems that $\\theta^x$ should be interpreted as pseudo-labels of examples. In section 3, the authors introduce another variable $\\tilde{\\theta^x}$ as estimations of $\\theta^x$, which makes me more confused. Although this is a theoretical paper, I think the authors should improve the notations and provide some simple examples for helping readers understand this framework.\n- The idea of using logic rules as constraints (or other non-differentiable constraints) on label space is not a novel idea in statistical learning (Zhang and Zhou, 2013). In section 3, the first category of \"deterministic, distinct classes\" is multi-class learning; the \"deterministic non-exclusive features\" is multi-label learning. In both contexts, there are many works talking about constraining the label space with prior knowledge, for example, Cesa-Bianchi et al. (2006) learns multi-label classifier with hierarchical labels; Tsochantaridis et al. (2004) proposes SVM-struct deal with structural output with support vector machines; Mei et al. (2014) introduces logic rules as label constraints within a Bayesian learning framework. The authors have surveyed a wide range of related works in the area of semi-supervised learning and neuro-symbolic learning. However, I think the works I have mentioned above is highly related to this work; the authors should discuss the relationship between this work and those early works in statistical learning.\n- Last but not least, this paper lacks experiments to support the authors' claims. Meanwhile, although this paper presents a general framework, it provides little insight into this area. The authors propose three methods to compute the non-differentiable $p(\\theta)$, while (B) ignores the relationship between rules (e.g., recursive theories) and treat rules independently; (C) is equivalent to multi-label learning setting or the ECOC method in multi-class learning.\n  \n#### Recommendation\n\nOverall, I appreciate that this paper is trying to solve a crucial problem in neuro-symbolic learning, and the idea of viewing this problem as semi-supervised statistical learning is natural and reasonable. However, this paper should include more discussion about related works in statistical learning. Furthermore, it would be better if the authors can include some examples and experiments to demonstrate the idea of the proposed model.\n\n#### Additional questions and comments\n\n- The first to sub-figures in Fig. 1 are the same; the generative model should have arrows from $y$ and $\\theta$ to $x$.\n- In section 3, first paragraph, line 6: \"... outputting $\\theta^x$, an estimate of $\\theta^x$\". The first $\\theta^x$ should be $\\tilde{\\theta^x}$.\n- $\\alpha$ first appears in Eq. 6 and Fig. 1. However, there is no explanation about what is it.\n  \n\n#### References\n- Zhang, Min-Ling, and Zhi-Hua Zhou. \"A review on multi-label learning algorithms.\" IEEE transactions on knowledge and data engineering 26.8 (2013): 1819-1837.\n- Cesa-Bianchi, Nicolò, Claudio Gentile, and Luca Zaniboni. \"Incremental algorithms for hierarchical classification.\" Journal of Machine Learning Research 7.Jan (2006): 31-54.\n- Tsochantaridis, Ioannis, et al. \"Support vector machine learning for interdependent and structured output spaces.\" Proceedings of the twenty-first international conference on Machine learning. 2004.\n- Mei, Shike, Jun Zhu, and Jerry Zhu. \"Robust regbayes: Selectively incorporating first-order logic domain knowledge into bayesian models.\" International Conference on Machine Learning. 2014.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that this is a very promising paper, with an excellent overview of existing techniques for semi-supervised and neuro-symbolic learning. However, reviewers also agree that the paper is not ready. With one more revision for clarity, some limited empirical validation and illustration of the theory, and focus on the essential message, this could become a seminal paper for our understanding of semi-supervised learning. Luckily the reviews provided ample feedback, and the authors should be able to submit a very competitive paper next time around.
Decision: Reject