A Probabilistic Model for Discriminative and Neuro-Symbolic Semi-Supervised Learning

1 INTRODUCTION . In semi-supervised learning ( SSL ) , a mapping is learned that predicts labels y for data points x from a dataset of labelled pairs ( xl , yl ) and unlabelled xu . SSL is of practical importance since unlabelled data are often cheaper to acquire and/or more abundant than labelled data . For unlabelled data to help predict labels , the distribution of x must contain information relevant to the prediction ( Chapelle et al. , 2006 ; Zhu & Goldberg , 2009 ) . State-of-the-art SSL algorithms ( e.g . Berthelot et al. , 2019b ; a ) combine underlying methods , including some that leverage properties of the distribution p ( x ) , and others that rely on the label distribution p ( y|x ) . The latter include entropy minimisation ( Grandvalet & Bengio , 2005 ) , mutual exclusivity ( Sajjadi et al. , 2016a ; Xu et al. , 2018 ) and pseudo-labelling ( Lee , 2013 ) , which add functions of unlabelled data predictions to a typical discriminative supervised loss function . Whilst these methods each have their own rationale , we propose a formal probabilistic model that unifies them as a family of discriminative semi-supervised learning ( DSSL ) methods . Neuro-symbolic learning ( NSL ) is a broad field that looks to combine logical reasoning and statistical machine learning , e.g . neural networks . Approaches often introduce neural networks into a logical framework ( Manhaeve et al. , 2018 ) , or logic into statistical learning models ( Rocktäschel et al. , 2015 ) . Several works combine NSL with semi-supervised learning ( Xu et al. , 2018 ; van Krieken et al. , 2019 ) but lack rigorous justification . We show that our probabilistic model for discriminative SSL extends to the case where label components obey logical rules , theoretically justifying neuro-symbolic SSL approaches that augment a supervised loss function with a function based on logical constraints . Central to this work are ground truth parameters { θx } x∈X of the distributions p ( y|x ) , as predicted by models such as neural networks . For example , θxmay be a multinomial parameter vector specifying the distribution over all labels associated with a given x . Since each data point x has a specific label distribution defined by θx , sampling from p ( x ) induces an implicit distribution over parameters , p ( θ ) . If known , the distribution p ( θ ) serves as a prior over all model predictions , θ̃x : for labelled samples it may provide little additional information , but for unlabelled data may allow predictions to be evaluated and the model improved . As such , p ( θ ) provides a potential basis for semi-supervised learning . We show that , in practice , p ( θ ) can avoid much of the complexity of p ( x ) and have a concise analytical form known a priori . In principle , p ( θ ) can also be estimated from the parameters learned for labelled data ( fitting the intuition that predictions for unlabelled data should be consistent with those of labelled data ) . We refer to SSL methods that rely on p ( θ ) as discriminative and formalise them with a hierarchical probabilistic model , analogous to that for generative approaches . Recent results ( Berthelot et al. , 2019b ; a ) demonstrate that discriminative SSL is orthogonal and complementary to methods that rely on p ( x ) , such as data augmentation and consistency regularisation ( Sajjadi et al. , 2016b ; Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ; Miyato et al. , 2018 ) . We consider the explicit form of p ( θ ) in classification with mutually exclusive classes , i.e . where each x only ever pairs with a single y and y|x is deterministic . By comparison of their loss functions , the SSL methods mentioned ( entropy minimisation , mutual exclusivity and pseudo-labelling ) can be seen to impose continuous relaxations of the resulting prior p ( θ ) and are thus unified under our probabilistic model for discriminative SSL . We then consider classification with binary vector labels , e.g . representing concurrent image features or allowed chess board configurations , where only certain labels/attribute combinations may be valid , e.g . according to rules of the game or the laws of nature . Analysing the structure of p ( θ ) here , again assuming y|x is deterministic , we show that logical rules between attributes define its support . As such , SSL approaches that use fuzzy logic ( or similar ) to add logical rules into the loss function ( e.g . Xu et al. , 2018 ; van Krieken et al. , 2019 ) can be seen as approximating a continuous relaxation of p ( θ ) and so also fall under our probabilistic model for discriminative SSL . Our key contributions are : • to provide a probabilistic model for discriminative semi-supervised learning , comparable to that for classical generative methods , contributing to current theoretical understanding of SSL ; • to consider the analytical form of the distribution over parameters p ( θ ) , by which we explain several SSL methods , including entropy minimisation as used in state-of-art SSL models ; and • to show that our probabilistic model also unifies neuro-symbolic SSL in which logical rules over attributes are incorporated ( by fuzzy logic or similar ) to regularise the loss function , providing firm theoretical justification for this means of integrating ‘ connectionist ’ and ‘ symbolic ’ methods . 2 BACKGROUND AND RELATED WORK . Notation : xli∈X l , yli∈Y l are labelled data pairs , i∈ { 1 ... Nl } ; xuj ∈Xu , yuj ∈Y u are unlabelled data samples and their ( unknown ) labels , j ∈ { 1 ... Nu } ; X , Y are domains of x and y ; x , y are random variables of which x , y are realisations . θx parameterises the distribution p ( y|x ) , and is a realisation of a random variable θ . To clarify : for each x , an associated parameter θx defines a distribution over associated label ( s ) y|x ; and p ( θ ) is a distribution over all such parameters . 2.1 SEMI-SUPERVISED LEARNING . Semi-supervised learning is a well established field , described by a number of surveys and taxonomies ( Seeger , 2006 ; Zhu & Goldberg , 2009 ; Chapelle et al. , 2006 ; van Engelen & Hoos , 2020 ) . SSL methods have been categorised by how they adapt supervised learning algorithms ( van Engelen & Hoos , 2020 ) ; or their assumptions ( Chapelle et al. , 2006 ) , e.g . that data of each class form a cluster/manifold , or that data of different classes are separated by low density regions . It has been proposed that all such assumptions are variations of clustering ( van Engelen & Hoos , 2020 ) . Whilst ‘ clustering ’ itself is not well defined ( Estivill-Castro , 2002 ) , from a probabilistic perspective this suggests that SSL methods assume p ( x ) to be a mixture of conditional distributions that are distinguishable by some property , e.g . connected dense regions . This satisfies the condition that for unlabelled x to help in learning to predict y from x , the distribution of x must contain information relevant to the prediction ( Chapelle et al. , 2006 ; Zhu & Goldberg , 2009 ) . In this work , we distinguish SSL methods by whether they rely on direct properties of p ( x ) , or on properties that manifest in p ( θ ) , the distribution over parameters of p ( y|x ; θx ) , for x∼ p ( x ) . State-of-art models ( Berthelot et al. , 2019b ; a ) combine methods of both types . A canonical SSL method that relies on explicit assumptions of p ( x ) is the classical generative model : p ( X l , Y l , Xu ) = ∫ ψ , π p ( ψ , π ) p ( X l|Y l , ψ ) p ( Y l|π ) ∑ Y u∈YNu p ( Xu|Y u , ψ ) p ( Y u|π ) ︸ ︷︷ ︸ p ( Xu|ψ , π ) ( 1 ) Parameters ψ , π of p ( x|y ) and p ( y ) are learned from labelled and unlabelled data , e.g . by the EM algorithm , and predictions p ( y|x ) =p ( x|y ) p ( y ) /p ( x ) follow by Bayes ’ rule . Figure 1 ( left ) shows the corresponding graphical model . Whilst generative SSL has an appealing probabilistic rationale , it is rarely used in practice , similarly to its counterpart for fully supervised learning , in large part because p ( x ) is often complex yet must be accurately described ( Grandvalet & Bengio , 2005 ; Zhu & Goldberg , 2009 ; Lawrence & Jordan , 2006 ) . However , properties of p ( x ) underpin data augmentation and consistency regularisation ( Sajjadi et al. , 2016b ; Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ; Miyato et al. , 2018 ) , in which true x samples are adjusted , using implicit domain knowledge of p ( x|y ) , to generate artificial samples of the same class , whether or not that class is known . Other SSL methods consider p ( x ) in terms of components p ( x|z ) , where z is a latent representation useful for predicting y ( Kingma et al. , 2014 ; Rasmus et al. , 2015 ) . We focus on a family of SSL methods that add a function of the unlabelled data predictions to a discriminative supervised loss function , e.g . : • Entropy minimisation ( Grandvalet & Bengio , 2005 ) assumes classes are “ well separated ” . As a proxy for class overlap , the entropy of unlabelled data predictions is added to a discriminative supervised loss function ` sup : ` MinEnt ( θ ) = − ∑ i ∑ k yli , k log θ xli k︸ ︷︷ ︸ ` sup − ∑ j ∑ k θ xuj k log θ xuj k ( 2 ) • Mutual exclusivity ( Sajjadi et al. , 2016a ; Xu et al. , 2018 ) assumes no class overlap , i.e . correct predictions form ‘ one-hot ’ vectors . Viewed as vectors of logical variables z , such outputs exclusively satisfy the logical formula ∨ k ( zk ∧ j 6=k¬zj ) . A function based on the formula applies to unlabelled predictions : ` MutExc ( θ ) = ` sup − ∑ j log ∑ k θ xuj k ∏ k′ 6=k ( 1− θx u j k′ ) ( 3 ) • Pseudo-labelling ( Lee , 2013 ) assumes that predicted classes kj ( t ) =arg maxk θ xuj k for unlabelled data xuj at iteration t , are correct ( at the time ) and treated as labelled data : ` Pseudo ( θ , t ) = ` sup − ∑ j log ∑ k 1k=kj ( t ) θ xuj k ( 4 ) These methods , though intuitive , lack a probabilistic rationale comparable to that of generative models ( Eq . 1 ) . Summing over all labels for unlabelled samples is of little use ( Lawrence & Jordan , 2006 ) : p ( Y l|X l , Xu ) = ∫ θ p ( θ ) p ( Y l|X l , θ ) ∑ Y up ( Y u|Xu , θ ) ︸ ︷︷ ︸ =1 = ∫ θ p ( θ ) p ( Y l|X l , θ ) . ( 5 ) Indeed , under the associated graphical model ( Fig . 1 ( centre ) ) , parameters θ of p ( Y l|X l , θ ) are provably independent of Xu ( Seeger , 2006 ; Chapelle et al. , 2006 ) . Previous approaches to breaking this independence include introducing additional variables to Gaussian Processes ( Lawrence & Jordan , 2006 ) , or an assumption that parameters of p ( y|x ) are dependent on those of p ( x ) ( Seeger , 2006 ) . Taking further the ( general ) assumption of ( Seeger , 2006 ) , we provide a probabilistic model for discriminative SSL ( DSSL ) , analogous and complementary to that for generative SSL ( Eq . 1 ) .

Summary: This paper proposes a probabilistic model to describe semi-supervised/unsupervised learning, which is further applied to model neuro-symbolic learning. Comparing to traditional unsupervised/semi-supervised learning formulations, the proposed model imposes a prior on the label distribution instead of input features. When applying this formulation to neuro-symbolic learning, the symbolic part can be regarded as a prior on label space to constrain the learning process. Finally, the authors propose three methods to calculate the loss of violating the symbolic prior constraints on label space.