Gaussian Conditional Random Fields for Classification

1 INTRODUCTION . Increased quantity and variety of sources of data with correlated outputs , so called structured data , created an opportunity for exploiting additional information between dependent outputs to achieve better prediction performance . One of the most successful probabilistic models for structured output classification problems are conditional random fields ( CRF ) ( Sutton & McCallum , 2006 ) . The main advantages of CRFs lie in their discriminatory nature , resulting in the relaxation of independence assumptions and the label bias problem that are present in many graphical models . Aside of many advantages , CRFs also have many drawbacks mostly resulting in high computational cost or intractability of inference and learning . A wide range of different approaches of tackling these problems has been proposed , and they motivate our work , too . One of the popular methods for structured regression based on CRFs – Gausian conditional random fields ( GCRF ) – has the form of multivariate Gaussian distribution ( Radosavljevic et al. , 2010 ) . The main assumption of the model is that the relations between outputs are presented in quadratic form . It has convex loss function and , consequently , efficient inference and learning , and expensive sampling methods are not used . In this paper , a new model of Gaussian conditional random fields for binary classification is proposed ( GCRFBC ) . GCRFBC builds upon regression GCRF model which is used to define latent variables over which output dependencies are defined . The model assumes that discrete outputs yi are conditionally independent conditioned on continuous latent variables zi which follow a distribution modeled by a GCRF . That way , relations between discrete outputs are not expressed directly . Two different inference and learning approaches are proposed in this paper . The first one is based on evaluating empirical Bayes by marginalizing latent variables ( GCRFBCb ) , whereas MAP value of latent variables is the basis for learning and inference in the second model ( GCRFBCnb ) . In order to derive GCRFBCb model and its learning procedure the variational approximation of Bayesian logistic regression ( Jaakkola & Jordan , 2000 ) is generalized . Compared to CRFs and structured SVM classifiers , the GCRFBC models have some appealing properties : • The model is applicable to classification problems with undirected graphs , intractable for standard classification CRFs . Thanks to the GCRF latent structure , the model becomes tractable , efficient and open to improvements previously applied to GCRF regression models . • Defining correlations directly between discrete outputs may introduce unnecessary noise to the model ( Tan et al. , 2010 ) . This problem can be solved by defining structured relations on a latent continuous variable space . • In case that unstructured predictors are unreliable , which is signaled by their large variance ( diagonal elements in the covariance matrix ) , it is simple to marginalize over latent variable space and obtain better results . GCRFBC model is relying on the assumption that the underlying distribution of latent variables is multivariate normal distribution , due to that in the case when this distribution can not be fitted well to the data ( e.g . when the distribution of latent variables is multimodal ) the model will not perform as well as it is expected . The proposed models are experimentally tested on both synthetic and real-world datasets in terms of predictive performance and computation time . In experiments with synthetic datasets , the results clearly indicate that the the empirical Bayes approach ( GCRFBCb ) better exploits output dependence structure , more so as the variance of the latent variables increases . We also tested both approaches on real-world datasets of predicting ski lift congestion , gene function classification , classification of music according to emotion and highway congestion . Both GCRFBC models outperformed ridge logistic regression , lasso logistic regression , neural network , random forest , and structured SVM classifiers , demonstrating that the proposed models can exploit output dependencies in a real-world setting . 2 RELATED WORK . An extensive review of binary and multi-label classification with structured output is provided in Su ( 2015 ) . A number of different studies related to graph based methods for regression can be found in the literature ( Fox , 2015 ) . CRFs were successfully applied on a variety of different structured tasks ( Cotterell & Duh , 2017 ; Zhang et al. , 2015 ; Masada & Bunescu , 2017 ; Zia et al. , 2018 ) and different model adaptations can be found in literature Kim ( 2017 ) ; Maaten et al . ( 2011 ) . Recently , successful unifications of deep learning and CRFs have been proposed Chen et al . ( 2016 ) ; Kosov et al . ( 2018 ) . Moreover , implementation of deep neural networks as potential functions is presented in form of structure prediction energy networks ( SPEN ) Belanger & McCallum ( 2016 ) ; Belanger et al . ( 2017 ) . Adaptation of normalazing flows in SPEN structure is presented in Lu & Huang ( 2019 ) . An extensive review on topic of binary and multi-label classification with structured output is provided in Su ( 2015 ) . Large number of different studies related to graph based methods for regression can be found in the literature ( Fox , 2015 ) . CRFs were successfully applied on a variety of different structured tasks , such as : low-resource named entity recognition ( Cotterell & Duh , 2017 ) , image segmentation ( Zhang et al. , 2015 ) , chord recognition ( Masada & Bunescu , 2017 ) and word segmentation ( Zia et al. , 2018 ) . The mixture of CRFs capable to model data that come from multiple different sources or domains is presented in Kim ( 2017 ) . The method is related to the well known hidden-unit CRF ( HUCRF ) ( Maaten et al. , 2011 ) . The conditional likelihood and expectation minimization ( EM ) procedure for learning have been derived there . The mixtures of CRF models were implemented on several real-world applications resulting in prediction improvement . Recently , a model based on unification of deep learning and CRF was developed by Chen et al . ( 2016 ) . The deep CRF model showed better performance compared to either shallow CRFs or deep learning methods on their own . Similarly , the combination of CRFs and deep convolutional neural networks was evaluated on an example of environmental microorganisms labeling ( Kosov et al. , 2018 ) . The spatial relations among outputs were taken in consideration and experimental results have shown satisfactory results . The GCRF model was first implemented for the task of low-level computer vision ( Tappen et al. , 2007 ) . Since then , various different adaptations and approximations of GCRF were proposed ( Radosavljevic et al. , 2014 ) . The parameter space for the GCRF model is extended to facilitate joint modelling of positive and negative influences ( Glass et al. , 2016 ) . In addition , the model is extended by bias term into link weight and solved as a part of convex optimization . Semi-supervised marginalized Gaussian conditional random fields ( MGCRF ) model for dealing with missing variables was proposed by Stojanovic et al . ( 2015 ) . The benefits of the model were proved on partially observed data and showed better prediction performance than alternative semi-supervised structured models.A comprehensive review of continuous conditional random fields ( CCRF ) was provided in Radosavljevic et al . ( 2010 ) . The sparse conditional random fields obtained by l1 regularization are first proposed and evaluated by Wytock & Kolter ( 2013 ) . Additionaly , Frot et al . ( 2018 ) presented GCRF with the latent variable decomposition and derived convergence bounds for the estimator that is well behaved in high dimensional regime . An adaptation of GCRF on discrete output was briefly discussed in Radosavljevic ( 2011 ) , as a part of future work . This discussion motivates our work , but our approach is different in technical aspects . 3 METHODOLOGY . In this section we first present already known GCRF model for regression and then we propose GCRFBC model for binary classification and two approaches to inference and learning . 3.1 BACKGROUND MATERIAL . GCRF is a discriminative graph-based regression model ( Radosavljevic et al. , 2010 ) . Nodes of the graph are variables y = ( y1 , y2 , . . . , yN ) , which need to be predicted given a set of features x . The attributes x = ( x1 , x2 , . . . , xN ) interact with each node yi independently of one another , while the relations between outputs are expressed by pairwise interaction function . In order to learn parameters of the model , a training set of vectors of attributes x and real-valued response variables y are provided . The generalized form of the conditional distribution P ( y|x , α , β ) is : P ( y|x , α , β ) = 1 Z ( x , α , β ) exp − N∑ i=1 K∑ k=1 αk ( yi −Rk ( xi ) ) 2 −∑ i 6=j L∑ l=1 βlS l ij ( yi − yj ) 2 ( 1 ) First sum models relations between outputs yi and corresponding input vector xi and the second one models pairwise relations between nodes . Rk ( xi ) represents an unstructured predictor of yi for each node in the graph and Slij is value that expresses similarity between nodes i and j in graph l. Unstructured predictor can be any regression model that gives prediction of output yi for given attributes xi . K is the total number of unstructured predictors . L is the total number of graphs ( similarity functions ) . Graphs can express any kind of binary relations between nodes e.g. , spatial and temporal correlations between outputs . Z is a partition function and vectors α and β are learnable parameters . One of the main advantages of GCRF is the ability to express different relations between outputs by variety of graphs and ability to learn which graphs are significant for prediction . The quadratic form of interaction and association potential enables conditional distribution P ( y|x , α , β ) to be expressed as multivariate Gaussian distribution ( Radosavljevic et al. , 2010 ) : P ( y|x , α , β ) = 1 ( 2π ) N 2 |Σ| 12 exp ( −1 2 ( y − µ ) TΣ−1 ( y − µ ) ) ( 2 ) Precision matrix Σ−1 = 2Q and distribution mean µ = Σb are defined as , respectively : Q = { ∑K k=1 αk + ∑N h=1 ∑L l=1 βlS l ih , if i = j − ∑L l=1 βlS l ij , if i 6= j ( 3 ) bi = 2 K∑ k=1 αkRk ( xi ) ( 4 ) Due to concavity of multivariate Gaussian distribution , the inference task argmax y P ( y|x , α , β ) is straightforward . The maximum posterior estimate of y is the distribution expectation µ . The objective of the learning task is to optimize parameters α and β by maximizing conditional log likelihood argmax α , β ∑ y logP ( y|x , α , β ) . One way to ensure positive definiteness of the covariance matrix of GCRF is to require diagonal dominance ( Strang et al. , 1993 ) . This can be ensured by imposing constraints that all elements of α and β be greater than 0 ( Radosavljevic et al. , 2010 ) .

Summary: The work involves modifiying gaussian conditional random fields to work for classification problems instead of regression problems. The main idea is to apply a bernoulli distribution on top of the regression values to convert them to work with binary classification problems. Two variations are discussed along with the inference and learning methodology. The inference can be done using numerical approximation and learning using variational methods and is still untracktable. Comparisons with other modeling strategies is done using experiments.