Gaussian Conditional Random Fields for Classification
Authors: Andrija Petrovic
Mladen Nikolic
Milos Jovanovic
Boris Delibasic
Venue:  (2020)
Keywords: Structured classification
Gaussian conditional random fields
Empirical Bayes
Local variational approximation
discriminative graph-based model

Abstract: In this paper, a Gaussian conditional random field model for structured binary classification (GCRFBC) is proposed. The model is applicable to classification problems with undirected graphs, intractable for standard classification CRFs. The model representation of GCRFBC is extended by latent variables which yield some appealing properties. Thanks to the GCRF latent structure, the model becomes tractable, efficient, and open to improvements previously applied to GCRF regression. Two different forms of the algorithm are presented: GCRFBCb (GCRGBC - Bayesian) and GCRFBCnb (GCRFBC - non-Bayesian). The extended method of local variational approximation of sigmoid function is used for solving empirical Bayes in GCRFBCb variant, whereas MAP value of latent variables is the basis for learning and inference in the GCRFBCnb variant. The inference in GCRFBCb is solved by Newton-Cotes formulas for one-dimensional integration. Both models are evaluated on synthetic data and real-world data. It was shown that both models achieve better prediction performance than relevant baselines. Advantages and disadvantages of the proposed models are discussed.

Under review as a conference paper at ICLR 2020
GAUSSIAN CONDITIONAL RANDOM FIELDS FOR CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, Gaussian conditional random field model for structured binary classification (GCRFBC) is proposed. The model is applicable to classification problems with undirected graphs. intractable for standard classification CRFs. The model representation of GCRFBC is extended by latent variables which yield some appealing properties. Thanks to the GCRF latent structure, the model becomes tractable, efficient, and open to improvements previously applied to GCRF regression. Two different forms of the algorithm are presented: GCRF- BCb (GCRGBC Bayesian) and GCRFBCnb (GCRFBC- non-Bayesian) The extended method of local variational approximation of sigmoid function is used for solving empirical Bayes in GCRFBCb variant, whereas MAP value of latent variables is the basis for learning and inference in the GCRFBCnb variant. The inference in GCRFBCb is solved by Newton-Cotes formulas for mie-dimeniional integration. Both models are evaluated on synthetic data and real-world data. It was shown that both models achieve better prediction performance than relevant baselines. Advantages and disadvantages of the proposed models are discussed.
INTRODUCTION
Increased quantity and variety of sources of data with correlated outputs, called structured data, created an opportunity for exploiting additional information between dependent outputs achieve better prediction performance. One of the most successful probabilistic models for structured out- put classification problems are conditional random fields (CRF) (Sutton & McCallum (006). The main advantages of CRFs lie in their discriminatory nature, resulting in the relaxation of indepen- dence assumptions and the label bias problem that are present in many graphical models. Aside of many advantages, CRFs also have many drawbacks mostly resulting in high computational cost or intractability of inference and learning. A wide range of different approaches of tackling these problems has been proposed, and they motivate our work, too.
One of the popular methods for structured regression based on CRFs Gausian conditional random fields (GCRF) has the form of multivariate Gaussian distribution (Radosavijevic al., 2010). The main assumption of the model is that the relations between outputs are presented in quadratic form. has convex loss function and, consequently, efficient inference and learning, and expensive sampling methods are not used.
In this paper, new model of Gaussian conditional random fields for binary classification is pro- posed (GCRFBC). GCRFBC builds upon regression GCRF model which is used to define latent variables over which output dependencies are defined. The model assumes that discrete outputs y: are conditionally independent conditioned on continuous latent variables which follow a distribu- tion modeled by GCRF. That way, relations between discrete outputs are not expressed directly. Two different inference and learning approaches are proposed in this paper. The first one is based on evaluating empirical Bayes by marginalizing latent variables (GCRFBCb), whereas MAP value of latent variables is the basis for learning and inference in the second model (GCRFBCnb). In order to derive GCRFBCb model and its learning procedure the variational approximation of Bayesian logistic regression (Jaakkola & Jordan 2000 is generalized.
Compared to CRFs and structured SVM classifiers, the GCRFBC models have some appealing properties:

---

Under review as a conference paper at ICLR 2020
The model is applicable classification problems with undirected graphs, intractable for standard classification CRFs. Thanks to the GCRF latent structure, the model becomes tractable, efficient and open improvements previously applied to GCRF regression mod- els.
Defining correlations directly between discrete outputs may introduce unnecessary noise to the model (Tan et al. 2010). This problem can be solved by defining structured relations latent continuous variable space.
In case that unstructured predictors are unreliable, which signaled by their large variance (diagonal elements in the covariance matrix), it is simple to marginalize over latent variable space and obtain better results.
GCRFBC model is relying on the assumption that the underlying distribution of latent variables is multivariate normal distribution, due to that in the case when this distribution cannot be fitted well to the data (e.g. when the distribution of latent variables is multimodal) the model will not perform as well it is expected. The proposed models are experimentally tested on both synthetic and real-world datasets in terms of predictive performance and computation time. In experiments with synthetic datasets, the results clearly indicate that the the empirical Bayes approach (GCRFBCb) better exploits output dependence structure, more so as the variance of the latent variables increases. We also tested both approaches on real-world datasets of predicting ski lift congestion, gene function classification, classification of music according to emotion and highway congestion. Both GCRFBC models outperformed ridge logistic regression, lasso logistic regression, neural network, random forest, and structured SVM classifiers, demonstrating that the proposed models can exploit output dependencies in a real-world setting.
RELATED WORK
An extensive review of binary and multi-label classification with structured output is provided in Su (2015). A number of different studies related to graph based methods for regression can be found in the literature (Fox 2015). CRFs were successfully applied on a variety of different structured tasks
(Cotterell & Duh 2017 Zhang et al. 2015 Masada & Bunescu 2017 Zia et al. 2018) and different model adaptations can be found in literature Kim (2017): Maaten et al. (2011). Recently, successful
unifications of deep learning and CRFs have been proposed Chen al. (2016): Kosov et al. (2018).
Moreover, implementation of deep neural networks as potential functions is presented in form of
structure prediction energy networks (SPEN) (Belaner & McCallum (2016); Belanger et Bellll
(017
Adaptation of normalazing flows in SPEN structure is presented in Lu & Huang (2019).
An extensive review on topic of binary and multi-label classification with structured output is pro- vided in Su (2015). Large number of different studies related to graph based methods for regression can be found in the literature (Fox) 2013). CRFs were successfully applied on a variety of different structured tasks, such
low low-resource named entity recognition (Cotterell & Duh (017). image segmentation (Zhang et al.] 2015),, chord recognition (Masada & Bunescul 2017) and word seg- mentation (Zia et al. 2018). The mixture of CRFs capable to model data that come from multiple different sources or domains is presented in Kim (2017). The method is related to the well known hidden-unit CRF (HUCRF) (Maaten al. (2011). The conditional likelihood and expectation min- imization (EM) procedure for learning have been derived there. The mixtures of CRF models were implemented on several real-world applications resulting in prediction improvement. Recently, model based on unification of deep learning and CRF was developed by CCen et al. (2016). The deep CRF model showed better performance compared to either shallow CRFs or deep learning methods on their own. Similarly, the combination of CRFs and deep convolutional neural networks was evaluated on an example of environmental microorganisms labeling (Kosov et al. 2018). The spatial relations among outputs were taken in consideration and experimental results have shown satisfactory results.
The GCRF model was first implemented for the task of low-level computer vision (Tappen et al.) 2007]. Since then, various different adaptations and approximations of GCRF were proposed (Ra- dosavljevic et al.) 2014). The parameter space for the GCRF model is extended to facilitate joint modelling of positive and negative influences (Glass et al., 2016). addition, the model is ex- tended by bias term into link weight and solved part of convex optimization. Semi-supervised
2

---

Under review as a conference paper at ICLR 2020
marginalized Gaussian conditional random fields (MGCRF) model for dealing with missing vari- ables was proposed by Stojanovic et al. (2013). The benefits of the model were proved on partially observed data and showed better prediction performance than alternative
supervise structured models. comprehensive review of continuous conditional random fields (CCRF) was provided in Radosavljevic al. (2010). The sparse conditional random fields obtained by l regularization first proposed and evaluated by [Wytock& & Kolter (2013). Additionaly, Frot et al. (2018) presented GCRF with the latent variable decomposition and derived convergence bounds for the estimator that is well behaved in high dimensional regime. An adaptation of GCRF on discrete output was briefly
discussed in Radosavljevic (201T), as a part of future work. This discussion motivates our work, but our approach is different in technical aspects.
3
METHODOLOGY
In this section we first present already known GCRF model for regression and then we propose GCRFBC model for binary classification and two approaches to inference and learning.
3.1 BACKGROUND MATERIAL
GCRF is a discriminative graph-based regression model (Radosavljevic et al. 2010). Nodes of the graph are variables y (v1, 12.2..... WN), which need to be predicted given set of features a. The attributes æ = 111, T2..... NN) interact with each node Yi independently of one another. while the relations between outputs expressed by pairwise interaction function. In order to learn parameters of the model, a training set of vectors of attributes I and real-valued response variables y are provided. The generalized form of the conditional distribution P ar, az B) is:
P Pyla, 2 (I, exp
Σ
a& (v/ Rk (2:))²
BiS (vi -11}²
First sum models relations between outputs yi and corresponding input vector æ; and the second one models pairwise relations between nodes. R& (x;;) represents an unstructured predictor of yı for each node in the graph and St is value that expresses similarity between nodes and in graph 1. Unstructured predictor can be any regression model that gives prediction of output y; for given attributes 24. K is the total number of unstructured predictors. L is the total number of graphs (similarity functions). Graphs can express any kind of binary relations between nodes e.g., spatial and temporal correlations between outputs.
a partition function and vectors a and /3 are learn- able parameters. One of the main advantages of GCRF is the ability to express different relations between outputs by variety of graphs and ability to learn which graphs are significant for predic- tion. The quadratic form of interaction and association potential enables conditional distribution Plylæ, o. B) to be expressed as multivariate Gaussian distribution (Radosavljevic et al. 2010):
P(y xr a, B) (2m) exp (-2(0
(y
-) 14)
Precision matrix Σ-1 -QQ and distribution mean Σb are defined as, respectively:
Q = Σ on + + Σ B.St
if
(3)
Ei=1 BIS.
i
K
b, = Σ az Rx
(4)
Due to concavity of multivariate Gaussian distribution, the inference task argmaxP(yyla a., (3) is y
straightforward. The maximum posterior estimate of y is the distribution expectation μ..
The objective of the learning task is to optimize parameters and /3 by maximizing conditional log
likelihood argmax Σ, (oy x a. 3). One way to ensure positive definiteness the covariance
x, matrix of GCRF is to require diagonal dominance (Strang et al. 1993). This can be ensured by
imposing constraints that all elements of and B be greater than 0 (Radosavljevic al. 2010).
3

---

Under review as a conference paper at ICLR 2020
3.2 GCRFBC MODEL REPRESENTATION
One way of adapting GCRF to classification problem is by approximating discrete outputs by suit- ably defining continuous outputs. Namely, GCRF can provide dependence structure over contin- uous variables which can be passed through sigmoid function. That way the relationship between regression GCRF and classification GCRF similar the relationship between linear and logistic regression, but with dependent variables. Aside from allowing to define a classification variant of GCRF, this may result in additional appealing properties: (i) The model is applicable to classifi- cation problems with undirected graphs, intractable for standard classification CRFs. Thanks to the GCRF latent structure, the model becomes tractable, efficient and open to improvements previously applied GCRF regression models. (ii) Defining correlations directly between discrete outputs may introduce unnecessary noise to the model (Tan et al.] 2010). We avoid this problem by defining structured relations on a latent continuous variable space. (ii) In case that unstructured predictors are unreliable, which is signaled by their large variance (diagonal elements in the covariance matrix), it is simple marginalize over latent variable space and obtain better results.
It is assumed that are discrete binary outputs and 2; are continuous latent variables assigned to each y:- Each output y; conditionally independent of the others, given 2,.
The conditional probability distribution P(v)(zs) is defined as Bernoulli distribution:
22))22)) Ber(y. (2:)
(5)
where (-)) is sigmoid function. Due to conditional independence assumption, the joint distribution of outputs y can be expressed as:
N
P(y,, U2 U2|z) = II
(6)
Furthermore, the conditional distribution P(z (z)) is the same in the classical GCRF model and has canonical form defined by multivariate Gaussian distribution. Hence, joine,joint distribution of continuous latent variables and outputs y given and 0 (a1...., aK, .......... the GCRFBC model defined as:
.,. ,3) is is the general form of
N
Ply,2|æ,0)
N22) [[ 0)
1=1
exp (- -= 1(x ((z=-(zz0))
()) (z, 0) p(xx 0))
We consider two ways of inference and learning in GCRFBC model: (i) GCRFBCb with condi- tional probability distribution P(y) /z. 0), in which variables z are marginalized over, and (ii) GCRE- BCnb- with conditional probability distribution P (y x. 0, pz), in which variables z are substituted by their expectations.
3.3 INFERENCE IN GCRFBCE MODEL
Prediction of discrete outputs y for given features æ and parameters 0 is analytically intractable due to integration of the joint distribution P(y, z|æ, 0) with respect to latent variables. However, due to conditional independence between nodes, it possible to obtain P(y: 1|x, 0).
P(y: = 1/ 0) ((z))) 0)dz
where (=2;) models P(y.(z). As result of independence properties of the distribution, it holds p(y = 1|x) P(# = 1/2,), and it is possible to marginalize P(z|æ, x. 0) with respect to latent variables z' (Z1...., 2-11 24.1,..,,
P(y = 1|x, 0) = 1( o(z)) (L P(z1,2110, O)dx'))))) dz,
(((z)
(9)
4

---

Under review as a conference paper at ICLR 2020
where J-. P(z', 2 0)dz' is normal distribution with mean μ H4, and variance of Ex. There- fore, it holds:
p(y: 1|
of (2,)) of
(10)
The evaluation of P(y. 0/z, 0) straightforward: P(y: - 0/2 0) P(y: 1 0).) 0).
The one-dimensional integral is still analytically intractable, but can be effectively evaluated by one- dimensional numerical integration. The proposed inference approach can effectively used in case of huge number of nodes, due to low computational cost of one-dimensional numerical integration.
3.4 INFERENCE GCRFBCNB MODEL
The inference procedure in GCRFBCub is much simpler, because marginalization with respect to latent variables is not performed. To predict y. it is necessary to evaluate posterior maximum of latent variable maax = argmax P(z (x 0), which is straightforward due to normal form of GCRF.
Therefore, it holds mmax = Mezi The conditional distribution P(ys = 1 1 Mzzi 0), where Ma,i is expectation of latent variable 2,, can be expressed as:
1
P(y: = 1/2, Us: 0) = o(M2,)) 1 + exp(-Maal
(11)
3.5 LEARNING IN GCRFBCB MODEL
In comparison with inference, learning procedure is more complicated. Evaluation of the condi- tional log likelihood is intractable, since latent variables cannot be analytically marginalized. The conditional log likelihood expressed as:
M
log {
M Σ j=1
C CY 0) = loog P(Y, 2 0)dZ
P(yj+ 2y kz 0)dz
(3 zz 0)
1=1
#j
(12)
(2))) a (2,:))) -- exp( - [zz M1) m3
N II
(zj -1i)
(y)lla log
dzj
(2T) |Σ 1/2
(13)
where Y E MRN is complete dataset of outputs, X E M is complete dataset of features, M is the total number of instances and A the total number of features. Please note that each instance is structured, so while different instances are independent of each other, variables within one instance are dependent.
One way to approximate integral in conditional log likelihood is by local variational approximation. Jaakkola & Jordan (2000) derived lower bound for sigmoid function, which can be expressed as:
a (x) > o{E) exp 4 X(E) {²)
(14)
[o(E) - }] and is a variational parameter. The Eq. i4 is called & transfor- mation of sigmoid function and yields maximum value when E z. This approximation can be applied to the model defined by Eq. [3] but the variational approximation has to be further extended because of the product of sigmoid funetions, such that:
where X(E) -
P(yj- 2y|m, 0) P(yr|zy) P(zylaz 0) P(yj | 0.E,) Eg)
(15)
N (E exp Aزالازز 1=1
Plys, 2,1,,,,003
II
Zja + Egi
X(E))) -Ex))
2
(16)
1 (-2
- my m1) - m4,
(2T) N/2 Σ, 1/2 exp
The I c6 arranged in the form suitable for integration. Detailed derivation of lower bound of conditional log likelihood is presented in Appendix A. The lower bound of conditional log likelihood
5

---

Under review as a conference paper at ICLR 2020
[3y zy. 0, E2E is defined as:
Eú
(y 0, E5) Pog
(Eji
+ >{ 122
2
(17)
1
1
2 b
jíj + 2 _mJsjj S my + log [S
2
where:
5 = Σn + 2A,
my ES
(wJ)
1 D
+ 2し
(18)
(EG1) 0
0
0
(5 0
0
A3
(19)
X(E
GCRFBCb uses the derivative of conditional log likelihood in order to find the optimal values for parameters a, and matrix of variational parameters & E RMxN, In order ensure positive definiteness of normal distribution involved, it is sufficient to constrain parameteres a >> O and , >
The partial derivatives of lower bound of conditional log likelihood are presented in Appendix B. For constrained optimization, the truncated Newton algorithm was used Nocedal & Wright (2006); Facchinei et al. (2002). The target function not convex, so finding global optimum cannot be guaranteed.
3.6 LEARNING GCRFBCNB MODEL.
In GCRFBCnb the mode of posterior distribution of continuous latent variable z is evaluated di- rectly, there is no need for approximation. The conditional log likelihood can be expressed as:
M
N
M N
C (Y = log P p)
Σ>
log P(yje M.)) Σ
Σ
(y 0 Mgj)
(20)
m, 0 1,1')) = Vyı log a11))) (1 yot) log (1 - a(nji))
(21)
The partial derivatives of conditional log likelihood are presented Appendix
EXPERIMENTAL EVALUATION
Both proposed models were tested and compared on synthetic data and real-world tasks All com- pared classifiers were compared in terms of the area under ROC curve (AUC) and accuracy (ACC). Moreover, the lower bound (in case of GCRFBCb) of conditional log likelihood & (Y| 6, μ and actual value (in case of GCRFBCnb) of conditional log likelihood L X 0) of obtained values on synthetic test dataset were also reported.
4.1 SYNTHETIC DATASET
The main goal of experiments on synthetic datasets examine models under various controlled conditions, and show advantages and disadvantages each. all experiments on synthetic datasets two different graphs were used (hence E R²2) and two unstructured predictors (hence a IR²). The results of experiments on synthetic datasets are presented in Appendix D.
It be noticed, that in where norm of the variances of latent variables is small, both mod- els have equal performance considering AUC and conditional log likelihood L (Y|X, 0). This is the case when values of parameters used in data generating process are greater or equal to the
IImlemmnntation can he found at Py Struct package does have option of returning SSVM and CRF confidence values for AUC evaluation
NB

---

Under review as a conference paper at ICLR 2020
values of parameters B. This means that the information provided by unstructured predictors is more important for classifications task than the information provided by output structure. There- fore, conditional distribution Ply, ax, 0) is concentrated around mean value and MAP estimate is satisfactory approximation. However, when data is generated from distribution with signifi- cantly higher values of /3 than a, the GCRFBCb performs significantly better than GCRFBCnb. For the larger values of variance norm, this difference is also large. This means that the structure between outputs has significant contribution to solving the classification task. It can be concluded that GCRFBCb has at least equal prediction performance as GCRFBCnb. Also, it can be argued that the models were generally able utilize most of the information (from both features and the structure between outputs), which can be seen through AUC values. In addition, distribution of local variational parameters were analyzed during learning. is noticed that each epoch, the variance of this distribution is small and that the parameters can be clustered and their number significantly reduced. Therefore, it is possible to significantly lower down computational and memory costs of GCRFBCb learning procedure, but that's out of the scope of this paper.
4.2 PERFORMANCE ON REAL-WORLD DATASETS
4.2.1 SKI CONGESTION
Data used in this research includes information on ski lift gate entrances in Kopaonik ski resorts, for the period March 15 to March 30 for the seasons from 2006 to 2011. The goal is to predict occurrence of crowding on ski lifts for 40 minutes in advance. Total number of instances in dataset was 4,850 for each ski lift, which is 33,950 in total.
Relatively simple method for crowding detection was devised for labelling data. We assume that, the crowding some gate occurs, distributions of sking times from other gates to that gate within some time window get shifted towards larger values. We model probability distribution of skiing time between two gates by the well-know parametric method of kernel density estimation (KDE) (Silverman 2018). The distribution shift is measured with respect to the mode of the distribution. The dataset generated by observing shifts in time windows of 5 minutes. When the mode of the distribution of skiing times within that window is greater than the mode for the whole time-span, the instance is labeled by 1 (crowding) and otherwise, is labeled by (no crowding). In order to obtain more information from the data distribution, additional 18 features were extracted.
Four different unstructured predictors that were trained on each class separately were used: ridge lo- gistic regression, LASSO logistic regression, neural network and random forest, whereas additional two unstructured predictors: decision tree and neural network were trained on all nodes together. Additionally, three structural support vector machine and two CRFs classifiers were used (Müller & Behnke, 2014). Fully connected graph of SSVM and CRF models are defined as SSVM-full and CRF-full, whereas Chow-Liu tree method for specifying edge connections are defined as SSVM- tree and CRF-tree, respectively. In the SSVM-independeet model the nodes of the graph are not connected.
Six different weighted graphs were used to capture dependence structure between ski lifts (nodes):x² statisties on labels of training set, mutual information between labels, correlation matrix between outputs of over-fitted neural networks, norm of difference between vectors of labels and two graphs were defined based on difference of vectors of historical labels and on differences of historical averages of skier times.
The AUC score and ACC of structured and unstructured predictors, along with the total computa- tional time are shown in Table [ It can be observed that GCRFBCb and GCRBCnb outperformed unstructured and other structured predictors in all cases. Based on evaluated parameters it could be concluded that dependence structure has significant impact overall prediction performance, even though, due to low values of norm of variance, GCRFBCb and GCRFBCnb have equal AUC scores. It can be summarized that advantages of structured models compared to unstructured are obvious, but in this particular task due to equal prediction performance and its lower computational and memory complexity, GCRFBCob is the best choice for this specifie application.
7

---

Under review as a conference paper at ICLR 2020
Table Prediction performance and computation time of classifiers Ski lifts congestion problem
Model
AUC
ACC
Calculation time [sec]
GCRFBCOb
0.831
0.749
119.554
GCRFBCb
0.831
0.749
3364.326
Ridge logistic
0.793
0.736
0.41
LASSO logistic
0.793
0.735
1.799
Neural network
0.790
0.720
151,571
Random forest
0.783
0.720
7.983
Decision tree together
0.681
8.297
Neural network together
0.711
13,997
SSVM full
0.622
517.412
SSVM- tree
0.615
580.475
SSVM independent
0.635
1029.172
CRF tree
0.745
16415.723
CRF- full
0.740
13942.542
Table 2: Prediction performance and computation time of classifiers Music classification accordint emotion
Model
AUC
ACC
Calculation time [sec]
GCRFBCиb
0.859
0.811
7.248
GCRFBCb
0.860
0.813
353.328
Ridge logistic
0.826
0.794
0.138
LASSO logistic
0.832
0.797
0.874
Neural network
0.811
0.783
98.132
Random forest
0.843
0.798
2.469
Decision tree together
0.736
0.564
Neural network together
0.782
8.471
SSVM full
0.755
76.817
SSVM tree
0.795
75.93
SSVM independent
0.784
146.867
4.2.2 MULTI-LABEL CLASSIFICATION OF MUSIC ACCORDING TO EMOTION
The dataset used for this work consists of 100 songs from 7 different genres. The collection was created from 233 musical albums choosing three songs from each album. rhythmic and 64 timbre features are extracted. The music is labeled in 6 categories of emotions: mmazed-surprised happy- pleased, relaxing-calm, quiet-still, sad-lonely and angry-fearful (Trohidis al., 2008). Total number of instances in dataset was 593. Four different weighted graphs were used: statistics labels of training set, mutual information between labels, correlation matrix between outputs of over-fitted neural networks and norm of difference between vectors of labels. Same unstructured predictors as in ski lift congestion problem were used, along with three structural support vector machine classifiers.
The performances of models are evaluated by 10 fold cross validation. The AUC score and ACC of structured and unstructured predictors, along with the total computational time are shown in Table 2 It can be seen that GCRFBCb has achieved the best prediction performances. The ACC of GCRFBC models significantly better than the SSVM performances. The AUC score and ACC of GCRGBCb are higher than the best result (AUC = 0.8237) presented in original paper Trohidis al., 2008). As in previous cases, computational time of GCRFBCb is significantly longer compared to GCRFBCnb and SSVM models.
4.2.3 GENE FUNCTION CLASSIFICATION
This dataset is formed by micro-array expression data and phylogenetic profiles with 2417 genes (instances). The number of features 103, whereas each gene associated with the set of 14 groups (Hlisseeff & Weston 2002). The same unstructured, structured predictors and weighted graphs, as
8

---

Under review as a conference paper at ICLR 2020
Table Prediction performance and computation time of classifiers
Gene classification problem
Model
AUC
ACC
Calculation time [sec]
GCRFBCOb 0.775
0.766
48.167
GCRFBCb
0.797
0.775
2297.727
Ridge logistic
0.582
0.539
0.079
LASSO logistic
0.583
0.540
0.188
Neural network
0.580
0,567
70.298
Random forest
0.601
0.615
5.529
Decision tree together
0.691
1.218
Neural network together
0.775
28.381
SSVM full
0.771
10137.049
SSVM- tree
0.768
722.156
SSVM independent
0.539
78.8870
in music according to emotion classification, were used. The 10-fold cross validation results of the classification are shown in Table 3
It can be observed that both GCRFBCb and GCRFBCnb achieved significantly better results in com- parison with unstructured predictors. However, neural network trained on all data together achieved the same ACC scores GCRFBCb. The AUC of GCRFBCb has outperformed Random forest clas- sifier by 19%, whereas SSVM tree has better ACC compared to GCRFBCnb. It also outperformed GCRFCnb, but expected, its computation time was longer. In addition, the computation time of CRFs models are longer compared to GCRFBCb
4.2.4 HIGHWAY CONGESTION
The E70-E75 motorway is a major transit motorway in Serbia. With 504 kilometers, is the one the major transit motorway in Serbia. It crosses the country from north-west to south, starting at Batrovei border crossing with the Republic of Croatia and ending with Preševo border crossing with the Republic of North Macedonia.
One of the biggest problems in E70-E75 motorway is high congestion that frequently occurs. One of the reasons lies in lack of open toll stations. In order to mitigate congestion problem, it is necessary to predict its occurrence and open enough toll stations. Data used this research includes infor- mation of car entrance and exit for the year 2017. Two different sections were analyzed: Belgrade Adaševci and Niš Belgrade. The section Belgrade Adaševci was analyzed for the period of January 2017, whereas section Niž Belgrade congestion was labeled using the similar technique based on KDE as presented in the ski lifts con- gestion problem. Based on raw datasets for sections Niž Belgrade and Belgrade Adaševei with 5,132,918 88 and 487,767 instances, respectively, a new dataset for section Niš Belgrade is generated by observing shifts in time windows of 10 minutes due large number of vehicles, whereas in the case of section Belgrade- Adaševci the shifts are observed in time windows of 20 minutes. Total numbers of instances for sections Belgrade- Adaševci and Niš Belgrade are 50,964 and 235,872, whereas numbers of highway exits (outputs) are 6 and 18, respectively. The extracted features are similar to the ones presented correlation matrix and difference of vectors of historical labels were used to capture dependence structure, whereas the same unstructured predictors as ski lifts congestion problem were evalu- ated. The classification results, validated by 10 fold cross validation,
analyzed for the period of April July 2017. The
ski congestion problem. The x2 statistics, mutual information,
presented in Table 4
The GCRFBCub achieved the highest AUC and ACC scores in the section Belgrade Adaševci, whereas GCRFBCb has better prediction performance in section Niš Belgrade. Moreover, in of section Niš Belgrade, GCRFBCb has worse ACC score than fully connected CRF, whereas CRF-tree outperformed GCRFBCnb in section Belgrade Adaševci

---

Under review as a conference paper at ICLR 2020
Table Prediction performance and computation time of classifiers Highway congestion problem
Ni8 Belgrade
Belgrade Adaševci
AUC
ACC
Calculation time [sec]
AUC
ACC
Calculation time [sec]
GCRFBCnb
0.740
0.684
344.166
0.974
0.925
90.321
GCRFBCb
0.751
0.692
13818.874
0.956
0.895
2103.749
Ridge logistic
0.716
0.681
10.73
0.917
0.856
1.771
LASSO logistic
0.716
0.680
30.12
0.917
0.856
1.657
Neural network
0.72
0.682
857.602
0.956
0,904
125,339
Random forest
0.739
0.683
209.589
0.965
0.914
3.826
Decision tree together
0.625
635.464
0.898
1.893
Neural network- together
0.664
125.441
0.880
16.475
SSVM
0.588
7637.794
0.739
340.806
SSVM tree
0.588
3684.138
0.755
392.597
SSVM - independent
0.602
3262,208
0.814
704.07
CRF tree
0.685
29749.054
0.88
26539.250
CRF- full
0.683
52563.972
0.898
25339.97
CONCLUSION
In this paper, new model, called Gaussian Conditional Random Fields for Binary Classification (GCRFBC) is presented. The model is based on latent GCRF structure, which means that intractable structured classification problem can become tractable and efficiently solved. Moreover, the im- provements previously applied to regression GCRF can be easily extended to GCRFBC. Two differ- ent variants of GCRFBC were derived: GCRFBCb and GCRFBCnb. Empirical Bayes (marginaliza- tion of latent variables) by local variational methods is used in optimization procedure of GCFRBCb, whereas MAP estimate of latent variables is applied in GCRFBCub. Based on presented method- ology and obtained experimental results synthetic and real-world datasets it can be concluded that both GCRFBCb and GCRFBCnb models have better prediction performance compared to the analysed structured unstructured predictors. Additionaly, GCRFBCb has better performance con- sidering AUC score, ACC and lower bound of conditional log likelihood L (Y X, 0) compared of the variances of latent variables is high. However, in cases
GCRFBCnb, in cases where where norm of the variances is close to zero, both models have equal prediction performance. Due to high memory and computational complexity of GCRFBCb compared GCRFBCub, in cases where norm of the variances is close to zero, it is reasonable to use GCRFBCub. Additionally, the trade off between complexity and accuracy can be made situation where norm of the variances is high. Further studies should address extending GCRFBC to structured multi-label classification problems, and lower computational complexity of GCRFBCb by considering efficient approximations.
REFERENCES
David Belanger and Andrew McCallum. Structured prediction energy networks. In International Conference Machine Learning, pp. 983-992, 2016.
David Belanger, Bishan Yang, and Andrew McCallum. End-to-end learning for structured prediction energy networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 429 439. JMLR. org, 2017.
Gang Chen, Yawei Li, and Sargur Srihari. Word recognition with deep conditional random fields. arXiv preprint 1612::1611.00177 2016.
Ryan Cotterell and Kevin Duh. Low-resource named entity recognition with cross-lingual, character-level neural conditional random fields. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pp. 91-96, 2017.
André Elisseeff and Jason Weston. A kernel method for multi-labelled classification. In Advances in neural information processing systems, pp. 68 181-687, 2002.
Francisco Facchinei, Stefano Lucidi, and Laura Palagi. truncated newton algorithm for large scale box constrained optimization. SIAM Journal Optimization, 12(4): 100-1125, 2002.
10

---

Under review as a conference paper at ICLR 2020
John Fox. Applied regression analysis and generalized linear models. Sage Publications, 2015.
Benjamin Frot, Luke Jostins, and Gilean MeVean. Graphical model selection for gaussian con- ditional random fields in the presence of latent variables. Journal of the American Statistical Association, (justaccepted),, 2018.
Jesse Glass, Mohamed F Ghalwash, Milan Vukicevic, and Zoran Obradovic. Extending the mod- elling capacity of gaussian conditional random fields while learning faster. In AAAI, pp. 1596- 1602, 2016.
Tommi Jaakkola and Michael Jordan. Bayesian parameter estimation via variational methods. Statistics and Computing, 10(1):25-33) 37, 2000.
Minyoung Kim. Mixtures of conditional random fields for improved structured output prediction. IEEE transactions on neural networks and learning systems, 28(5):1233-1 1240, 2017.
Sergey Kosov, Kimiaki Shirahama, Chen Li, and Marcin Grzegorzek. Environmental microorganism classification using conditional random fields and deep convolutional neural networks. Pattern Recognition, 77:248-261, 2018.
You Lu and Bert Huang. Structured output learning with conditional generative flows. arXiv preprint arXiv: 13288, 2019.
Laurens Maaten, Max Welling, and Lawrence Saul. Hidden-unit conditional random fields. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 479-488, 2011.
Kristen Masada and Razvan C Bunescu. Chord recognition in symbolic music using semi-markov conditional random fields. In ISMIR, pp. 272-278, 2017.
Andreas C Müller and Sven Behnke. Pystruct: learning structured prediction in python. The Journal of Machine Learning Research, 15()):0055-2660 2014.
Jorge Nocedal and Stephen J Wright. Numerical optimization 2nd, 2006.
Vladan Radosavljevic. Gaussian conditional random fields for regression in remote sensing. Temple University, 2011.
Vladan Radosavljevic, Slobodan Vucetic, and Zoran Obradovic. Continuous conditional random fields for regression in remote sensing. In ECAI, pp. 809-814, 2010.
Vladan Radosavljevic, Slobodan Vucetic, and Zoran Obradovic. Neural gaussian conditional ran- dom fields. Joint European conference on machine learning and knowledge discovery in databases, pp. 614-629. Springer, 2014.
Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
Jelena Stojanovic, Milos Jovanovic, Djordje Gligorijevic, and Zoran Obradovic. Semi Semi-supervised learning for structured regression on partially observed attributed graphs. In Proceedings of the 2015 SIAM International Conference on Data Mining, pp. 217-225. SIAM, 2015.
Gilbert Strang. Gilbert Strang, Gilbert Strang, and Gilbert Strang. Introduction linear algebra, volume 3. Cellesley-Cambrrdge Press Wellesley, MA, 1993.
Hongyu Su. Multilabel Classification through Structured Output Learning -Mehhds and Applica- tions. Aalto University, 2015.
Charles Sutton and Andrew McCallum. An introduction to conditional random fields for relational learning, volume 2. Introduction to statistical relational learning. MIT Press, 2006.
Chenhao Tan, Jie Tang, Jimeng Sun, Quan Lin, and Fengjiao Wang. Social action tracking via noise tolerant vareing factor graphs. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1049-1058. ACM, 2010.

---

Under review as a conference paper at ICLR 2020
Marshall F Tappen. Ce Liu, Edward H Adelson, and William T Freeman. Learning gaussian ditional random fields for low-level vision. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, IEEE, 2007.
Konstantinos Trohidis, Grigorios Tsoumakas, George Kalliris, and Ioannis Vlahavas. Multi-label classification of music into emotions. In ISMIR, volume 8, pp. 325-330, 2008.
Matt Wytock and Zico Kolter. Sparse gaussian conditional random fields: Algorithms, theory, and application to energy forecasting. In International conference on machine learning, pp. 1265- 1273, 2013.
Peng Zhang, Ming Li, Yan Wu, and Hejing Li. Hierarchical conditional random fields model for semisupervised sar image segmentation. IEEE Transactions on Geoscience and Remote Sensing, 53(9):4933 4951, 2015.
Haris Bin Zia, Agha Ali Raza, and Awais Athar. Urdu word segmentation using conditional random fields (crfs). arXiv preprint arXiv: 05432, 2018.
12

---

Under review as a conference paper at ICLR 2020
A DERIVATION OF LOWER BOUND OF CONDITIONAL LIKELIHOOD
In this section we derive lower bound of conditional likelihood. In order to obtain suitable form of joint distribution that be easily integrated, the lower bound for sigmoid function was used Jaakkola & Jordan 2000). The lower bound of joint distribution P(yj. žj 22 0) can be expressed as:
P(ys, Zylæs, 0) P(uylzy) 0) 2 P(ys. 2j 25| 0. Eg)
(22)
P(y,, (,,,,,),
II i=1
(E exp
3js + Eyi X(E))) - -E:)
2
(23)
1
1 (2) (25
My) 11)
1/2 exp
N/2 (2T) (2T)N/2
The simplified form of Eq. 23 can be represented by rearranging terms in the following form:
23 25(00 1) (/,5,,lzz,0055) 0,E,) = T({)) exp
1 ーズ 2j
23 23 1)
(24)
N (E,i) exp
1 (- - E,r 2 + ((S)))
T(Es)
(25)
N/2 |
F=1
The lower bound of likelihood P(yj (ay. 0. Eg) can be obtained by marginalization of 2j as:
P(yjlaz, 0,E;) = / P(yj. 25lz,. 0, Eg dzj
= =T(E)) / exp
2) (y I A1232
+ Us) dzj
= - / exp
1 2 (x,² 2A,)zz+
( 2A,) + 22,)
- 1 =, dzj
(26)
The lower bound of likelihood P(y,læy, 0. Eg) can be transformed in the following form:
(uull,,,,E3) = -(ss)) / exp (-2
1
2 (zj 55) (zj -m²) + _ m 55mmm
mm dzz
2
(1
my m S7 m3) ) / exp [ - mz)² 5 (zg my) dzj
=T(E;) exp
5/'m))
(27)
where 57 - + 2A, and m, E, ((y) - inn 1
This integration is easily performed by noting that is the integral over an unnormalized Gaussian distribution, which yields:
|Σ, 1/2 T(E,) exp {mm s mmm) s-mm)ssm)))))) 1/2
Plys Jn 0,5,)
(28)
The final form of the lower bound of conditional log likelihood Lyllll 0, Eg) is:
13

---

Under review as a conference paper at ICLR 2020
[, 0,E5) - Plys
Σ log (ogg((5;))
Eúi 2
+ >(E)))
(29)
1 1
1
2 2 pj + my 1 my + log [Sg]
2
B PARTIAL DERIVATIVE OF LOWER BOUND OF CONDITIONAL LOG LIKELIHOOD
The partial derivative of lower bound of conditional log likelihood (GCRFBCb) 0c, (My 0.5,) computed as:
is
dog
05, ( (ys|an. 0, E5)
a5j1
Jmj
Tr
S
S
m, 2 m T
m,
dor.
dank
dos
Box
(30)
Enj وīj!
0o
14j
11 T
Dok
Tr
8o
where:
aS_1
02-
2, if =j
(31)
Oak
Dak
0, if $
0u7
00,
yj
+ m 1
Sg
Sj +
E Sj
+
1
Sj
(32)
daz
2
dak
dak
ak
02,
20% (x))
1í
(33)
dak
Similarly partial derivatives with respect to 3 can be defined as:
ac, (1r jzj. O, Ej)
05,7
Jmg
T 85,
Tr
S
5 m, + 2 m 08,
m
03
2
0
03:
(34)
O 1
0Σ
2
-rr
ΣS 03,
03,
03,
2
where:
aSp1
0Σ,1
N Sy (2)), ifi=
(35)
05:
03
St (x), if
dmy
1 Uj
I + 11
S
0S-1
SJ
0u
Mj E S +
11
T
02,1
S
(36)
0
2
03,
03
B1
0p
05,
E
(37)
03
03
In the same manner partial derivatives of conditional log likelihood with respect to Ey₂ are:
Oc, (yylæ,,0.E) 0.Es)
1
0A
= Tr 25
2 yj
1
S
Sj
2
KE,
m T. .Ai mj
N
1 + S3
(o
1
3
Σ
+ -
((()i))
a{Ext)
2
f=1
(38)
14

---

Under review as a conference paper at ICLR 2020
where:
aA
aM(S
(39)
(E1))
(Efi) o(Esi))
(40)
0hij
@A(E)))
1 (5(()
1 (( (o(ss)) - Ej
(o(5)i)- 2
(41)
2z
2
C PARTIAL DERIVATIVE OF CONDITIONAL LOG LIKELIHOOD
The derivatives of the conditional log likelihood (GCRFBCnb) with respect to and /3 are defined as, respectively:
(Uya o. 0. M...))
Opye
(y3s o (pfi))
(42)
dock
Das
( [r O, M.j)
0pgi
(リ3² a (pzi))
(43)
don
05
where
By' and 0py Bag respectively.
are elements of the vectors
3P4 and Bu4 and can be obtained by Eqs. and 37 day 89,
8%,
D SYNTHETIC DATASET RESULTS
In order to generate and label graph nodes, edge weights S and unstructured predictor values R were randomly generated from uniform distribution. Besides, was necessary choose values of param- eters and B3 Greater values of a indicate that the model is more confident about performance of unstructured predictors, whereas for larger value of / the model is putting more emphasis on dependence structure of output variables.
Six different values of parameters a and & were used. In the first group a and f have similar values, unstructured predictors and dependence structure between outputs have similar importance. In the second group, a has higher values compared to 3,, which means that unstructured predictors more important than the dependence structure. In the third group 3 has higher values than a, meaning that dependence structure is more important than unstructured predictors.
Along with the AUC and conditional log likelihood, norm of the variances of latent variables (di- agonal elements in the covariance matrix) is evaluated and presented Table 55) addition, the results of experiments are presented in Fig. where for different values of and B we show dif- ferences between GCRFBCb and GCRFBCnb (a) AUC scores, (b) log likelihoods, and (c) norm of the variances of latent variables.
15

---

Under review as a conference paper at ICLR 2020
6.5
600
5875
600
23770 2 50 35 50
10
13
16
25
28
Figure 1: Experimental evaluation of differences between GCREBCb and GCRFBCub (a) AUC scores, (b) log likelihoods, and (c) norms of the variances of latent variables for different values of a and |
Table 5: Comparison of GCRFBCb and GCRFBCnb prediction performance for different values of at and B, as measured by AUC, log likelihood, and norm of diagonal elements the covariance matrix
No.
Parameters
GCRFBCb
GCRFBCnb
AUC
C
|
AUC
L Y X,0
1
a
5, 4
0.812
-71.150
0.000
0.812
-71.151
5, 22
2
Q
1, 18
0.903
-75.033
0.001
0.902
-75.033
3 11 18
[22, 21
3
0.988
-83.957
0.000
0.988
-83,957
3 = [5,22]
[2,
4
0.866
-83.724
0.000
0.886
-83,466
β 00.1,
Q 0.8, 0.5)
5
0.860
-83.353
34.827
0.817
-84.009
3 5.22
a = 0.2,
6
0.931
-70.692
35.754
0.821
-70.391
8 1,1
16


Reviews:
[{"review_id": "r1gpwXPsYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work involves modifiying gaussian conditional random fields to work for classification problems instead of regression problems. The main idea is to apply a bernoulli distribution on top of the regression values to convert them to work with binary classification problems. Two variations are discussed along with the inference and learning methodology. The inference can be done using numerical approximation and learning using variational methods and is still untracktable. Comparisons with other modeling strategies is done using experiments.\n\nThe paper is incremental and doesn't really provide improvements to learning parameters (or at least there is no theory showing this in the paper). The experiments do not seem satisfactory as discussed below.\na) Applying a bernoulli distribution on the output of the GCRF seems trivial. It is not very clear when the GCRFBCb model would be better than the GCRFBCnb. The learning procedure is untracktable and hard to follow on why this might provide better results.\nb) The datasets (music classification and gene classification) don't seem to be good datasets for structured predictions i.e. the interaction needed between the nodes is not clear. Since they are multilabel problems, one could have just modeled the system with N independent nodes or design a multinomial distribution instead of only for binary classification.\nc) There should be more thorough fine-tuning of other models, for e.g. in the ski lifts experiment, the CRF does much worse than logistic regression in the results. This is most likely because the parameters were not initialized properly using normal tricks like using logistic regression. Typically for truly structured problems, CRFs do better than their logistic regression counter parts. It is also not clear how the other models (CRF and SSVM) pairwise potentials were modeled.\n\nIt would really help to make this paper stronger by showing the new modeling technique does better than CRFs (that are tuned properly) on better structured datasets. It would be good to have a discussion on when this model would do worse than the other structured models and why.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkxXn-jJ9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors break the double blind anonymity with the code link provided. I'll leave how to deal with this to the meta reviewer. \n\nThe authors provide a method to modify GRFs to be used for classification. The idea is simple and easy to get through, the writing is clean. The method boils down to using a latent variable that acts as a \"pseudo-regressor\" that is passed through a sigmoid for classification. The authors then discuss learning and inference in the proposed model, and propose two different variants that differ on scalability and a bit on performance as well. The idea of using the \\xi transformation for the lower bound of the sigmoid was interesting to me -- since I have not seen it before, its possible its commonly used in the field and hopefully the other reviewers can talk more about the novelty here. The empirical results are very promising, which is the main reason I vote for weak acceptance. I think the paper has value, albeit I would say its a bit weak on novelty, and I am not 100% convinced about the this conference being the right fit for this paper. The authors augment MRFs for classification and evaluate and present the results well. \n\nCan the authors intuit why random forests and neural nets dont perform as well ? It seems there are many knobs one can tune to get better performance, so I will take the presented results with a grain of salt. Also, it seems one can also use other \"link\" functions with MRFs (similar to link functions in generalized linear models) to not just do logistic but other possible losses as well. How about multiclass classification using softmax ? I think such generalizations would make this paper lot stronger. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lu-0I2Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "TITLE\nGaussian Conditional Random Fields for Classification\n\nREVIEW SUMMARY\nA well justified approach to structured classification with demonstrated good performance. \n\nPAPER SUMMARY\nThe paper presents methods for structured classification based on a Gaussian conditional random field combined with a softmax Bernoulli likelihood. Methods for inference and parameter learning are presented both for a \"Bayesian\" and maximum likelihood version. The method is demonstrated on several data sets.\n\nQUALITY\nIn general, the technical quality of the paper is good. Except for minor typos, derivations appear to be correct, although I did not check everything in detail. \n\nCLARITY\nThe paper could be improved by a careful revision with focus on improving grammar, but as it stands the paper is easy to follow.\nIt is not clear to me exactly how the numbers in Table 1 were computed. Is this based on 10-fold crossvalidation as in the following tables?\n\nORIGINALITY\nI am not familiar enough with the field to assess the novelty of the contribution. It would be great if the paper provided a better overview of competing structured classification methods.\n\nFURTHER COMMENTS\n\n\"structured classification\" ?\n\n\"It was shown\" -> We show\n\n\"for given\"\n\nIs the second sum over k=1 to K in eq. 1 a mistake?\n\n\"We void\" -> We avoid\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

Blind review #2 summarizes it well:

The authors provide a method to modify GRFs to be used for classification. The idea is simple and easy to get through, the writing is clean. The method boils down to using a latent variable that acts as a "pseudo-regressor" that is passed through a sigmoid for classification. The authors then discuss learning and inference in the proposed model, and propose two different variants that differ on scalability and a bit on performance as well. The idea of using the \xi transformation for the lower bound of the sigmoid was interesting to me -- since I have not seen it before, its possible its commonly used in the field and hopefully the other reviewers can talk more about the novelty here. The empirical results are very promising, which is the main reason I vote for weak acceptance. I think the paper has value, albeit I would say its a bit weak on novelty, and I am not 100% convinced about the this conference being the right fit for this paper. The authors augment MRFs for classification and evaluate and present the results well. 

--

Discussion:

As blind review #1 points out:

Even from the experiments (including the new traffic one), it is unclear how much better the method is either because we don't know if the improvements are statistically significant and that in many of the results, unstructured models like RF or logistic regression are very competitive casting some doubt on whether these datasets were well suited for structured prediction.

--

This paper is a desk reject as review #2's points out that anonymity was broken by the inclusion of a code link that reveals the authorship, which is true as a simple search on the GitHub user "andrijaster" immediately brings us to https://arxiv.org/pdf/1902.00045.pdf which is a draft of this submission showing all author names.
Decision: Reject