Deep Dependency Networks for Action Classification in Video
Authors: Shivvrat Arya
Yu Xiang
Vibhav Giridhar Gogate
Venue: Submitted to ICLR 2023 (2023)
Keywords: probabilistic graphical models
action classification
multi-label classification
combining probabilistic models with deep learning
end-to-end learning

Abstract: We propose a simple approach which combines the strengths of probabilistic graphical models and deep learning architectures for solving the multi-label action classification task in videos. At a high level, given a video clip, the goal in this task is to infer the set of activities, defined as verb-noun pairs, that are performed in the clip. First, we show that the performance of previous approaches that combine Markov Random Fields with neural networks can be modestly improved by leveraging more powerful methods such as iterative join graph propagation, $\ell$-1 regularization based structure learning and integer linear programming. Then we propose a new modeling framework called deep dependency network which augments a dependency network, a model that is easy to train and learns more accurate dependencies but is limited to Gibbs sampling for inference, to the output layer of a neural network. We show that despite its simplicity, joint learning this new architecture yields significant improvements in performance over the baseline neural network. In particular, our experimental evaluation on three video datasets: Charades, Textually Annotated Cooking Scenes (TaCOS), and Wetlab shows that deep dependency networks are almost always superior to pure neural architectures that do not use dependency networks.

Under review as a conference paper at ICLR 2023
DEEP DEPENDENCY NETWORKS FOR ACTION CLAS- SIFICATION IN VIDEOS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a simple approach which combines the strengths of probabilistic graphical models and deep learning architectures for solving the multi-labe ac- tion classification task in videos. At a high level, given video clip, the goal in this task is to infer the set of activities, defined as verb-noun pairs, that are per- formed in the clip. First, we show that the performance of previous approaches that combine Markov Random Fields with neural networks can be modestly im- proved by leveraging more powerful methods such iterative join graph propa- gation, l regularization based structure learning and integer linear programming. modeling framework called deep dependency network which augments a dependency network, a model that is easy to train and learns more accurate dependencies but is limited to Gibbs sampling for inference, the output layer of neural network. We show that despite its simplicity, joint learn- ing this new architecture yields significant improvements performance over the baseline neural network. In particular, our experimental evaluation on three video datasets: Charades, Textually Annotated Cooking Scenes (TACoS), and Wetlab shows that deep dependency networks are almost always superior to pure neural architectures that do not use dependency networks.
Then we propose
INTRODUCTION
We focus on the following multi-label action classification (MLAC) task: given a video partitioned into segments (or frames) and pre-defined set of actions, label each segment with subset of actions from the pre-defined set that are performed in the segment (or frame). We consider each action to be verb-noun pair such as "open bottle", "pour water" and "cut vegetable". MLAC is special case of the standard multi-label classification (MLC) task where given a pre-defined set of labels and a test example, the goal is to assign each test example to a subset of labels.
well known that MLC notoriously difficult because in practice the labels are often correlated and thus predicting them independently may lead to significant errors. Therefore, most advanced methods explicitly model the relationship or dependencies between the labels, either using proba- bilistic techniques (cf. Wang al. (2008); Guo & Xue (2013); Tan al. (2015): Di Mauro et al. (2016); Wang et al. (2014); Antonucci et al. (2013) or non- non-probabllisticneeu
methods (cf. Papa- giannopoulou et al. (2015); Kong al. (2013) Wang al. (202 Nguyen al. (2021) Wang al. (2021b)) Intuitively, because MLAC is special case of MLC. methods used for solving MLAC should model relationships between the actions well between the actions and features. To this end, the primary goal of this paper is to develop simple, general- general-purposs purpose scheme that models the relationships between actions using probabilistic representation and reasoning techniques and works on top of neural feature extractors in order to improve their generalization performance.
Our work is inspired by previous work on hybrid models that combine probabilistic graphical mod- els (PGMs) with neural networks (NNs) under the assumption that their strengths are often comple- mentary (cf. JJonson et al. (016)); Krishnan et al. (015)). At high level, in these hybrid models NNs perform feature extractions while PGMs model the relationships between the labels as well as between features and labels. In previous work, these hybrid models have been used for solving a range of computer vision tasks such image crowd counting (Han et al. 2017), visual relationship detection (Yu et al., 2022), modeling for epileptic seizure detection in multichannel EEG (Craley et al. 2019), face sketch synthesis (Zhang et al. 2020), semantic image segmentation (Chen al.

---

Under review as a conference paper at ICLR 2023
2018 Lin et al. 2016), animal pose tracking (Wu et al. 2020 and pose estimation (Chen & Yuille 2014 In this paper, we seek to extend and adapt these previously proposed PGM+NN approaches for solving the multi-label action classification task in videos.
Motivated by approaches that combine PGMs with NNs, as a starting point, we investigated using Markov random fields (MRFs), type of undirected PGM to capture the relationship between the labels and features computed using convolutional NNs. Unlike previous work which used these MRF+CNN or CRF+CNN hybrids with conventional inference schemes such as Gibbs sampling (GS) and mean-field inference, our goal was to evaluate whether advanced reasoning and learning approaches, specifically (1) iterative join graph propagation (IJGP) (Mateescu al. 2010) (a type of generalization Belief propagation technique Yedidia et al.] (2000)), (2) integer linear programming (II.P) based techniques for computing most probable explanations, and (3) logistic regression with l l--reuularization based methods (Lee et al., [006] Wainwright et al. (006) for learning the structure of pairwise MRFs, improve the generalization performance of MRF+CNN hybrids. To measure generalization accuracy, we used several metrics such as mean average precision (mAP), label rank- ing average precision (LRAP), subset accuracy (SA) and the jaceard index (JI) and experimented on three video datasets: (1) Charades (Sigurdsson al. 2016), (2) TACoS (Regneri et al. 2013) and (3) Wetlab (Naim et al. 2014). We found that, generally ly speaking, both IJGP and ILP are superior to the baseline CNN and Gibbs sampling in terms of JI and SA but are sometimes inferior to the CNN in terms of mAP and LRAP. We speculated that because MRF structure learners only allow pairwise relationships and impose sparsity or low-treewidth constraints for faster, accurate inference, they often yield poor posterior probability estimates in high-dimensiona settings. Since both mAp and LRAP require good posterior probability estimates, GS, IJGP, and ILP exhibit poor performance when mAP and LRAP are used to evaluate the performance.
To cireumvent this issue and in particular, to derive good posterior estimates, we propose a new PGM+NN hybrid called deep dependency networks (DDNs). At a high level, a dependency network (DN) (Heckerman et al. 2000) represents a joint probability distribution using a collection of con- ditional distributions, each defined over a variable given all other variables in the network. A DN is consistent if the conditional distributions come from a unique joint probability distribution. Other- wise, it is called inconsistent. A consistent DN has the same representation power as a MRF in that any MRF can be converted to a consistent DN and vice versa. A key advantage of DNs over MRFs is that they are easy to train because each conditional distribution can be trained independently and modeled using classifiers such as logistic regression, decision trees, and multi-layer perceptrons. These classifiers can be easily defined over a large number of features and as a result the conditional distributions (defined by the classifiers) are often more accurate than the ones inferred from a sparse or low-treewidth MRF learned from data. However, DNs admit very few inference schemes and are more or less restricted to Gibbs sampling for inference in practice. The second disadvantage of DNs is that because the local distributions are learned independently, they can yield an inconsis- tent DN, and thus one has to be careful when performing inference over such DNs. Despite these disadvantages, DNs often learn better models which typically translates to superior generalization performance (cf. (Neville & Jensen 2003; Guo & Gu 2011)).
In our proposed deep dependency network (DDN) architecture, a dependency network sits on the top of convolutional neural network (CNN). The CNN converts the input image or video segment to set of features, and the dependency network uses these features to define a conditional distribution over each label (action) given the features and other labels. We show that deep dependency models are easy to train either jointly or via a pipe-line method where the CNN is trained first, followed by the DNN by defining appropriate loss function that minimizes the negative pseudo log- likelihoooo of the data. We conjecture that because DDNs can be quite dense, they often learn a better represen- tation of the data, and as result, they are likely outperform MRFs learned from data in terms of posterior predictions.
We evaluated DDNs using the four aforementioned metrics and three datasets. We observed that they are often superior to the baseline neural networks as well as MRF+CNN methods that use GS, IJG and ILP on all four metries. Specifically, they achieve the highest score on the mAP metric on all the datasets. We compared the pipeline model with the jointly learned model and found that the joint model is more accurate than the pipeline model.
In summary, this paper makes the following contributions:
2

---

Under review as a conference paper at ICLR 2023
We propose a new hybrid model called deep dependency networks that combines the strengths of dependency networks (faster training and ability to use advanced classifiers) and neural networks (flexibility. feature representation).
We define new loss function for jointly training the neural and dependency network com- ponents of the deep dependency network. We show that DDNs are easy to train using these loss functions.
We experimentally evaluate DDNs on three video datasets and using four metrics for solv- ing the multi-label action classification task. We found that jointly trained DDNs consis- tently outperform the baseline CNNs and MRF+CNN hybrids on all metrics and datasets.
2 PRELIMINARIES
A log-linear model or a Markov random field (MRF), denoted by M, is an undirected probabilistic graphical model (cf. Koller & Friedman (2009)) that widely used in many real-world domains for representing and reasoning about uncertainly. It is defined as triple (X, F, θ) where X = {X1....: .,,}
set of Boolean random variables, F (. is a set of features such that each feature fi (we assume that a feature is a Boolean formula) is defined over subset D; of X, and O (O,,..., J is a set of real-valued weights or parameters, namely V0, E O; 0, E I such that each feature Ji is associated with parameter 0z. M represents the following probability distribution:
k
P(x) = 7(0) exp Σ 0.f (XD.)
1
(1)
where XD, is the projection of x on the variables D; of the feature fi, (xD.) is an indicator function that equals when the assignment XD, evaluates fs to True and is 0 otherwise, and (()) is the normalization constant called the partition function.
We focus on three tasks over MRFs: (I) structure learning which is the problem of learning the fea- tures and parameters given training data; (2) posterior marginal inference which is the task of com- puting the marginal probability distribution over each variable the network given evidence (evi- dence is defined as an assignment of values to a subset of variables); and (3) finding the most likely assignment to all the non-evidence variables given evidence (this task is often called maximum-a- posteriori or MAP inference in short). All of these tasks are at least NP-hard in general and therefore approximate methods are often preferred over exact ones in practice.
A popular and fast method for structure learning is to learn binary pairwise MRFs by training an L G-regularized logistic regression classifier for each variable given all other variables as features (Wainwright et al. 2006 Lee et al. 2006). f--regularization induces sparsity in that it encourages many weights to take the value zero. All non-zero weights are then converted into conjunctive features. Each conjunctive feature evaluates True if both variables are assigned the value 1 and to False otherwise. Popular approaches for posterior marginal inference are the Gibbs sampling algorithm and generalized Belief propagation (Yedidia al. 2000) techniques such Iterative Join Graph Propagation (Mateescu et al. 2010). For MAP inference, a popular approach is to encode the optimization problem as linear integer programming problem (cf. Koller & Friedman (2009)) and then using off-the-shelf approaches such as Gurobi Optimization, (L.C22022 to solve the latter.
Dependency Networks (DNs) (Heckerman et al. 2000) represent the joint distribution using a set of conditional probability distributions, one for each variable. Each conditional distribution defines the probability of a variable given all the others. A DN consistent if there exists a joint probability distribution P(x) such that all conditional distributions Pı(xx(x_.) where x is the projection of x on X \ {X,). are conditional distributions of P(x).
A DN is learned from data by learning probabilistic classifier (c.g., logistic regression, multi-layer perceptron, etc.) for each variable, and thus DN learning is embarrassingly parallel. However, because the classifiers are independently learned from data, we often get inconsistent DN. It has been conjectured (Heckerman et al. 2000) that most DNs learned from data are almost consistent in that only few parameters need to be changed in order to make them consistent.
The most popular inference method over DNs is fixed-order Gibbs sampling Lu 2008). the DN is consistent, then its conditional distributions are derived from joint distribution P(x), and
3

---

Under review as a conference paper at ICLR 2023
F
X1
P1 (x11))
Neural Network
E2
22
Pa(2g|x-3,e)
Video frames
Output of NN
Figure 1: Dependency Network structure for three actions. Video frames are given as input to the NN and it produces the features for the DN (E1, E2, E3). These features are then used to model the conditional distributions each variable X,. At each node X,, the form of the conditional distribution is the variable given its parents (incoming arrows) in the graph.
the stationary distribution (namely the distribution that Gibbs sampling converges to) will be the same P(x). If the DN is inconsistent, then the stationary distribution of Gibbs sampling will be inconsistent with the conditional distributions.
3 DEEP DEPENDENCY NETWORKS
In this section, we describe how to solve the multi-label action classification task in videos using hybrid of dependency networks and neural networks. At high level, the neural network provides high quality features given videos and the dependency network represents and reasons about the probabilistic relationships between the labels and features.
3.1
THE MODEL
Let V denote the set of random variables corresponding to the pixels and v denote the RGB values of the pixels frame or video segment. Let E denote the (continuous) output nodes of neural network which represents function N v - that takes as input and outputs an assignment e to E. Let X = {X1,- X,,} denote the set of actions (labels). For simplicity, we assume that Given (V, E, X). deep dependency network (DDN) is pair (A D) where A a neural network that maps V = v to = e and D is a conditional dependency network (Guo & Gu 2011) that models Pxxle) where = N(v). The conditional dependency network represents the distribution P(xle) using collection of local conditional distributions P. (x. e), one for each action (label) X,, where x { Tz 1 -.... )n) Fig. shows the network architecture for = 3. Note that the goal is to model v(x|v) as accurately as possible and we assume that P(x|v)
E X
Pxxje). Our hope that Pxxle) is easier learn and represent compared to P(x(v), ), and thus the loss functions at the neural network and the conditional dependency network must be carefully chosen facilitate this objective.
3.2 INFERENCE
Once a DDN is learned from data, at test time, it can be used to provide labels for given video frame or segment as follows. We first send the video frame through the neural network N to yield assignment to the output nodes of the neural network. Then, perform fixed-order Gibbs Sampling over the conditional dependency network where the latter represents the distribution P(xle). Finally, given samples (x(1) ..... xN) generated via Gibbs sampling, we estimate the marginal probability distribution of each label X; using the following mixture estimator [Lu 2008
1 P(..vv N
Σ
P. (1
x xx
4

---

Under review as a conference paper at ICLR 2023
Poi
E1
P1
Neural Network
Frames
Output of NN
Figure 2: Computational graph for the forward pass. Each edge is associated with a weight (param-
eter), either fgg or Ttj.
3.3 LEARNING
We can either train the DDN using pipeline method or via joint training. the pipeline method, we first train the neural network using conventional approaches (e.g., using cross-entropy loss) or we can use a pre-trained model. Then for each training example (v, x), we send the video through the neural network to obtain a representation of v. The aforementioned process transforms each training example (v. x) into new feature representation (e, x) where e = N(v). Finally, for each action (label) X,, we learn probabilistic classifier to model the conditional distribution Pr(xx(x-j. e) using the transformed training data. Specifically, given a training example (e, x), each probabilistic classifier indexed by i, uses X, the class variable and (E U x) the attributes. In our experiments, we used two classifiers, logistic regression and multi-layer perceptron.
key advantage of using the pipeline method is that training is relatively fast and can be easily paral- lelized. Moreover, it requires only modest computational resources. As a result, the pipeline method especially beneficial in the following scenario: limited computational resources are available but a pre-trained neural network that is trained on high- powered GPUs is readily available.
In practice, in order perform joint training, we need to select appropriate loss functions defined over the parameters of the conditional distributions and then perform backpropagatioongradiee de- scent to minimize the loss functions. Intuitively, given pre-trained model with output nodes E and set of probabilistic classifiers which model P(Xile, x.)), we can simply define an appropriate loss function at X; and back-propagate it to the neural network. For instance, given a training example (v, x) and under the assumption that logistic regression classifier with parameters (o,, II,) is used to model (X,,,e,, x-1) where O are weights associated with x and II; are weights associated with e, we can the following cross-entropy loss at each classifier indexed by i:
L,DDN z, log PX, 1/c, x : II, (1 x;)) log P, x II,
(2)
We can then back-propagate this loss via the parameters II; to each output node of the NN.
Fig. 2 shows an example computation graph (forward pass) associated with the loss function given in equation for three labels. At a high level, when logistic regression is used, learning time, the loss function alters the structure of the neural network by adding extra layer (dependency network) as shown. While training, this layer acts as multiple MLPs (each representing the distribution for X,). However, note that test time, we have use Gibbs sampling described in the previous section.
An issue with the loss function just described that large parameter gradients from the dependency network may rapidly change the parameters of the pre-trained network; thus undoing much of the work done during pre-training. Therefore, we propose to use the following loss function at each output node of the neural network
A Σ CADDN n ==1
LNNN (-x, log e; (1 x.)) log(1 ex)) +
(3)
5

---

Under review as a conference paper at ICLR 2023
where À 2 0 is a hyper- tlper--aramettr. Note that the loss given in equation is a generalization of the one given in equation 2 because for large values of X, the effect of the first two terms disappears.
The main advantage of the loss given in equation Bis that it allows to dampen the negative effects of any wrong predictions made at the dependency network layer, on the correctly predicted output node in the neural network. For example, in Fig. 2 let us consider node E₁ which is connected to all nodes in the dependency network. Let us assume that Ej and X, were correctly predicted for a training example, while the predictions at X2 and X3 were incorrect. In such situations, we do not want E, to be unnecessarily penalized (since it was correctly predicted). However, the loss given in equation 2 will penalize E1 via backpropagation through the edges (E1, X2) and (E1, Xg). On the other hand, the loss given in equation 3 will dampen the effect via A.
Our proposed loss function (see equation B) is similar the ones used in multi-task learning (cf. (Caruana 1997h) where we have two more networks that share substructure. In our setting, we have two networks, one which explicitly reasons about dependencies between the labels and another which does not.
4 EXPERIMENTAL EVALUATION
In this section, we evaluate the proposed models on the multi-label activity classification task using three video datasets. We begin by describing the datasets and metrics, followed by the experimental setup and conclude with the results. All models were implemented using PyTorch, and one NVIDIA A40 GPU was used to train and test all the models.
4.1 DATASETS AND METRICS
We evaluated our algorithms on the following three video datasets: (1) Charades (Sigurdsson et al,, 2016); (2) Textually Annotated Cooking Scenes (TACoS) (Regneri al. 2013); and (3) Wetlab (Naim et al. 2015). In the Charades dataset, the videos are divided into segments (video clips) and each segment is annotated with one or more action labels. In the TACoS and Wetlab datasets, each frame is associated with one or more actions.
Charades Dataset (Sigurdsson et al., 2016) comprises of videos of people performing daily indoor activities while interacting with various objects. It is multi-label activity classification dataset; thus, more than one activity can happen at a given time. the standard split, there are 7,986 training videos and 1,863 validation videos, averaging 30 seconds. We used the training videos to train the models and the validation videos for testing purposes (because the authors have not released the test set). The dataset has roughly 66,500 temporal annotations for 157 action classes. We report mean Average Precision (mAP) following the standard evaluation protocols for this dataset. We also report Label Ranking Average Precision (LRAP) and Jaccard Index (JI), these metrics have been used extensively in previous work for evaluating the performance of multi-label classifiers.
The Textually Annotated Cooking Scenes (TaCOS) dataset (Regneri al., 2013) consists of third- person videos of a person cooking in a kitchen. The dataset comes with hand-annotated labels of actions, objects, and locations for each video frame. From the complete set of these labels, we selected 28 labels to create our dataset. Each of these labels corresponds to a location, an object, or verb, and each action defined as a (location, object, verb) triple. By selecting the videos that correspond to these labels and dividing them into train and test sets, we get a total of 60313 frames for training and 9355 frames for testing, spread out over 17 videos.
The Wetlab dataset (Naim et al. 2014) is comprised of videos where experiments are being per- formed by lab technicians that involve hazardous chemicals in a wet laboratory. The dataset con- tains 6 videos where each video follows three protocols: Cellobiose M9 Media (CELL), LB Liquid Growth Media (LLGM), and Yeast (YPAD) Media. We used one video for testing and five videos for training. Specifically, the training set comprises of 100054 frames, and the test set comprises of 11743 frames. Each activity in the dataset can have multiple objects per activity (for example, an activity "transfer contents"
have two objects linked with it, test tube and a beaker) and there are 57 possible labels. Each label corresponds to an object or a verb, and each action is made of one more labels from each category.

---

Under review as a conference paper at ICLR 2023
We evaluated the performance on the TACoS and Wetlab datasets using the following four metries: mean Average Precision (mAP), Label Ranking Average Precision (LRAP), Subset Accuracy (SA), and Jaccard Index (JI). Note that SA is not relevant for the Charades dataset. mAP and LRAP require access to an estimate of the posterior probability distribution at each label while SA and JI con-prbbabilistic and only require assignment to the labels.
4.2 EXPERIMENTAL SETUP AND METHODS
We used three types of architectures in our experiments: (1) Baseline convolutional neural networks (CNNs) which are specific to each dataset; (2) CNNs augmented with MRFs, which we will refer to as deep random fields or DRFs in short; and (3) The method proposed in this paper which uses a dependency network on top of a CNN called deep dependency networks (DDNs).
Convolutional Neural Networks. We choose two different types of models for CNNs, and they act as baseline for the experiments and as a feature extractor for DRFs and DDNs. We chose two types of models for this, 2D CNNs and 3D CNNs. In doing this, we want to demonstrate that the proposed method can work well for two different types of feature extractors. For the Charades dataset we use the PySlowFast (Fan et al. 2020) implementation of the SlowFast Network (Feichtenhofer et al. 2019) which uses a 3D ResNet model as the backbone. For the TACoS and Wetlab datasets, we use Inception V3 (Szegedy al., 2016), one of the sfate-of-the- art 2D CNN models for image clas- sification. For extracting the features from the Charades dataset, we use the pre-trained model and hyper-parameters given in PySlowFast. For the TaCOS and Wetlab datasets, we used a Inception V3 model pre-trained on the ImageNet data set and then retrained it for each dataset.
Deep Random Fields (DRFs). As a baseline, we used a model that combines MRFs with CNNs. This DRF model is similar to the DDN except that we use a MRF instead of a DDN to compute P(x(e). We trained the MRFs generatively, namely we learned joint distribution P(x, e), which can be used to compute P(x(e) by instantiating evidence. We chose generative learning because we learned the structure of the MRFs from data and discriminative structure learning is slow in practice (cf. Koller & Friedman| (2009). Specifically, we used the logistic regression with e regularization method Wainwright et al. (2006) to learn a pairwise MRF The training data for this method is obtained by sending each annotated video clip (or frame) (v,x) x) through the CNN and transforming it to (e, x) where e = N(v). At termination, this method yields a graph G defined over X K.E.
For parameter/weight learning, we converted each edge over X U E to a conjunctive feature. For example, if the method learns an edge between X2 and Ej. we use a conjunctive feature X, A E, which is true iff both X; and Ej are assigned the value 1. Then we learned the weights for each conjunctive feature by maximizing the pseudo log-likelihood of the data (Besag 1975).
For inference over MRFs, we used Gibbs sampling (GS), Iterative Join Graph Propagation (UGP) Mateescu et al. (2010) and Integer Linear Programming (ILP) methods. Thus, we used three versions of DRIs corresponding the inference scheme used. We refer these schemes as DRF-GS, DRF- UGP and DRF-ILP respectively. Note that both IJGP and ILP are advanced schemes, and we are not aware of their use for multi-label action classification. Our goal is to test whether advanced inference schemes help improve the performance of deep random fields.
Deep Dependency Networks (DDNs). We experimented with four versions of DDNs: (1) DDN- LR-Pipeline; (2) DDN-MLP-Pipplin
(3) DDN-LR-Joine and (4) DDN-MLP-Joint, The first and the third versions use logistic regression (LR) while the second and fourth versions use multi-layer perceptrons (MLP) respectively to represent the conditional distributions. The first two versions are trained using the pipeline method while the last two versions are trained using the joint learning loss given in equation 3
Hyperparameters: For DRFs, in order to learn a sparse structure (using the logistic regression with & regularization method of Wainwright et al. (2006)), we increased the regularization constant associated with the & regularization term until the number of neighbors of each node in ς is bounded between 2 and 10. We enforced this sparsity constraint in order ensure that the inference schemes (specifically, IJGP and ILP) are accurate, and the model does not overfit the training data. IJGP, ILP and GS are anytime methods and for each, we used a time-bound of 60 seconds per example.
For DDNs, we used LR with l regularization and we tried MLPs with 3 and 4 hidden layers. We selected the MLP with 4 layers since it performed better. The regularization constants for LR
7

---

Under review as a conference paper at ICLR 2023
Table I: Comparison of all methods for Charades dataset
mAP
LRAP JI
CNN Feichtenhofer et al.
2019)
0.3884
0.5347 0.2937
DRF GS
0.2654
0.4386 0.2244
DRF ILP
0.1926
0.2782 0.3063
DRF IJGP
0.3115 0.4370 0.3194
DDN LR -ipeline
0.3450 0.4837 0.2897
DDN LR Jooint
0.3902 0.5479 0.3025
DDN MLP -Pipeline
0.3755 0.5493 0.2950
DDN MLP- -Joint
0.4052 0.5696 0.3356
Table 2: Comparison of all methods for TACoS and Wetlab datasets
TACoS
Wetlab
mAp
LRAP
SA
JI
mAP
LRAP
SA
JI
CNN (Szegedy et al. 2016)
0.7012 0.8076 0.4017
0.6079
0.7911
0.8208
0.3526
0,6383
DRF- GS
0.5584
0.7940 0.4693 0.6499
0.5392
0.7567
0.3532
0.5154
DRF ILP
0.4026 0.6720 0.5094 0.6471
0.6304
0.7339
0.5967
0.7271
DRF LJGP
0.5609 0.8141 0.4387
0.7010
0.7875
0.8525
0.5801
0.7366
DDN LR Pipplline
0.7164 0.8264 0.5042
0.6717
0.775
0.8549
0.5734
0.7018
DDN LR Joint
0.7463 0.8390 0.5371 0.6861
0.8432
0.869
0.6343
0.779
DDN MIPP Pipeline
0.7294
0.8602
0.5785 0.6948
0.8116
0.8819
0.6184
0.7267
DDN MLP -Joint
0.7797
0,8752 0.5963 0.7040
0.8809
0.8969
0.697
0.7925
(chosen from the set (0.1, 0.01, 0.001}) and the number of layers for MLPs were chosen via cross- validation. For joint learning, we chose the value of (see equation 3) via validatiinn and reduced the learning rates of both LR and MLP models by expanding on the learning rate scheduler given PySlowFast (Fan et al. 2020). We used the steps with relative learning rate policy, where the learning rate depended on the current epoch, and was chosen in the range of [le-3. 3, le - 5]. We have provided the apeerpaaramters used to do the experimentation the configuration files (which is provided with the supplementary material).
4.3 RESULTS
We compare the baseline CNN with three versions of DRFs and four versions of DDNs using the four metrics and three datasets given in section 4.1 The results presented in tables and
Comparison between Baseline CNN and DRFs. We observe that IJPP and ILP outperform the baseline CNN the two non probaabbaiiiiitic metrics JI and SA. IJGP typically outperforms GS and ILP on JI while ILP, because it performs accurate maximum-a-posterinn IJG and GS on SA (notice that SA if there exact match between predicted and true labels and 0 otherwise and thus accurate MAP inference on an accurate model is likely to yield high SA). However, on metrics which require estimating the posterior probabilities, mAP and IRAP, the DRF schemes sometimes hurt and at other times are marginally better than the baseline CNN. We observe that advanced inference schemes, particularly IJGP and ILP are superior on average GS.
inference, outperforms
Comparison between Baseline CNN and DDNs. We observe that the best performing DDN model, DDN-MLP with joint learning outperforms the baseline CNN on all the three datasets and on all metrics. Sometimes the improvement is substantial (e.g., 9% improvment in mAP on the wetlab dataset). Roughly speaking, the MI.P versions are superior to the LR versions, and the joint models superior to the ones trained using the pipeline method (in some cases because of joint learn- ing the LR models outperform the pipeline MLP models). On the non-probabilistic metrics, the pipeline models often superior the baseline CNN while on the mAP metric they may hurt the performance.
Comparison between DRFs and DDNs. We observe that the jointly trained DDNs almost always superior to the best performing DRFs on all datasets. Interestingly, on average, the pipeline DDN models outperform the DRF models when performance measured using the mAP and LRAP
8

---

Under review as a conference paper at ICLR 2023
metrics. However, when the SA and JI metrics are used, observe that there is no significant difference in performance between pipeline DDNs and DRFs.
In summary, jointly trained deep dependency networks are superior the baseline neural networks as well as networks that combine Markov random fields and neural networks. This clearly demon- strates the practical usefulness of our proposed method.
RELATED WORK
A large number of methods have been proposed that train PGMs and NNs jointly. For example, Zheng et al. (2013) proposed to combine conditional random fields (CRFs) and recurrent neural net- works (RNNs). SSchwing & Urtasun (2015) (100::arree Larsson et al. (017 2018); AAmab al. (2016) showed how to combine CNNs and CRFs, Chen et al. (2015) proposed to use densely connected graph- ical models with CNNs, and Johnson et al. (2016) combined latent graphical models with neural networks. As far as we know, ours is the first work that shows how to jointly train a dependency net- work, neural network hybrid. Another virtue of DDNs is that they are easy to train and parallelize, making them attractive choice.
The combination of PGMs and NNs has been applied to improve performance on a wide variety of real-world tasks. Notable examples include Semantic Segmentation (Arnab al.. (018: Guo &
Dou 2021). 1 human pose estimation (Tompson et al. 20144 [Liang et al. 2018] Song et al. 2017 Yang
al. 2016), semantic labeling of body parts (Kirillov et al. 20T6), stereo estimation Knöbelreiter
it al. (2017) language understanding (Yao et al.) 20144). joint intent detection and slot filling (Xu
& Sarikaya 2013), 2D Hand-poss Estimation (Kong et al, 2019), depth estimation from a single monocular image (Liu et al. 2015), Polyphonic Piano Music Transcription (Sigtia et al.) 2016), face sketch synthesis Zhu et al. (2021). Sea Ice Floe Segmentation (Nagi et al,] (021)) 1) and Crowdsourcing aggregation (Li et al.) 202 ID.. As far as we know, ours is the first work that uses jointly trained PGM+NN combinations to solve the multi-label action classification task in videos.
To date, dependency networks have been used to solve various tasks such as collective classifica- tion (Neville & Jensen 2003). binary classification (Gámez et al. 2006: 2008), multi-label classi- fication (Guo & Gul [011] D), part-of-specch tagging (Tarantola & Blanc, 2002), relation prediction
(Figueiredo et al. 2021)) and collaborative filtering (Heckerman et al. 2000). Ours is the first work
that combines DNs with sophisticated feature representations and performs joint training over these representations.
6 CONCLUSION AND FUTURE WORK
More and more state-of-the-art methods for challenging applications of computer vision tasks usu- ally use deep neural networks. Deep neural networks are good at extracting features in vision tasks like image classification, object detection, image segmentation, and others. Nevertheless, for more complex tasks involving multi-label classification, these methods cannot model crucial informa- tion like inter-label dependencies. In this paper, proposed a modeling framework called deep dependency networks (DDNs) that combines a dependency network with a neural network and demonstrated via experiments three video datasets that it outperforms the baseline neural net- work, sometimes by a substantial margin. The key advantage of DDNs is that they explicitly model and reason about the relationship between the labels, and often improve model performance without considerable overhead. In particular, DDNs are simple to use, admit fast learning and inference, are easy to parallelize, and can leverage modern GPU architectures.
Avenues for future work include: applying the setup described in the paper to multi-label classifica- tion, new tasks in computer vision, natural language understanding and speech recognition; devel- oping advanced inference schemes for dependency networks; converting deep dependency networks Markov random fields for better inference (cf. (Lowd) 2012)): etc.
REFERENCES
Alessandro Antonucci, Giorgio Corani, Denis Deratani Mauá, and Sandra Gabaglio. An ensemble of bayesian networks for multilabel classification. In Third international joint conference

---

Under review as a conference paper at ICLR 2023
artificial intelligence, 2013.
Anurag Arna,, Sadeep Jayasumana, Shuai Zheng, and Philip H. S. Torr. Higher order conditional random fields in deep neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 524 540, Cham, 2016. Springer International Publish- ing.
Anurag Arna,, Shuai Zheng. Sadcep Jayasumana. Bernardino Romera-Paredes, Mäns Larsson, Alexander Kirillov, Bogdan Savchynskyy. Carsten Rother, Fredrik Kahl, and Philip H.S. Torr. Conditional Random Fields Meet Deep Neural Networks for Semantic Segmentation: Combin- ing Probabilistic Graphical Models with Deep Learning for Structured Prediction. IEEE Signal Processing Magazine, 35(1):37-5,, January 2018. ISSN 1558-0792. doi: 10.1109/MPP200 2762355.
J. Besag. Statistical Analysis of Non-Lattice Data. The Statistician, 24:179-195, 1975.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Liang- Chieh Chen, Alexander Schwing, Alan Yuille, and Raquel Urtasun. Learning deep structured models. In International Conference on Machine Learning, pp. 1785-1794. PMLR, 2015.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 (4):834-848, April 2018. ISSN 0162-8828, 2160-9292. doi: 2..109/TAAM..201....000
Xianjie Chen and Alan I. Yuille. Artieulated Pose Estimation by Graphical Model with Image De- pendent Pairwise Relations. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Jeff Craley, Emily Johnson, and Archana Venkataraman. Integrating Convolutional Neural Networks and Probabilistic Graphical Modeling for Epileptic Seizure Detection in Multichannel EEG. In Albert C. S. Chung, James C. Gee, Paul A. Yushkevich, and Siqi Bao (eds.). Information Process- ing in Medical Imaging, volume 1 14492, pp. 291-303. Springer International Publishing, Cham, 2019. ISBN 778-3-030-20350- 978- 778-3-030-20351-11 doi: 10.
22. Se-
ries Title: Lecture Notes in Computer Science.
Nicola Di Mauro, Antonio Vergari, and Floriana Esposito. Multi-label classification with cutset net- works. In Alessandro Antonucci, Giorgio Corani, and Cassio Polpo Campos (eds.), Proceedings of the Eighth International Conference Probabilistic Graphical Models, volume 52 of Pro- ceedings of Machine Learning Research, pp. 147-158, Lugano, Switzerland, 06-09 Sep 2016. PMLR.
Haoqi Fan, Yanghao Li, Bo Xiong, Wan- Wan Lo,, and Christoph Feichtenhofer. Pyslowfast, https: //github. com htt.://aaocoooerrrchh.o.co www.aarchhss..w..........oo....e ssswfast] 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 6202-6211. October 2019.
Leticia Freire de Figueiredo, Aline Paes, and Gerson Zaverucha. Transfer learning for boosted rela- tional dependency networks through genetic algorithm. In International Conference on Inductive Logic Programming, pp. 139. Springer, 2021.
José A Gámez, Juan L Mateo, and José M Puerta. Dependency networks based classifiers: learning models by using independence test. In Third European Workshop on Probabilistica Graphical Models (PGM06), pp. 1 122. Citeseer, 2006.
José A Gámez, Juan Matco, Thomas Dyhre Niclsen, and José M Puerta. Robust classification using mixtures of dependency networks. In Proceedings of the Fourth European Workshop on Probabilistic Graphical Models, pp. 129-136, 2008
Qian Guo and Quansheng Dou. Semantic Image Segmentation based on SegNetWithCRFs. Proce- dia Computer Science, 187:300-306, 2021. ISSN 18770509. doi: 10. pross.2.. 1.0..666.
10

---

Under review as a conference paper at ICLR 2023
Yuhong Guo and Suicheng Gu. Multi-label classification using conditional dependency networks. In Twenty-Secone International Joint Conference on Artificial Intelligence, pp. 1300-1305, 2011.
Yuhong Guo and Wei Xue. Probabilistic multi-label classification with sparse feature learning. In
Proceedings of the Twenty-Thire International Joint Conference on Artificial Intelligence, IJCAI '13 pp. 1573-1379, AAAI Press, 2013. ISBN 9781577356332.
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2022. URL https::/ /ttt::://ww..
gurobi.comm com
Kang Han, Wanggen Wan, Haiyan Yao, Hou, and School of Communication and Informa- tion Engincering, Shanghai University Institute of Smart City, Shanghai University 99 Shangda Road, BaoShan District, Shanghai 200444, China. Image Crowd Counting Using Convolu- tional Neural Network and Markov Random Field. Journal of Advanced Computational Intel- ligence and Intelligent Informaties, 21 July 2017. ISSN 1883-8014, 1343-0130. doi: 20099///jaciii.211.......o
David Heckerman, David Maxwell Chickering, Christopher Meek, Robert Rounthwaite, and Carl Kadie. Dependency networks for inference, collaborative filtering, and data visualization. Journal of Machine Learning Research, 1(Oct):49755, 2000.
Matthew James Johnson, David Duvenaud, Alexander B Wiltschko, Sandeep R Datta, and Ryan P Adams. Composing graphical models with neural networks for structured representations and fast inference. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 2954-2962, 2016.
Alexander Kirillov, Dmitrij Schlesinger, Shuai Zheng, Bogdan Savchynskyy, Philip H. S. Torr, and Carsten Rother. Joint Training of Generic CNN-CRF Models with Stochastic Optimization, 2016.
Patrick Knöbelreiter, Christian Reinbacher, Alexander Shekhovtsov, and Thomas Pock. End-to-end training of hybrid cnn-crf models for stereo. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1456-1465, 2017. doi: 10.1109/CVP.2201.115)
D. Koller and N. Friedman. Probabilistic Graphical Models: Principles and Techniques. Adaptive computation and machine learning. MIT Press, 2009. ISBN 9780262013192.
Deying Kong, Yifei Chen, Haoyu Ma, Xiangyi Yan, and Xiaohui Xie. Adaptive graphical model network for 2d handpose estimation. In BMVC, 2019.
Xiangnan Kong, Bokai Cao, and Philip S. Yu. Multi-label classification by mining label and instance correlations from heterogeneous information networks. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 624-622, Chicago Illi- nois USA, August 2013. ACM. ISBN 978- 978-1-4503-2174-7. doi: 10. 1411144477775.2877777
Rahul G Krishnan, Uri Shalit, and David Sontag. Deep kalman filters. stat, 1050:25, 2015.
Mâns Larsson, Jennifer Alvén, and Fredrik Kahl. Max-Margin Learning of Deep Structured Models for Semantic Segmentation. In Puneet Sharma and Filippo Maria Bianchi (eds.), Image Analysis, Lecture Notes in Computer Science, pp. 28 40, Cham, 2017. Springer International Publishing. ISBN 978-3-319-591299 doi: 10.
Måns Larsson, Anurag Amab, Fredrik Kahl, Shuai Zheng, and Philip Torr. A Projected Gradient De- scent Method for CRF Inference allowing End-To-Emt Training of Arbitrary Pairwise Potentials. Technical Report arXiv: 1711. 106005, arXiv, January 2018. arXiv: 1000000055 [cs] type: article.
Su-in Lee, Varun Ganapathi, and Daphne Koller. Efficient structure learning markov networks using 1 L--regularization. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
Shao- Yuan Li. Sheng-Jun Huang, and Songcan Chen. Crowdsourcing aggregation with deep Bayesian learning. Science China Information Sciences, 64(3):130104, March 2021. ISSN 1674- 733X, 1869-1919. doi: 10.1007/s1 1110077/1114222000-111817

---

Under review as a conference paper at ICLR 2023
Guoqiang Liang, Xuguang Lan, Jiang Wang, Jianji Wang. and Nanning Zheng. A Limb-Base Bumb-Based Graphical Model for Human Pose Estimation. IEEE Transactions on Systems, Man, and Cy- bernetics: Systems, 48(7):1080- 2639788. Conference Name: IEEE Transactions on Systems, Man, and Cybernetics: Systems.
July 2018. ISSN 1168-2232. doi: 10.1109/TSMC.2016
Guosheng Lin, Chunhua Shen, Anton Van Den Hengel, and Ian Reid. Efficient piecewise training of deep structured models for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3194-32442 3203, 2016.
Fayao Liu, Chunhua Shen, and Guosheng Lin. Deep convolutional neural fields for depth estima- tion from a single image. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5162-5170, 2015.
J.S. Liu. Monte Carlo strategies in scientific computing. Springer Verlag, New York, Berlin, Hei- delberg, 2008 ISBN 0-387-95230-6.
Daniel Lowd. Closed-form learning of markov networks from dependency networks. In Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence, pp. 533-542, 2012.
Robert Mateescu, Kalev Kask, Vibhav Gogate, and Rina Dechter. Join-graph propagation algo- rithms. Journal of Artificial Intelligence Research, 37:279-3255 2010.
Anmol Sharan Nagi, Devinder Kumar, Daniel Sola, and K. Andrea Scott. RUF: Effective Sea Ice Floe Segmentation Using End-to-Enn End RES-UNNET-CR CRF with Dual Loss. Remote Sensing, 13(13): 2460, June 2021. ISSN 2072-4292. doi: 10.3390/rs 33132460.
Iftekhar Naim, Young Song, Qiguang Liu, Henry Kautz, Jiebo Luo, and Daniel Gildea. Unsuper- vised alignment of natural language instructions with video segments. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1), Jun. 2014. doi: 00.1609/aai..2281..888
Iftekhar Naim, Young C. Song, Qiguang Liu, Liang Huang, Henry Kautz, Jiebo Luo, and Daniel Gildea. Diseriminative unsupervised alignment of natural language instructions with corre- sponding video segments. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 164-174, Denver, Colorado, May-June 2015. Association for Computational Linguistics. doi: 10.3115//1/N15-1017.
Jennifer Neville and David Jensen. Collective classification with relational dependency networks. In Workshop on Multi-Relationn Data Mining (MRDM-2003), pp. 2003.
Hoang D. Nguyen, Xuan-Son Vu, and Duc-Trong Le. Modular Graph Transformer Networks for Multi-Label Image Classification. Proceedings of the AAAI Conference on Artificial Intelligence 35(10):9022-9100, May 2021. ISSN 2374-3468, 2159-5399. doi: 10.1609/aaii. v35i10.
Christina Papagiannopoulou. Grigorios Tsoumakas, and Ioannis Tsamardinos. Discovering and Exploiting Deterministic Label Relationships in Multi-Label Learning. In Proceedings of the 2 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp- 915-924, Sydney NSW Australia, August 2015. ACM. ISBN 978- 1-5-003---6664222 doi: 10. 145/ 2783258.2783302.
Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. Transactions of the Association for Computa- tional Linguistics (TACL), 1:25-36, 2013.
Alexander G. Schwing and Raquel Urtasun. Fully Connected Deep Structured Networks. Technical report, arXiv, March 2015.
Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon. An End-to-Enn Neural Network for Poly- phonic Piano Music Transcription. IEEE/ACM Transactions Audio, Speech, and Language Processing, 24(5):927-955 939,, May 2016. ISSN 2329-9304. doi: 10.1 Conference Name: IEEE/ACM Transactions Audio, Speech, and Language Processing.
12

---

Under review as a conference paper at ICLR 2023
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference Computer Vision, pp. 510-526. Springer, 2016.
Jie Song, Limin Wang, Luc Van Gool, and Otmar Hilliges. Thin-slicing network: A deep structured model for pose estimation in videos. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5563-5572, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978-1-4673-887 doi: 10. 10.1109/CVP..2206...00
Mingkui Tan, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Junbin Gao, Fuyuan Hu, and Zhen Zhang. Learning graph structure for multi-label image classification via clique generation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 4100-4109, June 2015.
C Tarantola and E Blanc. Dependency networks and bayesian networks for web mining. WIT Transactions on Information and Communication Technologies, 28, 2002.
Jonathan Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of convolutional network and a graphical model for human pose estimation. CoRR, waos/1446.29900 2984, 2014.
Martin J Wainwright, John Lafferty, and Pradeep Ravikumar. High- High-dimensional graphical model selection using & reguullar logistic regression. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
Hongning Wang, Minlie Huang, and Xiaoyan Zhu. A generative probabilistic model for multi-label classification. In 2008 Eighth IEEE International Conference on Data Mining, pp. 628-637.
IEEE, 2008
Lichen Wang, Yunyu Liu, Hang Di, Can Qin, Gan Sun, and Yun Fu. Semi-supervised dual relation learning for multi-label classification. IEEE Transactions Image Processing, 30:9125-9135, 2021a.
Ran Wang, Robert Ridley, Xi'ao Su, Weiguang Qu, and Xinyu Dai. A novel reasoning mechanism for multi-label text classification. Information Processing & Management, 58(2):002441, March 2021b. ISSN 03064573. doi: 10. 10. htpm.2020.1022.
Shangfei Wang, Jun Wang, Zhaoyu Wang, and Qiang Ji. Enhancing multi-label classification by modeling dependencies among labels. Pattern Recognition, 47(10):3405 3413, October 2014. ISSN 00313203. doi: 10. 101101 patcog.2014.044.
Anqi Wu, Estefany Kelly Buchanan, Matthew Whiteway, Michael Schartner, Guido Meijer, Jean- Paul Noel, Erica Rodriguez, Claire Everett, Amy Norovich, Evan Schaffer, Necli Mishra, C. Daniel Salzman, Dora Angelaki, Andrés Bendesky, The International Brain Laboratory The International Brain Laboratory, John P Cunningham, and Liam Paninski. Deep graph pose: semi-supervised deep graphical model for improved animal pose tracking. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 6040-6052. Curran Associates, Inc., 2020.
Puyang Xu and Ruhi Sarikaya. Convolutional neural network based triangular CRF for joint intent detection and slot filling. In 2013 IEEE Workshop on Automatic Speech Recognition and Under- standing, pp. 78 83, Olomoue, Czech Republic, December 2013. IEEE. ISBN 978- 100-1-1-999-55555 2. doi: 10. A0R1109//SRU..OO 2U11
Wei Yang, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang. End-to-Eine Learning of Deformable Mixture of Parts and Deep Convolutional Neural Networks for Human Pose Estimation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3073-3082, June 2016. doi: 10. 109/CVPR.2000..335.. ISSN: 1063-6919.
13

---

Under review as a conference paper at ICLR 2023
Kaisheng Yao, Baolin Peng, Geoffrey Zweig, Dong Yu, Xiaolong Li, and Feng Gao. Recurrent conditional random field for language understanding. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4077-4081, Florence, Italy, May 2014. doi: 10.1 109/CASSP 4.6854368.
IEEE. ISBN
Jonathan Yedidia, William Freeman, and Yair Weiss. Generalized Belief Propagation. In Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
Dongran Yu, Bo Yang, Qianhao Wei, Anchen Li, and Shirui Pan. probabilistic graphical model based on neural-symbolic reasoning for visual relationship detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10609-10618, 2022.
Mingjin Zhang, Nannan Wang. Yunsong Li, and Xinbo Gao. Neural Probabilistic Graphical Model for Face Sketch Synthesis. IEEE Transactions on Neural Networks and Learming Systems, 31(7): 2623-2637, July 2020. ISSN 2162-237X, 2162-2388. doi: 10.1109/TNNIS, 2019.29339900
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Da- long Du, Chang Huang, and Philip H. S. Torr. Conditional Random Fields Recurrent Neural Networks. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1529-1537, Santiago, Chile, December 2015. IEEE, ISBN 97--1-4773--83311122 doi: 10.1109//CCV22015.179
Mingrui Zhu, Jie Li, Nannan Wang, and Xinbo Gao. Learning Deep Patch representation for Proba- bilistic Graphical Model-Basee Face Sketch Synthesis. International Journal of Computer Vision, 12966)1820-18886 June 2021. ISSN 0920-5691, 1573-1405.
doi: 10. 1007/s1 10077s1116600 1263-021-01442-2
A APPENDIX
A.1 COMPARING TRAINING AND INFERENCE TIME
Table 3: Time Comparisons for the Proposed methods. Learning Time is in Hours and Inference
times is in seconds. Inference was performed on a CPU while training was performed on a GPU.
Tacos
Wetlab
Charades
Train
Inference
Train
Inference
Train
Inference
DRF GS
~ 5 hrs
0.58 sec
6.5 hrs
0.61 sec
~ hrs
~ 1.93 sec
DRF ILP
~ 5 hrs
1.54 sec
6.5 hrs
1.46 sec
9 hrs
2.42 sec
DRF LJGP
~ hrs
2.31 sec
6.5 hrs
~ 2.15
hrs
5.8 sec
DDN LR Pipeline
~ 1.5 hrs
~ 0.1 sec
~ 2 hrs
~ 0.15 sec
~ hrs
~ 0.39 sec
DDN LR Joint
hrs
0.1 sec
7.25 hrs
0.15 sec
hrs
0.39 sec
DDN MLP- Pipeline
2.25 hrs
0.19 sec
~ 3 hrs
0.31
4.25 hrs
0.58 sec
DDN MI.P-Joim
~ hrs
0.19 sec
~ hrs
~
0.31 sec
~ 14.5 hrs
~ 0.58 sec
Let us compare different methods based on the information given in table
A.1.1 COMPARISON AMONG DRFs
The learning time for the DRFs remains the same across the different methods, because use the same models and apply different inference techniques on them. But for inference time, we can see that use more sophisticated methods, the inference time goes up.
A.1.2 COMPARISON AMONG DDNs
For DDNs the inference times remain the same for both, pipeline and joint models. But for learning we see that joint model takes more time than the pipeline model. This is due to the fact that we are jointly learning the NN and the DN, and the inclusion of NNs for learning drives the time up.
14

---

Under review as a conference paper at ICLR 2023
A.1.3 COMPARISON BETWEEN DRFS AND DDNS
The learning time for pipeline DDN models are significantly less than that of the DRF model. Both the pipeline model and the joint model are exceptionally faster than the DRF methods, These two observations confirm our comments that DDNs are very fast and thus can be used in real time.
15


Reviews:
[{"review_id": "md8yYYrnuF-", "reviewer": "Reviewer_KzJi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple joint training deep dependency network, which can be built upon CNN baseline models to improve the performance in multi-label action classification tasks. The authors conduct experiments on three open datasets (Charades, TACoS, and Wetlab) and show their superiority on four metrics (mAP, LRAP, SA, and JI)", "summary_of_the_review": "The work lack novelty and convincing experiment detail, also the paper is hard to understand and the illustration is full of assumption and conceptual description.", "strength_and_weaknesses": "Strength\n+ Well motivated in Introduction\n\nWeaknesses\n- Unclear diagram, it is hard to figure out the joint learning methodology and how the LOSS is designed according to the figure.\n- Hard to understand due to typo/unclaimed terminology (e.g., 'ds outperforms CNN' in Preliminaries). Meanwhile, the advantages claimed in this work are mainly explained on conceptual/abstract induction or assumption instead of convincing analysis (e.g., 'An issue with the loss function just described is that it significantly changes the structure and semantics of the output nodes of the neural network').\n- Experiments lack a detailed description of author-design modules (even with an unclear setting like 'MLPs with 3 or 4 hidden layers), which makes the experimental procedure unconvincing. \n- For some parts of the experiment results, baseline like Slowfast seems to have higher performance in public disclosure compared with the effect provided here, which is confusing.\n- The authors state the work is a method for action classification. However, the majority of the work lies on optimizing the correlation of feature-label and label-label, while the adjustment for the action classification task is not significant; in other words, I prefer to categorize the work as an application of the previous general method other than insightful research in the scope of action recognition. Therefore, the novelty is relatively limited, in my opinion. Also, if the work is mainly effective in solving the 'MLAC problem, the title of the paper is too big for the work done.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lZA-Eiw2TD", "reviewer": "Reviewer_apzy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work experiments with hybrid modeling with CNN and PGM for the task of multi-label action classification.\nThis work considers (1) `CNN` and (2) `CNN + MRF + GS/ILP/IJGP` as two baselines.\nThis work proposes an approach called `Deep Dependency Networks` to yield better posterior probability estimates.\nExperiments show the proposed approach output performs baseline (1) and (2) in terms multi-label action classification metrics.", "summary_of_the_review": "Overall I like the presentation of the problem motivation and the proposed solution. However the baselines used in experiments are not up to date. It's unclear whether the performance gain hold against more recent deep learning models.", "strength_and_weaknesses": "Strength\n + This work noticed a particularly interesting problem on the simple implementation of `CNN + MRF + GS/ILP/IJGP` for multi-label action classification, i.e., they yield poor posterior probability estimates.\n + The proposed approach surely addresses the discovered problem and outperforms the aforementioned baseline.\n\nWeakness\n + the baselines used in experiments are out of date. On Charades dataset, the CNN baseline is from 2019. On TACoS and Wetlab, the baseline is from 2016. While the experiments w.r.t. these baselines can provide some proof of usefulness of the proposed method. It is unclear if this performance gain can hold against more recent deep learning models.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MOnpoYjm6l", "reviewer": "Reviewer_rXD3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple approach that combines the strengths of probabilistic graphical models and deep learning architectures for solving the multi-label video classification task. This work proves that jointly learning the proposed model can yield significant improvements in performance over the baseline neural network. They do experiments on three video datasets: Charades, Textually Annotated Cooking Scenes (TACoS), and Wetlab, showing that deep dependency networks are almost always superior to pure neural architectures that do not use dependency networks.\n", "summary_of_the_review": "This paper proposes a new hybrid model called DDNs, combining the strengths of dependency networks.", "strength_and_weaknesses": "Strength:\n- DDNs are simple and easy to use, which is GPU friendly.\n\n- DDNs can model and reason about the relationship between the labels.\n\nWeakness:\n- The transformer model can also be regarded as modeling the graph between tokens. I suggest the author should compare with transformer-based models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject