Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple approach that combines the strengths of probabilistic graphical models and deep learning architectures for solving the multi-label video classification task. This work proves that jointly learning the proposed model can yield significant improvements in performance over the baseline neural network. They do experiments on three video datasets: Charades, Textually Annotated Cooking Scenes (TACoS), and Wetlab, showing that deep dependency networks are almost always superior to pure neural architectures that do not use dependency networks.

strength_and_weaknesses: Strength:
- DDNs are simple and easy to use, which is GPU friendly.

- DDNs can model and reason about the relationship between the labels.

Weakness:
- The transformer model can also be regarded as modeling the graph between tokens. I suggest the author should compare with transformer-based models.
clarity,_quality,_novelty_and_reproducibility: I am satisfied with the clarity, quality, and novelty of this paper.
summary_of_the_review: This paper proposes a new hybrid model called DDNs, combining the strengths of dependency networks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work experiments with hybrid modeling with CNN and PGM for the task of multi-label action classification.
This work considers (1) `CNN` and (2) `CNN + MRF + GS/ILP/IJGP` as two baselines.
This work proposes an approach called `Deep Dependency Networks` to yield better posterior probability estimates.
Experiments show the proposed approach output performs baseline (1) and (2) in terms multi-label action classification metrics.
strength_and_weaknesses: Strength
 + This work noticed a particularly interesting problem on the simple implementation of `CNN + MRF + GS/ILP/IJGP` for multi-label action classification, i.e., they yield poor posterior probability estimates.
 + The proposed approach surely addresses the discovered problem and outperforms the aforementioned baseline.

Weakness
 + the baselines used in experiments are out of date. On Charades dataset, the CNN baseline is from 2019. On TACoS and Wetlab, the baseline is from 2016. While the experiments w.r.t. these baselines can provide some proof of usefulness of the proposed method. It is unclear if this performance gain can hold against more recent deep learning models.

clarity,_quality,_novelty_and_reproducibility: Strength
 + Overall clear presentation of motivation, design, experiments, and results

Weakness
  + In the best performing setting (`DDN-MLP-Joint`), is the network architecture the same between `CNN` and `DDN-MLP-Joint`? Basically a CNN for extracting features and a MLP for mapping features to multi-class predictions. What's different between the two is only the losses applied, is this understanding correct?
  + need to remove a comment at the beginning of Section 2: "So just a quick question ......"
summary_of_the_review: Overall I like the presentation of the problem motivation and the proposed solution. However the baselines used in experiments are not up to date. It's unclear whether the performance gain hold against more recent deep learning models.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a simple joint training deep dependency network, which can be built upon CNN baseline models to improve the performance in multi-label action classification tasks. The authors conduct experiments on three open datasets (Charades, TACoS, and Wetlab) and show their superiority on four metrics (mAP, LRAP, SA, and JI)
strength_and_weaknesses: Strength
+ Well motivated in Introduction

Weaknesses
- Unclear diagram, it is hard to figure out the joint learning methodology and how the LOSS is designed according to the figure.
- Hard to understand due to typo/unclaimed terminology (e.g., 'ds outperforms CNN' in Preliminaries). Meanwhile, the advantages claimed in this work are mainly explained on conceptual/abstract induction or assumption instead of convincing analysis (e.g., 'An issue with the loss function just described is that it significantly changes the structure and semantics of the output nodes of the neural network').
- Experiments lack a detailed description of author-design modules (even with an unclear setting like 'MLPs with 3 or 4 hidden layers), which makes the experimental procedure unconvincing. 
- For some parts of the experiment results, baseline like Slowfast seems to have higher performance in public disclosure compared with the effect provided here, which is confusing.
- The authors state the work is a method for action classification. However, the majority of the work lies on optimizing the correlation of feature-label and label-label, while the adjustment for the action classification task is not significant; in other words, I prefer to categorize the work as an application of the previous general method other than insightful research in the scope of action recognition. Therefore, the novelty is relatively limited, in my opinion. Also, if the work is mainly effective in solving the 'MLAC problem, the title of the paper is too big for the work done.

clarity,_quality,_novelty_and_reproducibility: Clarity: poor; the paper is unclear and hard to understand
Quality: poor, 
Novelty: poor; the paper proposes a simple dependency network with a joint training method, which does not show significant novelty
Reproducibility: poor; the paper lack details about their design, and it is hard to reproduce if the authors do not release the code

summary_of_the_review: The work lack novelty and convincing experiment detail, also the paper is hard to understand and the illustration is full of assumption and conceptual description.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a novel approach which jointly trains a conditional dependency network and a deep neural network for multi-label action recognition in video. The idea is to use the DNN for feature extraction and the dependency network represents to reasons about the probabilistic relationships between the labels and the extracted features. Empirical evaluation is performed on Charades, Textually Annotated Cooking Scenes (TACoS), and Wetlab and show that the proposed method improves the results across multiple benchmarks.

The reviewers appreciated the motivation behind the problem, but raised several critical concerns. (1) Empirical evaluation against transformer-based models which essentially model the dependencies between tokens with an implicit graph. (2) Empirical evaluation with (severely) out-of-date baselines. (3) Clarity of writing, undefined terminology. (4) Limited scope and novelty in the current instantiation. (5) Worries about reproducibility given the lack of open-sourced code and other details. These concerns were not satisfyingly addressed in the rebuttal and I will hence recommend rejection.
justification_for_why_not_higher_score: - Several critical issues in the empirical evaluation
- Clarity, limited scope and novelty
- Worries about reproducibility given the lack of open-sourced code and other details
Decision: Reject