Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Update**
I have read the author's rebuttal, and happy to see that a discussion regarding parameters is added (Figure 9). Other than that, my personal concern is similar to Anon Review 3's -- it seems that the core idea of the paper is drowned in too many technical details (granted, many of these are needed in order to implement this correctly). I wonder if a clearer discussion can be made like this -- you have a variational inference problem with certain independence assumptions, so write this out in the most abstract manner possible. To come up with a concrete objective, the question then becomes "given the factor graph, how do we add the networks (this basically lines 11 - 16 in Algorithm but not in the flow of the main text)". I think a better presentation and clarity in the main paper would greatly help acceptance.

**Overview**
The paper proposes AutoBayes, which enumerates all the plausible graphical models between data, label, and nuisance variables, remove redundant edges with d-separation rules, and learns neural networks to represent the parent-to-child information.

**Strengths**
Empirical results are very strong for certain EEG datasets such as ErrP and RSVP. It appears that on the EEG datasets, different graph structures would have very different performances so there might not be a graph structure that works equally well on all of them, which motivates the need for searching for the optimal graph structure.

**Weaknesses**

Scalability: unlike existing methods in AutoML, AutoBayes does not seem to attempt to optimize the process from which graphs are selected (i.e. pruning of graphs that are unlikely to work well), resulting in the need to enumerate all possible graphs (where the complexity is doubly-exponential with respect to the number of variables). This means that the method will have difficulty even scaling to a small amount of variables (e.g. 10).

Interpretation of the learned structures: it seems that on some datasets, CVAE already provides comparable performance to the best AutoBayes architecture, and on others best AutoBayes architecture perform much better. Can we extract any insight from the AutoBayes procedure?

Fairness of experiments: for each component of the graph, the network structure is the same; therefore, compared to a structure X->Z->Y, we have fewer parameters if we use X->Y. It is unclear as to whether the empirical improvement can be gained simply by representing edges with larger networks.

Empirical comparison: it appears that model ensembles have a significant effect over the performance. However, since one can also ensemble models from different initializations but the same architecture, it is unclear what the performance gap would become if we also do ensemble on one model but with different parameters.

**Minor suggestion**
Perhaps spend some text describing what is special about EEG datasets, and why do we expect having to predict nuisance variables to improve performance.
review_title: Enumerating all possible graphs can achieve better performance on EEG datasets, but with potential issues on scalability.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 
The paper presents AutoBayes: a new approach for nuisance-robust deep learning which explores different Bayesian graph models to search for the best inference strategy. It automatically builds connections between classifier, encoder, decoder, nuisance estimator and adversary DNN blocks. The approach also enables disentangling the learned representations in terms of nuisance variation and task labels. Different benchmark datasets have been used for evaluation.
 
##################################################################

Strenghts:
- The idea of automatically exploring various graphical models to select the best performing one is interesting.  
- Overall, the paper is well written and easy to read.
- Ensemble learning is performed by stacking the explored graphical models allowing to improve performance.

##################################################################

Weaknesses:
- My main concern about the paper is that even though quantitative evaluation has been performed on several datasets with different modalities to show the benefit of using AutoBayes, the experimental evaluation section is not convincing enough as it lacks interpretation and analysis of the obtained results. For instance, one major problem addressed in this paper is models robustness to nuisance factors, however this was not discussed in this section. Hence, it would be good to include an experimental evaluation on this point. 
- In Table 4 in Appendix, the simple Model B which assumes independence between Z and S performs remarkably well on task classification of the 5 first datasets since it outperforms state-of-the-art methods on all of them except QMNIST and achieves classification accuracies that are very close to the best ones (with either variational or non-variational inference), outperforming most of the presented graphical models. On the remaining datasets, Model A which is independent of S and Z, performs well compared to other graphical models especially on Faces Basics and Faces Noisy datasets. Considering these observations and perhaps the time consumption of AutoBayes, what would be the motivation of exploring all the presented graphical models (besides stacking them for ensemble learning) rather than using the simplest models A or B with good hyperparameter search? Could these results be explained by the fact that potentially some of these datasets do not present high nuisance variabilities?
- One of the main contributions as presented in Section 2 is the extensibility of the proposed framework to multiple latent representations and nuisance factors. Although this was demonstrated theoretically, it would be interesting to demonstrate this experimentally.
- One of the properties of AutoBayes is its ability to learn disentangled representations in terms of nuisance factors. In practice, how can this be evaluated?

Minor comments: 
- Typo in Equation 6: xˆ  = p_μ(z1, z2) instead of xˆ  = p_μ(z1)?
- Steps to derive Equation 7 are not straightforward and can be more clarified.

review_title: Interesting idea but further motivations and experimental discussions are needed.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a framework, called AutoBayes, to automatically
detect the conditional relationship between data features (X), task
labels (Y), nuisance variation labels (S), and potential latent
variables (Z) in DNN architectures.  Assuming a Bayesian network (BN)
which represents the (possibly) conditional independencies between the
aforementioned variables, the authors propose a learning algorithm
which consists of applying Bayes-ball to detect and prune unnecessary
edges in the graph (effectively finding a subgraph, independence map
of the BN), train the resulting DNN architecture, and choose the network
which achieves the highest validation performance.  This idea is
interesting, especially compared to hyperparameter optimization
approaches for model tuning, and the results seem convincing.

However, relevant previous work is not cited and discussed in the paper.
Specifically, BN structure learning and inference in BNs (both of
which are well studied and have extensive literature) are fully
relevant, but are not discussed or mentioned at all.  For instance,
the paper uses undefined terms such as "Bayesian graph model," "Bayesian
graphs," and "graph model," in place of Bayesian network (which is
rigorously defined).  It is important that such related previous work
be discussed to delineate what is novel in the presented approach and
place its contributions within the greater context of this previous
work.  This inclusion would also help the presentation of concepts in
the paper.  For instance, the discussion surrounding equations 1
and 2, i.e.:
"The chain rule can yield the following factorization for a generative
model from Y to X (note that at most 4! factorization orders exist
including useless ones such as reverse direction from X to Y )...," is
the concept of an elimination ordering in the elimination
algorithm for BNs (and graphical models in general).  Showcasing the
presented work in this light, (i.e., as a natural
combination of BN structure learning with macro-level neural-architecture
optimization) would be particularly novel and compelling.

Finally, it is important to discuss the complexity of the presented
algorithm.  Given the Bayesian networks (BNs) in Algorithm 1,  each
independence map (and the underlying DNN
architecture) must be trained then validated.  This algorithm scales
factorially in the number of nodes in the BN.  It is great that the
selected subgraph performs so well (Figure 2), but super-exponential
complexity multiplied by DNN cross-validation training is going to be
very hard to do as m and n grow in Algorithm 1.

Other comments:

-The definition of nuisance and nuisance-variables are implicitly
assumed throughout the paper.  An exact definition of what is meant by
nuisance, in the context, of the paper would be very helpful.

-Algorithm 1 is mostly one large block of text, and is very hard to
parse on the first read.

-In the main contributions enumerated from pages 2-3, contribution 3
looks redundant given contribution 1.

-"Besides fully-supervised training, AutoBayes can automatically build some relevant graphi-
cal models suited for semi-supervised learning." <- please include a
link to where this is discussed.  The enumerated list of contributions
would be a perfect roadmap for the paper (just include references to
sections after every contribution)

-"We note that this paper relates to some existing literature... as addressed in Appendix A.1. Nonetheless, AutoBayes
is a novel framework that diverges from AutoML, which is mostly employed to architecture tuning
at a micro level.  Our work focuses on exploring neural architectures at a macro level, which is not
an arbitrary diversion, but a necessary interlude." <- Appendix A.1
should really be included in the
paper, related work is a not an optional section.  For instance, the
reader may not know what the authors mean in terms of micro versus
macro level.  Reading this without further explanation until later
in the paper, it would
seem that micro-level is more nuanced than macro-level and the former
perhaps subsumes the latter; the authors should detail what they mean
and distinguish their work from previous works in the main paper.

-The terminology is very clumsy: "Bayesian graph models," "Bayesian
graph," and "graph model" are not established terms and, as such,
should be defined so the reader knows what type of ML method is being discussed.  The authors should specify that they have a Bayesian
network whose factorization describes the conditional relationship
between (random) variables.

-"VAE Evidence Lower Bound (ELBO) concept" <- please include citation
for the ELBO

-"How to handle the exponentially growing search space of possible
Bayesian graphs along with the number of random variables remains a
challenging future work." <- this is exactly structure learning in
Bayesian networks (see the Bayesian information criterion, i.e., BIC score).
review_title: Very interesting idea and results, but relevant previous work regarding Bayesian networks and structure learning is not discussed at all.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors present a novel method dubbed AutoBayes that tries to find optimal Bayesian graph models for "nuisance-robust" deep learning. They employ the Bayes-Ball algorithm to construct reasonable inference graphs from a generative model given by iterative search. The corresponding DNN modules are then built/linked and trained using a form of variational inference with adversarial regularization where applicable. The authors also propose the use of an ensembling approach to further improve robustness of the "best" model.

Pros:
- Interesting and novel approach
- Experimental results are encouraging and seem to demonstrate a clear advantage of AutoBayes over other methods

Cons:
- The organization of the paper is erratic; overall structure doesn't flow well.
- Multiple figures have details (e.g. color or style of arrows, shading, etc.) which are never explained.
- Little to no context is given for what nuisance-robust learning (and correspondingly, nuisance "variation" variables) actually is. Similarly for the datasets and baselines.
- No discussion of time/space complexity or computational demands, which is surprising given the apparent combinatorial complexity of the nested for-loops in Algorithm 1.
- Placement with respect to existing work is somewhat vague (the authors refer to "similarities" and "relationships" but do not concretely describe them).
- The paper needs to be proof read several more times. There are numerous grammatical errors and poorly phrased sentences. I will give a few examples below, but this is largely up to the authors to sort out.

Overall, the paper reads more like a rough draft of a technical report than a standalone research article. It's confusing, difficult to read (I found that I needed to jump around and re-read a lot in order to understand what the authors were trying to say), and fails to give necessary context to readers who aren't familiar with a very specific subset of the deep learning literature. While it's of course always fine to defer to references for details, it's important that the reader can broadly understand your method and the problem it's trying to solve without hunting down a dozen references. This paper seems to assume that the reader has read every relevant paper and is arriving at this work in sequence.

If the authors are willing to do some reorganizing and more diligent proof-reading, I'd be happy to reconsider my rating after seeing the revisions.

A couple of examples of poorly phrased or grammatically incorrect sentences:

"It may be because the probabilistic ~relation~ *relationships* *in the* underlying data ~varies~ vary ~over~ across datasets" (pg 2)

"The ~whole~ DNN blocks are trained with ~adversary~ *adversarial* learning ~in a~ *using* variational Bayesian inference." (pg.3)
review_title: Interesting idea, encouraging results,  but poorly organized paper

Meta-review: This work proposes a method for identifying appropriate graphical models through enumeration, pruning of redundant dependencies, and neural network conditionals. While structure learning is an interesting application and there are some promising results, there were a number of concerns around experimental evaluation, computational complexity of the method, clarity of the presentation, and connections to prior work. In particular, R1's concerns around the large field of structure learning in Bayesian Networks, and unwillingness to use the established terminology (and comparing to methods there) was not sufficiently addressed in the rebuttal.
Decision: Reject