Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update: after reading the feedback and discussing with the other reviewers, I decide to keep my score unchanged.

Original comments:
In this paper, the authors provide new interpretation of neural networks via chain graphs, which can be used as a new theoretical framework to understand the behavior of neural networks.

Pros.

1. Although both PMG and Neural networks are based on graphs, very little research has been proposed to bring the two fields together. This paper provides an interesting result to bridge the two previously unrelated fields, which equips the community with useful insights to spark new research directions.

2. This paper helps the community to fully utilize the existing works from PGM to solve current obstacles in deep learning.

3. The open questions are also interesting. I would be particular interested in the third question.

Overall, I think this is an interesting paper.
review_title: An interesting paper

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors look for ways to frame neural networks in terms of probabilistic models. This is a worthy goal and I believe the authors should pursue this path with enthusiasm. The theme is relevant, however the novel ideas that are in the paper seem to have relatively small significance. The translation from neural networks to chain graphs is not surprising; actually if I understand the translation we basically get a Bayesian network out of the directed acyclic graph encoding the connections in a neural network (I must say that the translation could be presented in a more didactic fashion. In any case, given this translation, the authors are able to frame some techniques from neural networks as techniques from chain graphs. But this feels too forced a translation, because neural networks do have a natural interpretation and usage as function approximators; why should one go all this ways to find a probabilistic translation that explains a few well known things, and opens the possibility of some probabilistic algorithms that could anyway be derived in the context of neural networks? One example: given the proposed translation, it is obvious that sequential models will appear as dynamic chain graphs, why is this useful in any way? 

On top of this, I find it troublesome that the authors "prove" results by saying things like "approximately linear" and "approximately" this and that. How can these results be proven given this level of informal justification. I really think this should be fixed before publication. 

Some sentences are a bit confusing. In Page 3, for instance, the authors say that a model with P(X|Pa(X)) is modeled by a CRF; usually a CRF models a discriminative model, is this the case here? What exactly is going on? Also the last paragraph of Section 3.2 is very hard to parse (the main point there is not clear at all). 
review_title: Promising idea but results are too simple and not adequately presented.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper tries to interpret neural networks with chain graphs that provides theoretical analysis on various neural network components. Furthermore, this chain graph interpretation has been used to propose a new approach (architecture), which is a partially collapsed feed-forward. A layered chain graph representation is adopted to formulate the neural networks with layered chain graphs. This further establishes to interpret feed-forward as an approximate probabilistic inference with using linear approximations. Some concrete examples are shown to be analyzed  based on the chain graph formulation. 

The overall context (analysis) seems straightforward to interpret the neural networks with chain graphs, but it is hard to achieve some meaningful information from this new interpretation to improve the current neural network models in terms of learning procedure or optimization.  The proposed partially collapsed feed-forward is a good example to come up with a new approach based on the chain graph interpretation. However, in terms of performance and complexity, it is practically not showing impressive improvements compared to the baseline methods. Moreover, it seems quite similar to previous works as far as I remember and one similar work is 'stochastic feedforward neural networks'. I fully agree the future works (open questions) in the conclusion and discussion section that this work still needs more investigations although this paper is a good initiative work.


review_title: alternative interpretation based on chain graph

final_score: rating: 3: Clear rejection
review_content: ###############################################################################

Summary

Summarize what the paper claims to contribute. Be positive and generous.

The authors propose an interpretation of feed-forward neural networks and recursive neural networks as chain graphs (CGs). They claim this new interpretation can provide novel theoretical support an insights to existing techniques, as well allow for the derivation of new approaches.

###############################################################################

Pros and cons

Pros:
 - the text is well-written
 - the attempt at studying neural networks under a graphical probabilistic model perspective is praisable

Cons:
 - the chain graph interpretation put forward by the authors is superfluous, as in reality the definition found in the apper is that of DAGs with particular parametric constraints
 - most of the results presented by the authors do not rely on the CG interpretation, and can already be found in the litterature
 - the paper is missing a connexion to stochastic feedforward neural networks (SFNNs), to which the proposed interpretation is extremely similar

###############################################################################

Recommendation

I recommend rejection of the paper, for two reasons. First, I believe most of the contributions proposed in the paper are not novel, as they can already be found in published form. See my detailed comments below. Second, the CG interpretation put forward by the authors is trivial and superfluous, since the authors in reality only consider CGs restricted to DAGs. As such, the whole claim of the paper that CGs can give a new, relevant perspective to NNs, is not credible.

###############################################################################

Questions to authors

I would like the authors to comment on the fact that their CG interpretation is indeed the LWF-CG interpretation, restricted to DAGs (singleton chain components). Furthermore, I would appreciate if the authors could relate their approach to the SFNN model, and what differs from that interpretation.

###############################################################################

Detailed comments:

p.1 §3: an efficient approximate probabilistic inference -> What is meant here by efficient ? Is it an unbiased estimator ? What is meant by inference ? Computing $p(y|x)$ ? $\arg \max_y p(y|x)$ ?

p.2 §2: The chain graph model -> There exists at least 4 chain graph interpretations. See
Drton, Mathias. „Discrete Chain Graph Models“. In: Bernoulli 15 (Sept. 2009),
pp. 736–753.
Among those, there are two which subsume UGs and DAGs, while the two others subsume BGs and DAGs. If you follow the CG interpretation from Koller and Friedman then you assume NNs are LWF-CGs.
I strongly suggest that you make it explicit which chain graph interpretation you follow, for the sake of clarity. These interpretations are not equivalent.

p.2 §2: there exists a series of works [...] -> I believe the list is much bigger than that. You forget very popular models which are direct instanciation of PGMs, such as VAEs, HMMs, LDAs, GMMs, CRFs, and all of their variants.

p.2 §4: an approximate probabilistic inference procedure -> What is meant by that ?

p.2 §4: provides additional insights -> such as ?

p.3 §2: layered chain graph -> A chain graph is always layered into chain components. The name "layered chain graph" is confusing, as it seems to imply that some chain graphs may not be layered. Since you are using the LWF-CG interpretation, you might point to the relevant papers where the graphical model was introduced, and simply re-use the establihsed LWF-CG name from the litterature, instead of inventing a new one.
Lauritzen, Steffen L. and Wermuth, N. „Graphical Models for Associations
between Variables, some of which are Qualitative and some Quantitative“. In:
The Annals of Statistics 17.1 (Mar. 1989), pp. 31–57
Frydenberg, M. (1990). The chain graph Markov property. Scand. J. Statist. 17 333–353. MR1096723

p.3 eq.1: This factorization is not that of a CG, but that of a DAG. Since you assume no undirected connection between variables in the same layer, then all chain component in your chain graph actually contain a single variable. What is the point of introducing the conecpt of CGs then, if you only consider CGs which are restricted to DAGs ?

p.3 eq.2: The LWF-CG interpretation, which you seem to follow since you refer to Koller and Friedman, does not imply this factorization for the graphs you consider. Since each chain component $K$ in your graph is a singleton, each $X_i$ and its parents form a clique in the closure graph $K$. As such, the CG structure does not imply any further factorization for p(x|pa(x))... It does seem your interpretation of NNs is not as general CGs, but as very specific DAGs with parametric constraints. Furthermore, you do not define what are $T$ and $f$ here.

p.3 Section 2.2: I believe your CG interpretation is simply a reformulation of stochastic neural networks, for which there already exists a great body of work. See, eg:

Eric Jang Shixiang Gu Ben Poole. Categorical Reparameterization with Gumbel-Softmax. ICLR (2017)

Yoshua Bengio, Nicholas Leonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.

Yichuan Tang, Ruslan Salakhutdinov. Learning Stochastic Feedforward Neural Networks. NIPS 2013

p.4 Proposition 2.1: This result seems very limited, since it assumes linear layers, and in the end a linear NN.

p.4 Corollary 2: What are alpha and beta here ? How do they relate to $f$, $e$ or $T$ from Definition 1 ? What are $e_i^l$ and $s_i^l$ here ?

p.4 §6: the modularity of chain components justifies transfer learning via partial reuse of pre-trained networks -> I do not understand this argument. What is meant here by "modularity" ?

p.4 §7: However, feed-forward is no longer applicable through these intra-connected layers. -> You finally give an example here that corresponds to non-trivial chain graphs (with chain components greater than 1), and recognize that your theoretical results do not apply any more. This contradicts your claim that CGs offer a general interpretation of NNs with theoretical support. I believe the CG interpretation is not justified, nor required to derive the results you present in this paper.

p.5 Definition 2: I fail to see the point of introducing this new concept, which is a re-definition of residual blocks.

p.5 §4: While a vanilla layered [...] -> I fail to understand where the CG interpretation fits in this argument.

p.5 Proposition 4: This result seems trivial to me, and does not require the concept of a CG. Once you show that a linear layer followed by a specific activation can be interpreted as a parametric model for $p(\textit{out}|\textit{in})$, then any NN that unrolls as an acyclic directed graph can be interpreted as a probabilistic model.

p.6 §1: The simple recurrent layer, [...] -> Again, I do not see where the CG interpretation gives any insight here.

p.6 Section 3.3: This result, again, does not require the CG interpretation. It is also already know. See, e.g., Pierre Baldi and Peter J. Sadowski. Understanding Dropout. NIPS 2013

p.6 Section 4: It seems to me you are reinventing SFFNs. See, Yichuan Tang and Ruslan Salakhutdinov. Stochastic Feedforward Neural Network. NIPS 2013

p.7 §2: the sampling operation is not differentiable -> This is not true. See, e.g.,  the reparameterization trick for VAEs. You mention this in the next sentence...
review_title: The paper fails to deliver on its promises

Meta-review: In this paper, the authors draw connections between probabilistic graphical models (specifically LWF chain graphs) and neural network models. There was general agreement amongst the reviewers that this is an interesting topic that merits further study, and would be of interest to the ICLR audience. At the same time, all of the reviewers have read the author response and there is a consensus that the novelty and significance of this work is limited. The connections between CGs and NNs are somewhat standard and well-known, and the significance of the results has not been convincingly demonstrated.

Decision: Reject