Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The incorporates Bayesian neural networks into probabilistic graphical models and apply their model to signal/noise decomposition.
strength_and_weaknesses: The considered problem of combining data-driven models with probabilistic models is clearly a timely one. Using Bayesian neural networks in that context promises to provide a seamless integration because of the joint probabilistic interpretation. While I would really like to see some fundamental progress being made in that direction, I am afraid that this paper does not do so. The main problem I see is that 
the authors simply put those two parts together without accounting for the subtleties that usually arise when considering practical scenarios. This does not become too obvious as the paper makes some strong implicit assumptions as e.g., the consideration of simple mixture distributions with few components and experiments with well separable signal and background noise.

There are also some technical inaccuracies that are sloppy at best and might suggest that the authors did not put sufficient work into understanding the probabilistic graphical models framework at worst. For example, in Figure 1 it is written that the observed random variable is denoted as x with a latent variable z, i.e., we have the underlying data generating process $P_\theta (x|z)$. Yet the left figure clearly corresponds to a data generating process of  $P_\theta (y|z)$.

Similarly, in Figure 2 the relative entropy is described (and illustrated) as a random variable, although it is clearly a function thereof.
clarity,_quality,_novelty_and_reproducibility: The paper is well motivated and easy to follow. 
Yet it hides some detail that are important for understanding the contribution. In particular, it overpromises by stating to unify the domains of BNNs and PGMs, while elegantly omitting some of the underlying assumptions.
I understand that this is a hard problem for which it is generally not straightforward to minimize the KL divergence. At the very least, the paper could explain this and make an argument for their assumptions.

The paper also has a very slim list of references with respect to work that has been done on the intersection of those two domains. For example, it does never mention the work of [1] that considers a similar setting.

[1] Johnson, Matthew J., et al. "Composing graphical models with neural networks for structured representations and fast inference." Advances in neural information processing systems 29 (2016).
summary_of_the_review: The paper tackles an interesting problem of combining data-driven models with probabilistic ones. The contribution of the paper is limited in that it makes strong assumptions on the probabilistic model and does not discuss how both aspects can be merged together effectively in a more general setting. Also, the authors seem to miss some important prior work on that topic.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposes to add non-linear transformation to Bayesian neural networks and uses them in an additive mixture model. They should that this can solve signal separation problems on 2 synthetic tasks.
strength_and_weaknesses: ## Strength
- The paper is easy to follow

## Weakness
- The techniques mentioned in the paper are not new and the contributions are over-claimed.
    - Having a transformation in the output layer of neural network or for any probability distribution is a common technique used in the community. Both transformations used in the paper are standard.
    - Additive mixture is another common trick, in both deep learning community and graphical modeling community.
    - Inference method used (SVI) is also standard.
- The experiment is too simple and if signal separation is the focus, some real-world datasets that used for benchmarking purpose should be considered. 
clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is simple and clear.
- Quality: The method is too simple.
- Novelty: All methods mentioned in the paper are known in the literature.
- Reproducibility: The URL to GitHub doesn't work.
summary_of_the_review: The proposed method is not novel at all and the contribution is over-claimed. The empirical study is also solely based on simple simulated data.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors incorporate Bayesian neural networks as non-parametric alternatives for standard distributions within a probabilistic graphical model, and demonstrate the approach in a toy example of separating two independent signals from a mixtures.
strength_and_weaknesses: Strengths:
- The basic approach is sound and well in line with the modern literature where flexible function approximations are being used to expand standard model families

Weaknesses:
- The specific contribution is unclear and the paper looks more like a case study of a well-known principle, where the case being studied is also quite simple
- The writing is a bit naive and known concepts (treating neural networks as probabilistic elements) are introduced as if they were new
clarity,_quality,_novelty_and_reproducibility: The main idea of the paper seems to be replacing standard distributions within a PGM with Bayesian neural networks whose outputs satisfy the properties of a distribution, to obtain some sort of non-parametric alternatives for conditional distributions. This standard approach is described as a novel idea and it is decorated with some additional terminology ('transformed BNN' that seems to mean standard BNN with specific activation function for the output layer), but I still fail to see where the actual contribution of the paper lies. Probabilistic programming libraries, like Pyro that the authors also use, are specifically designed to make building this kind of models easy and using neural networks as function approximations within those models is a routine practice. The idea itself is hence by no means bad, but it also has no novelty and the basic idea of using neural networks within a PGM and variational inference dates at least back to 2005 (Harva et al. "Bayes blocks: an implementation of the variational Bayesian building blocks framework", UAI 2005). The case study in Section 4 is something most Pyro users should be able to write if given the task and the solution they would build is roughly the same that is proposed here (including even the choice of the inference method), to the extent that it could perhaps even be used as an exercise on a MSc level course.

The story is also a bit problematic. I think the authors are building towards a general-purpose approach, but in practice only present two very simple examples (stick-breaking and exponentiation) and then proceed to study a very simplified case example. The paper would be more interesting if more challenging examples (perhaps normalising flows?) were characterised as the building blocks and the case-study would be for a problem that is clearly challenging and could not be easily addressed with the standard tools.
summary_of_the_review: Extended case study of how probabilistic graphical models can use neural networks within the model, assuming they are implemented using the standard libraries.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to use parametric transforms such as exponential function or stick-breaking transformation on the outputs of Bayesian neural network (BNN) and turn them into positive random vectors or simplex vectors. This new transformed BNN module can be plugged into probabilistic graphic models as a replacement prior. Sampling from transformed BNN is also possible, with the priors on the weights and biases. An additive mixture model is introduced for signal background separation with synthetic experiments. 
strength_and_weaknesses: It is an interesting idea to study combining Bayesian neural networks with probabilistic graphic models, and get the best of both worlds.  However, the current paper seems not in a form that can be accepted yet. The authors also quite frankly listed a few limitations of the current approach in Section 7. 

There are many aspects that require further development, listed below,  

From the probabilistic modeling point of view, it is unclear what is the benefit of bringing in the Bayesian neural network piece, compared to other conventional choices such as Dirichlet prior or Gamma vectors, or log-normal vectors.  The potential advantage might include being easier to incorporate covariates, with more flexible dependencies, and/or nonlinearity from neural networks, but this requires more supporting evidence.  

The model is tied to a variational model in Pyro, inference is conducted by maximizing the ELBO with an additional penalty term of relative entropy to encourage the separation of components. A key question unaddressed: how does the proposed framework compare to the standard VAE framework, with a probabilistic graphical model as the generative model, and NN or transformed NN as the recognition model? 

From the application point of view, it is unclear how the simple additive mixture model is compared to other choices of non-parametric models such as Gaussian processes.  The non-parametric aspect for unsupervised learning is especially interesting. Is it possible to explicitly show some differentiation achieved with the proposed approach? 

The argument of building a random field rather than point distributions also needs more development, e.g., what can be achieved beyond Gaussian random fields?  Although a few potential applications are listed in Sec. 7., the paper would certainly benefit from evaluations on some real-world datasets. 

clarity,_quality,_novelty_and_reproducibility: 
Questions to clarify

1. In Figure 2, the relative entropy r is treated as an observed variable. I found this confusing. 
2. What is the observation model for y in the additive mixture model? At the end of Section 4, it says “the observations are sampled using a small fixed variance (0.002)”, but it would seem a little bit inconsistent if the intensities are assumed to be non-negative. 
3. How to balance the ELBO term and relative entropy term in the objectives? 
4. Is the citation to “gradient clipping” precise?
5. What exactly is the stick-breaking transformation used? Citations missing? Is there any implicit bias in constructing a simplex vector in this way? 
6. The "training of unit BNNs" part reads very confusing. "(i) generate a data sample emulating desired shapes (ii) fit a unit BNN...obtaining guide parameters (ii) using the "inferred guide parameters" as prior for more complex model". What are the motivation and benefits of these procedures? 

summary_of_the_review: Overall, I think this paper needs further development to be accepted, including both a more formal justification of the proposed framework, comparisons to other methods such as VAE or GPs models,  and experimental validations on real-world datasets.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper describes a type of Bayesian neural network and applied this model to signal/noise decomposition.

While reviewers acknowledged the practical application is interesting, however they also agree that the proposed approach has very little technical novelty, and the experimental results are not significant enough to be published at ICLR.

I agree with the reviewers and I recommend the authors to consider submitting to venues considering ML applications mainly.

justification_for_why_not_higher_score: Reviewers all favour strong rejects due to significant novelty and clarity issues.
Decision: Reject