Unsupervised Non-Parametric Signal Separation Using Bayesian Neural Networks
Authors: Alexander V Belikov
Alessandro Montanari
Emmanuel Simon Moulin
Venue: Submitted to ICLR 2023 (2023)
Keywords: bayesian neural networks
probabilistic graphical models
signal disaggregation

Abstract: Bayesian neural networks (BNN) take the best from two worlds: the one of flexible and scalable neural networks and the one of probabilistic graphical models, the latter allowing for probabilistic interpretation of inference results. 
We make one extra step towards unification of these two domains and render BNN as an elementary unit of abstraction in the framework of probabilistic modeling, which allows us to promote well-known distributions to distribution fields. 
We use transformations to obtain field versions of several popular distributions and demonstrate the utility of our approach on the problem of signal/background separation.
Starting from prior knowledge that a certain region of space contains predominantly one of the components, in an unsupervised and non-parametric manner, we recover the representation of both previously unseen components as well as their proportions.

Under review as a conference paper at ICLR 2023
UNSUPERVISED NON-PARAMETRIC SIGNAL SEPARA- TION USING BAYESIAN NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian neural networks (BNN) take the best from worlds: the one of flexible and scalable neural networks and the one of probabilistic graphical models, the latter allowing for probabilistic interpretation of inference results. We make one extra step towards unification of these two domains and render BNN as an elemen- tary unit of abstraction the framework of probabilistic modeling, which allows us to promote well-known distributions to distribution fields. We use transforma- tions to obtaín field versions of several popular distributions and demonstrate the utility of our approach on the problem of signal/background separation. Starting from prior knowledge that a certain region of space contains predominantly one of the components, in an unsupervised and non-parametric manner, recover the representation both previously unseen components as well as their proportions.
INTRODUCTION
Neural networks predictive models have been wildly successful across variety of domains, be image recognition or language modeling. And while they may be used to make predictions on previously unseen samples, one of fundamental weaknesses of traditional neural networks is the inability to quantify the prediction uncertainty. Evaluation of prediction uncertainty is important in basic research (identification of fundamental laws), reinforcement learning (identification of value functions), anomaly detection, etc.
Uncertainty quantification in neural networks has been addressed both from frequentist (see, for instance, Pearce et al. (2018) and Bayesian (Kendall and Gal (2017) sides. In the Bayesian setting was naturally proposed to promote the weights of neural layers to normally distributed random variables (MacKay )192)). Later it was shown that the learnt uncertainty in the weights improves generalization in non-linear regression problems, and it can be applied to drive the exploration- exploitation trade-off in reinforcement learning (Blundell et al. (015)). [Depeweg al. (2017) designed a method of separation of uncertainty into epistemic and aleatoric. Epistemic uncertainty expresses uncertainties inherent to the model and can not be reduced with additional observations, whereas aleatoric uncertainty captures the amount of noise due to training on a specific sample. In physics the former and latter are referred as the systematic and statistical uncertainties, respectively.
In treating both types of uncertainty within the same framework authors essentially bridged the gap towards graphical models. Graphical models, unlike traditional neural networks, are probabilistic in nature and allow for incorporation of prior beliefs with respect to models. They are flexible in representing various processes and allow for introduction of latent degrees of freedom. Initially graphical models used various point distribution as building blocks, while mostly normal distribution has been promoted to random in the notable example of Gaussian random fields.
In this work we propose using Bayesian Neural Networks (BNN) as building blocks in graphical models and demonstrate the power of synthesis of Probabilistic Graphical Models (PGM) and BNNs on a synthetic example of signal/background separation. As demonstration of our approach we propose the additive mixture model: a superposition of signal and background spectra whose proportion varies in space. During inference we are able to learn the proportion of signal and background and their spectral shapes that match ground truth values to adequate precision. The paper organized as follows. In Section 2 we recapitulate the feed-forward (vanilla) BNN and the vanilla BNN that
variational inference approach. Section 3 we present the transformations allow to emulate various distribution fields. To illustrate the power of composition of transformed

---

Under review as a conference paper at ICLR 2023
BNNs in Section 4 we introduce a model of additive BNN mixture. In Section we describe experiments and in Section O we discuss the results and evaluate model performance. In Section 7 conclude by discussion of limitations well prospective domains of applications of our framework.
BACKGROUND: VANILLA BAYESIAN NEURAL NETWORK
We consider feed-forward deep neural architectures that are composed of dense layers. A dense layer is an affine transformation 4 with weight W and bias B that followed by an element-wise non-linear transformation h L h==1 function. In our experiments we set a to be ReLU, defined as Re max(0, x). In what follows work with simple linear deep architecture which is defined consecutive application (composition) of dense layers: = L o ... L oz.
f + 1²* also known the activation
In order enable probabilistic interpretation of inference using neural networks, the weights and the biases of each layer are promoted to random variables and are sampled from a Normal distribution with corresponding parameters: W* N(uw, Ew), 8 N (ug, Eg). In Fig. | right panel depict an elementary Bayesian Neural Network, composed of l layers and that takes as input æ, consisting of N samples, and rendering y as output, using plate notation.
We consider simple BNN in the spirit of (Blundell et al. (2015)). where authors use stochastic variational inference (SVI) (Hoffman et al. ( Wingate and Weber (2013) for Gaussian posterior distributions from prior distributions of weights, biases and observations. Under these conditions it
natural to use Evidence Lower Bound (ELBO) (Mehta et al. (2019) as the loss function. ELBO loss consists of two terms: log evidence of the observable variable with learnable parameters 0, log pa (I), and the Kullback-Leibler (KL) divergence between the approximation of the posterior distribution (2),, parametrized by d, and the true posterior pe (2(ar):
ELBO log po (a) KL. (gs (c>(=))ppoz((a))
Taking steps in increase ELBO, increases log evidence and decreases the distance between the prior and the posterior. We further illustrate this in Fig. [ left. Inference results depend on the choice of the optimizer, the learning rate and the number of iterations.
3 TRANSFORMED BNNS
Non-trivial examples of probabilistic models combine distributions of various types. Consider a K K-component Gaussian Mixture model: each component of the mixture is normally distributed, where the mean parameterized by real-valued parameters and the scale by a positive parameter, while the overall proportions are sampled from Dirichlet distribution X₃ Dir(a), which, in turn, is parameterized by positive vector a1.... > 0, and X₁ belong to a K - simplex: Σx+
We are therefore motivated to introduce family of transformed BNNs with various ranges. In this manuscript we consider exponential transformation (transforms unconstrained vector of K dimensions to positive vector of dimensions) and a stick breaking transformation (unconstrained vector of K dimensions into a simplex vector of K dimensions).
We propose to apply transformations after the last layer of BNN, such a way that the range output is constrained to be strictly positive for the exponential transform and a k-dimensional vector summing to unity (k-simplex) for the stick breaking transformation.
In what follows we denote vector sampled from Bayesian neural network as y BNN(Z ()) We denote BNN outputs transformed by exponential and stick- biick-br trans- forms BNN, and BNN,, respectively.
Another type of BNN transformation we consider is prompted by probability distributions: in certain applications it is particularly useful work not just with positive random fields but with normalized positive random fields. Practically such a transformation consists of an approximate normalization of the BNN output given the data.
2

---

Under review as a conference paper at ICLR 2023
Pw
Ew PB
MB
N
W
B
2
x
h
y
K
N
1
Figure Left panel: Graphical model representation of variational inference. Observed random variable is denoted latent variables that generate z via P(z|z) are Prior distribution of is parametrized by 0: P(z (),, , the true distribution of approximated by posterior P(zØø). The functional form P==lØ) may be different from P(z (it is important that P(zl)) is differentiable with respect to Φ). Sampling 2 from P(z and P(= allows to evaluate KL divergence term, quantifying distance between them. Mcanwhile sampled from P(z(ø) is used to evaluate data evidence. In the context of probabilistic programming language Pyro P(z(0) is part of the "model" and is denoted p=)=), P(= ()) is part of the "guide" and is denoted (=)... Right panel: Graphical model representation of an clementary BNN. Shaded circles represent observable variables (input and output y). empty circles latent variables (W, B and fi), standalone letters thyper-paaameten (Uw, dw. and Eg). Directed arrows represent dependence between the starting and terminal vertices, black squares with N represent sampling from a normal distribution. Internal plate of dimension K represents K BNN layers and plate of dimension N represents the size of the data sample.
0g
0.4
03
8
(xt, p)
022
L
Lg
311
30
y
X
N
Figure 2: Left panel: graphical model representation of additive mixture model. Shaded circles represent observable variables (input (x, p). outputs y and relative entropy r), empty circles latent variables associated with BNNs / and g. standalone letters hyper-ppaaae paaeerr-rramterrs of BNNs (ás. Φg). Directed arrows represent dependence between the starting and terminal vertices. black square represents the convolution given by Eq. 2 Internal plate of dimension N represents conditional independence of the data sample. Plates Lg and Ly represent conditional independence of parameter sampling of 3 and g BNNs. Right panel: a sample of 5000 datapoints representing the mixture f(. p) For negative values the total signal is dominated by the background component 91 (p). while positive values of the signal and the background are in superposition.
ADDITIVE MIXTURE
Having introduced architectures with single dense unit BNN, turn to non-trivial test of our framework: a generative additive mixture model. In this model we consider two types of coordinates: refer to as the spatial coordinate and p as the spectral coordinate.

---

Under review as a conference paper at ICLR 2023
We consider two positive spectral functions 91 (p) and (2(p) that are mixed in spatially dependent manner by a comple--valuee (zx), and suppose that only their sum f(z, p) is observed:
f(z,p) p) = ((x))) (p) + (1 3(z)))2(P).
(2)
This model is motivated by the problem of identification of spectrum of signal g2(p) in the presence of non-trivial noise 91 (p). We assume that (p()) are normalized: f dpgi (p) 1. It follows immediately that f()) = f dp f(; p) is equal to I for each æ. Spectral functions g (p) are positive-valued and
we model them by an exponentially transformed N,., We model ((x) as a Dirichlet random field ((x) Dir(ay())), where (2) = (x) and y BNN (x) Setick-breaking transformed BNN). In order to deal with degeneracy due to the permutation symmetry: 91 ++ 92 and 3 ++ 1 B we set the initial value of y/ to correspond to an asymmetric proportion. e.g (0.99, 0.01), where the first component represents the background, while the other the signal. To further discourage the inference engine from splitting the observed signal between components uniformly we introduce another observable, the relative entropy between spectral components 91 (p) and g2(p)
f (n(
(xi(O 91 (p))
and set it to a large number, e.g.. 100.
The first observed term forces the combination of (p) and ((x) to approximate y:. The term containing relative entropy forces to learn maximally different 9i (p) and g2()), since DKI(92(P) In (p)) is a proxy to distance in function space.
The objective of the inference is to identify optimal model parameters dg and 0g and thus to obtain the full spatial and spectral description of the mixture, i.e., the shapes of (p) and the mixing proportion (zz) In the current formulation the additive mixture model does not include Bayesian treatment aleatoric uncertainty, observations are sampled using a small fixed variance (0.002).
5 EXPERIMENTS
BNN abstractions and experiments were implemented Pyro (version 1.5.2) (Bingham al. (019)). ) a probabilistic programming language (PPL) written in Python and based on pytorch, which enables Bayesian probabilistic modeling thanks to Monte Carlo and variational inference engines. In our experiments we use unit BNNs containing 3 hidden layers of dimensions 32 x 128 x 32.
Inference. We use Pyro's 's stochastic variational inference (SVI) abstraction which computes ELBO loss and take steps in the space of "guide" parameters & along the gradients of loss function. As a whole the procedure is Bayesian update: it identifies variational parameters á of the true posterior approximation (z).). In what follows use Clipped Adaptive Moment Estimation (ADAM) (Kingma and Ba (0144) as the optimization method and, unless mentioned otherwise, set the learning rate to 10 the clipping norm to 10, and default values for the coefficients used for computing rumning averages of gradient and its square: (0.9, 0.999).
The value of the ELBO loss is computed for each epoch. The relative values of the loss with respect to the previous iterations is computed too. For a chosen window, the former are compared with two conditions.
We improve the inference procedure in an empiric manner by restarting it with a decreasing value of learning rate (factor of 0.5 in case), while keeping current "guide" parameters. The restart happens if (i) the last Sp steps resulted in an increased loss or (ii) the last S, steps resulted in a relative loss below a certain threshold. Both S, and S, are chosen to be 3 in the current project. Condition (i) means that the inference engine is diverging from optimal region of the parameter space (likely due to the stochastic nature of sampling). which manifests the divergence of loss function. This procedure is repeated until fixed number of steps are performed.
Training of unit BNNs. Unit vanilla and transformed BNNs introduced in Section may be used to approximate functions or as latent constituents of more complex models. When using them as latent parts of more complex models it often advantageous to impose certain priors on their functional
Implementation availahle at
4

---

Under review as a conference paper at ICLR 2023
form, g.g., might want to represent the prior knowledge of mixture composition, which is not directly observed.
In other words latent components of complex model can be pre-trained so that their priors condition the output to have the desired shape. If the prior and the "guide" have the same functional form, this procedure can be viewed as Bayesian update. Using learned during pre-training "guide" parameters to initialize the "guide" when a given unit BNN is used as a constituent may improve convergence.
In short we go through the following steps: (i) generate data sample emulating desired shape, (ii) fit a unit BNN to the generated data sample using non-informative priors, thus obtaining "guide" parameters, and (iii) use inferred guide parameters as prior parameters and (optionally) as the initial "guide" parameters as part of more complex model. The second step involves sampling BNN output match observations. Since we are not interested in aleatoric component this stage, we represent it as constant and use normal, log-normal and Dirichlet for sampling.
We illustrate the process of inference of a unit BNN on the example of approximating ("())
e--pa 1, shifted by b -2.5 (and normalized to 1). To fit this positive function use exponentially transformed BNN,, which is sampled using log-normal distribution after the last layer by convention. The inferred posterior distribution of the inference on a unit BNN N is shown in the left panel, the loss in the second-to-left, the location parameters μ of weights W of the last hidden layer of given BNN a function of epoch the second-to-right and the evolution of scale parameters E in the right panel of Fig. 3 The inferred posterior distribution matches well the true values within the 68% containment bands of the model.
600m
02001
02001
Figure 3: Left panel: true versus predicted values from a unit BNN, Mean value is shown as solid line and the 68% containment bands as the shaded area. Solid circle represent sampled true values. Second-to-lefi panel: ELBO loss function as function of training epoch for a unit BN N. Second- to-right panel: posterior location hyperparameters of the weight distribution W for 32 neurons of the last hidden layer of a unit BNN as function of training epoch n. The location parameters of W are initialized to zeros in the beginning the weight distribution W for 32 neurons of the last hidden layer of a unit BNN N as function of training epoch n. Scales are initialized to 0.1 in the beginning of SB inference.
SB inference. Right panel: posterior scale hyperparameters of
Synthetic Additive Mixture. By considering the additive mixture model answer the following fundamental question: if superposition of two spectral (i.e. positive) functions is observed as function of spatial coordinates, can we learn their shapes and proportions. given a minimum amount of information? To test our method for the additive mixture model we create a synthetic dataset containing triples = { Pt. f
N N. 22 E R. Pa E R. where I; and pt are sampled uniformly on [-5, 5] and refer to this particular implementation as the Signal/Background (SB) model. The values of [I,,, pt) are derived using the generative model of Sec. 4 with deterministic gr (p) and deterministic ((x). In what follows we refer to gi (p) and ga (p) as the background and the signal, respectively. We consider (i) case A where non con-onooreteeve priors are used, and (ii) case B where only 91 (p) is conditioned. For both of them we 1), which we shifted by b = 2.5 in two directions: 91 (p) = g" (p b) and g2 (p) = g" (p++1) 6).
previously introduced template g (p) (normalized to
First demonstrate that the inference works for specific choice of 3: choose 3(x) to be logistic function with values at -oo equal to 0.99 and too equal to 0.5. Afterwards we test the limit of our approach by considering synthetic datasets with decreasingly small proportion of "signal".
3(z) and (p) are represented by a stick- breckkbne and an exponentially transformed BNNs (see Sec. hereafter referred to BNN and BN Ng, respectively. Both of them have output dimensions
5

---

Under review as a conference paper at ICLR 2023
equal to 2. The prior distributions for (W, B) of all hidden layers of BNN, and BNN taken to be 0.10.
is instructive to represent BNN W, graphically as function of of where W, B are sampled from prior distributions: in Fig. 5 the mean values are plotted as dashed lines together with the 68% containment bands shaded areas for case A and case B in the top and bottom panel, respectively.
8.34
646
BNN,
BNH, non informnotaiive priors
non Informative prioss
For atn- priors
514
6re
200
200
358
400
300
125
200
351
10
200
epoch
opoch
epoch
3.100
BNN, consiitoned
BNN, conditioned g.
conditioned
3.000
35
200
opoch
opoch
opoch
Figure 4: Left panel: posterior location hyperparameters of the weight distribution W for 32 neurons of the last hidden layer of BN Na n as function training epoch n. The location parameters W are initialized to zeros in the beginning of SB inference. Middle panel: Posterior scale hyperparameters of the weight distribution W for neurons of last hidden layer of BN g as a function of training epoch. Scales initialized 0.1 in the beginning of SB inference. Right panel: ELBO loss function as a function of training epoch. For left, center and right panels top figures are given for the case non-informative prior (top panels) and the case of the prior, pre-trained on the background spectral shape 9ı (p) (bottom panels).
Location parameters of the W for the last hidden layer of BNN, as a function of epoch are shown in the left panel of Fig. 4 The scale parameters are shown in the middle panel of the same figure. The right panel shows the ELBO loss as function of epoch. The top panels show the results for case A and the bottom panels for case A. For the case of non-informative prior the variation of the means negligible after 700 epochs. the case where the 91 is conditioned, locations hyperparameters converge to their terminal values after 500 epochs.
RESULTS
In what follows, we set ((x) to vary continuously from 3712=-5 = 0.99 in the background-
dominated spatial region to 3 I 0.7 in the mixture region.
The inferred posterior distribution of 1-3(x) and (p) are shown in Fig. 5 for proportion in the mixture region of ( 0.7 for cases A and B in the upper and lower, left and second-to-lefi panels, respectively. The inferred posterior distribution for 1-3((z) matches well the true values within the 68% containment bands of the model for cases A and B.
For case B, the relative error between the true signal proportion at x = 5, i.e. 1-3T la and the mean proportion recovered by the model is lower than 8% and the prediction of g2 in agreement

---

Under review as a conference paper at ICLR 2023
12
da
022
0.4
0.0
5-
14
Figure 5: Predictions compared to true values of signal proportion 1 3 (left) and spectral shapes g for background and signal obtained from the SB inference, in blue and red respectively. True values of 1-=5 0.7 are assumed. Left panels: comparison between predictions and true values for the mixing signal proportion
(zx). The mean is shown as solid line and the 68% containment bands the shaded area. Solid circle represent sampled true values. Predictions from prior distributions are represented by their means (dashed line) and 68% containment bands (light shaded area). Second- to-left panels: comparison between predictions and true values for the spectral shapes (p). Second- to-right panels: model signal proportion model uncertainty extracted from the left panels. Right panels: model spectral shapes gd vs true spectral shapes of together with their 68% uncertainty extracted from the second-to to-left panels. Top and bottom panels show results for case A with non-informative priors and case B with conditioned gi (p), respectively.
BM vs the true proportion 1 8" together with 68%
with true values within 95% confidence level. For case A, the relative error between the true signal proportion and the mean proportion recovered by the model is lower than 9% but the ground truth values of 92 not contained in the 95% confidence band.
The second-to-right and right panels of Fig. 5 show the so- called y-y plot of predicted versus the ground truth values for the signal proportion 1-3 and the spectral shapes gi for the cases A and in the top and bottom panels, respectively. The uncertainties are given as 68% containment bands (shaded areas). The true values the proportion are recovered by the model at 68% containment level, in both cases. In case B, biases from 1% 5% are obtained for the model signal proportion. For the spectral shape g2. bias up 16% is obtained for g values larger than 0.1. I. In case A, biases from 1 1% to 10% obtained for the signal proportion. For the spectral shape G2. a bias up to 39% is obtained for g values larger than 0.1.
Bo
0.25
0.0
RMSD
0.35
25
0.30
0.00
P
Figure 6: RMSD for the comparison of the predictions after the SB infer- ence and the true values of 1-3 and gi the noise-to-signal ratio p.
We also run a series of experiments by changing 8 (xmm)
1-33(x=5) in the ground truth data, or, in other words, the ground truth background fraction value at the right boundary, which may be interpreted the noise-to-signal ratio. Each experiment is repeated several times with random seeds to help mapping the distribution. In Fig, 6 we plot the root mean squared deviation (RMSD) between the prediction of the corresponding constituent BNN after the SB inference and the true value of 1-3 and gl, respectively, as function of p. We note that the variation
7

---

Under review as a conference paper at ICLR 2023
of RMSD for 91 over the whole range of p is small. We also note that the error in identification of 92 and 1-3 increases with increase of p. potentially due to the fact that the inference procedure is no longer able to distinguish small signal from the fixed aleatoric noise.
7 DISCUSSION
In this work we introduced framework that merges the powers of Bayesian neural networks and graphical models and demonstrated its validity on a proof of concept model of signal/background separation. This framework allows to compose BNNs, representing random fields, in the same way point distributions are composed in graphical models.
Our proof of concept model is motivated by the astrophysical problem of learning of an unknown signal in the presence of a potentially unknown background, which manifests at various wavelengths. We lift this traditionally treated by parametric statistical methods and more recently using Bayesian statistics (see, for instance, AAddllah et al. (2016) 2018 2020); Abdalla et al. (202T)) problem, level of non-parametric inference. Our approach
thanks to the power of neural networks, is non non-poramae parametric with respect to proportions and spectral shapes and also Bayesian; it allows for Bayesian updates: to incorporate available extra information and aggregate spectral descriptions by class (study sources that we believe belong to the same class).
We outline several possible improvements and generalizations that will be addressed in future work.
At the heart of our framework of representing random fields by BNNs is a non-linear transformation of a neural network layer, that renders the output positive. According to Jensen inequality increasing the scales parametrizing BNN will lead to greater expectation value of the output (as well as hidden layers). This might lead to a potential problem during inference using ELBO loss, where matching expected values (of potentially latent random variables) might be achieved by increasing the scale parameter, instead of increasing the location parameter, at the same time rendering log probabilities for the sampled distribution irrelevant (as in any data fits equally well a very wide Gaussian).
From the point of view of physical modeling of signal/background separation, model is built upon several simplifications: does not include an explicit model of the mea- surement, nor does it contain microscopic emission model. It also posits space-energy factorization, that energy-spectra are spatially independent, which is not the case the target (gas, radiation) density fields are strongly spatially- -depeneet The gamma-ray signal expected from cosmic ray interaction in the interstellar medium would therefore exhibit dependent spectral behavior. All the above points maybe address within the current framework.
In SB model we considered the separation of total signal into two components in one spatial dimension. In astrophysics and cosmology raw observations are made in two spatial coordinates and time. Our framework allows very easily extend the number of spatial dimensions. Increasing the number of components on the hand should be done with caution as decreases the stability of inference due to appearance of extra permutation symmetries.
The additive mixture example was focused on identification of epistemic uncertainties. Aleatoric uncertainty may be represented as a parameter or another unit BNN.
SB model was demonstrated to apply in a limited signal- tonl-to--oiie ratio region. While this might be due to our choice of fixed aleatoric scale, it is possible that inference using ELBO loss becomes less stable, when smaller scales are chosen.
SB model may be directly applied in gamma-ray, cosmic-microwwve-bbcee www.woe--bbkkkuuddda and gravitational astrophysics. The framework, however, is far more general and could be used, in the context of Non Intrusive Load Monitoring (NILM), where the total household consumption signal is disaggregated into multiple house appliances and over-fitting problem is of particular importance (Jones al. (2020)). Other applications include Bayesian modeling of value function in reinforcement learning (Eriksson al. (2020)) and audio source separation (Schulze-Forse et al. 2002
8

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hassan Abdalla et al. Search for dark matter annihilation signals from unidentified Fermi-LAT objects with H.E.S.S. Astrophys. J., 918:17, 2021. doi: 10.3847/1538-4357/b1115,
Hassan Abdallah et Search for dark matter annihilations towards the inner Galactic halo from 10 years of observations with H.E.S.S. Phys. Rev. Lett., 117(11):111301, 2016. doi: 10.1 103/ PhysRevLett. 11..1tt1...1.113.....
Hassan Abdallah et al. Search for 2-Ray Line Signals from Dark Matter Annihilations in the Inner Galactic Halo from 10 Years of Observations with H.E.S.S. Phys. Rev. Lett., 120(20):201000 101, 2018. doi: 10. 101.100sseyylll..omm
120 101.
Hassan Abdallah et al. Search for dark matter signals towards a selection of recently detected DES dwarf galaxy satellites of the Milky Way with H.E.S.S. Phys. Rev. D, 102(6):06200,, 2020. doi: 10.1 1..110/PhysRevD.1222
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah D. Goodman. Pyro: Deep universal probabilistic programming. J. Mach. Learn. Res., 20:28:1 18:66 2019. URLhttp:
//jmlr. www.l.//gg/pappr////222/
403. .hmml
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, page 1613-1622. JMLR.org, 2015.
Stefan Depeweg, José Miguel Hernández-Lobatoo, Finale Doshi-Velez, and Steffen Udluft. Decompo- sition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning. arXiv e-prints, art. arXiv: 1IXiv:1710.0728: October 2017.
Hannes Eriksson, Emilio Jorge, Christos Dimitrakakis, Debabrota Basu, and Divya Grover. Inferential induction: A novel framework for bayesian reinforcement learning, 2020. URL https://
arxiv. org/ abs 03098
Matt Hoffman, David M. Blei, Chong Wang, and John Paisley, Stochastic variational inference, 2013.
Richard Jones, Christoph Klemenjak, Stephen Makonin, and Ivan V. Bajic. Exploring bayesian surprise to prevent overfitting and to predict model performance in non-intrusive load monitoring. www./rr.///b/......../.....ee.e/ 20099. 07756
2020. URL httpt: http:////wwwrrrrr//a///////s////
Alex Kendall and Yarin Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? arXiv e-prints, art.
March 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, art. arXiv:1411 141::441,.6980 December 2014.
David J. C. MacKay. A Practical Bayesian Framework for Backpropagation Networks. Neural Computatioon, 4(3):48-44 472, 05 1992. ISSN 7667. doi: htt.:///ncco.1992..3333 URL
https: www.o////.................
wwwpl//e/eecc.ooo neco. 1992. 4.3...444....................... 448
Pankaj Mehta, Marin Bukov, Ching Hao Wang, Alexandre G. R. Day, Clint Richardson, Charles K. Fisher, and David J. Schwab. A high-bias, low-variance introduction to Machine Learning for physicists. Physics Reports, 810:1-124, May 2019. doi: 10. phw.ei/jphyyrep2211..00
Tim Pearce, Mohamed Zaki, Alexandra Brintrup, and Andy Neely. Sigh-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach. arXiv e-prints, art. arXiv:180..07167,, February 2018.
Kilian Schulze-Forster Clement S. Doire, Gaël Richard, and Roland Badeau. Unsupervised audio source separation using differentiable parametric source models, 2022. URL https
/ Taaaxivv. /aaxiv/org/abs//201..co 09992
David Wingate and Theophane Weber. Automated Variational Inference in Probabilistic Programming. arXiv e-prints, art. arXiv: 1301. 1299, January 2013.


Reviews:
[{"review_id": "xp2v2WJ_x0M", "reviewer": "Reviewer_6vPR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use parametric transforms such as exponential function or stick-breaking transformation on the outputs of Bayesian neural network (BNN) and turn them into positive random vectors or simplex vectors. This new transformed BNN module can be plugged into probabilistic graphic models as a replacement prior. Sampling from transformed BNN is also possible, with the priors on the weights and biases. An additive mixture model is introduced for signal background separation with synthetic experiments. ", "summary_of_the_review": "Overall, I think this paper needs further development to be accepted, including both a more formal justification of the proposed framework, comparisons to other methods such as VAE or GPs models,  and experimental validations on real-world datasets. ", "strength_and_weaknesses": "It is an interesting idea to study combining Bayesian neural networks with probabilistic graphic models, and get the best of both worlds.  However, the current paper seems not in a form that can be accepted yet. The authors also quite frankly listed a few limitations of the current approach in Section 7. \n\nThere are many aspects that require further development, listed below,  \n\nFrom the probabilistic modeling point of view, it is unclear what is the benefit of bringing in the Bayesian neural network piece, compared to other conventional choices such as Dirichlet prior or Gamma vectors, or log-normal vectors.  The potential advantage might include being easier to incorporate covariates, with more flexible dependencies, and/or nonlinearity from neural networks, but this requires more supporting evidence.  \n\nThe model is tied to a variational model in Pyro, inference is conducted by maximizing the ELBO with an additional penalty term of relative entropy to encourage the separation of components. A key question unaddressed: how does the proposed framework compare to the standard VAE framework, with a probabilistic graphical model as the generative model, and NN or transformed NN as the recognition model? \n\nFrom the application point of view, it is unclear how the simple additive mixture model is compared to other choices of non-parametric models such as Gaussian processes.  The non-parametric aspect for unsupervised learning is especially interesting. Is it possible to explicitly show some differentiation achieved with the proposed approach? \n\nThe argument of building a random field rather than point distributions also needs more development, e.g., what can be achieved beyond Gaussian random fields?  Although a few potential applications are listed in Sec. 7., the paper would certainly benefit from evaluations on some real-world datasets. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UcaF5Dx-a8", "reviewer": "Reviewer_YQeZ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors incorporate Bayesian neural networks as non-parametric alternatives for standard distributions within a probabilistic graphical model, and demonstrate the approach in a toy example of separating two independent signals from a mixtures.", "summary_of_the_review": "Extended case study of how probabilistic graphical models can use neural networks within the model, assuming they are implemented using the standard libraries.", "strength_and_weaknesses": "Strengths:\n- The basic approach is sound and well in line with the modern literature where flexible function approximations are being used to expand standard model families\n\nWeaknesses:\n- The specific contribution is unclear and the paper looks more like a case study of a well-known principle, where the case being studied is also quite simple\n- The writing is a bit naive and known concepts (treating neural networks as probabilistic elements) are introduced as if they were new", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Tl1tOyytRGF", "reviewer": "Reviewer_CEDA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The incorporates Bayesian neural networks into probabilistic graphical models and apply their model to signal/noise decomposition.", "summary_of_the_review": "The paper tackles an interesting problem of combining data-driven models with probabilistic ones. The contribution of the paper is limited in that it makes strong assumptions on the probabilistic model and does not discuss how both aspects can be merged together effectively in a more general setting. Also, the authors seem to miss some important prior work on that topic.", "strength_and_weaknesses": "The considered problem of combining data-driven models with probabilistic models is clearly a timely one. Using Bayesian neural networks in that context promises to provide a seamless integration because of the joint probabilistic interpretation. While I would really like to see some fundamental progress being made in that direction, I am afraid that this paper does not do so. The main problem I see is that \nthe authors simply put those two parts together without accounting for the subtleties that usually arise when considering practical scenarios. This does not become too obvious as the paper makes some strong implicit assumptions as e.g., the consideration of simple mixture distributions with few components and experiments with well separable signal and background noise.\n\nThere are also some technical inaccuracies that are sloppy at best and might suggest that the authors did not put sufficient work into understanding the probabilistic graphical models framework at worst. For example, in Figure 1 it is written that the observed random variable is denoted as x with a latent variable z, i.e., we have the underlying data generating process $P_\\theta (x|z)$. Yet the left figure clearly corresponds to a data generating process of  $P_\\theta (y|z)$.\n\nSimilarly, in Figure 2 the relative entropy is described (and illustrated) as a random variable, although it is clearly a function thereof.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MI-io0JnzG6", "reviewer": "Reviewer_SrCA", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to add non-linear transformation to Bayesian neural networks and uses them in an additive mixture model. They should that this can solve signal separation problems on 2 synthetic tasks.", "summary_of_the_review": "The proposed method is not novel at all and the contribution is over-claimed. The empirical study is also solely based on simple simulated data.", "strength_and_weaknesses": "## Strength\n- The paper is easy to follow\n\n## Weakness\n- The techniques mentioned in the paper are not new and the contributions are over-claimed.\n    - Having a transformation in the output layer of neural network or for any probability distribution is a common technique used in the community. Both transformations used in the paper are standard.\n    - Additive mixture is another common trick, in both deep learning community and graphical modeling community.\n    - Inference method used (SVI) is also standard.\n- The experiment is too simple and if signal separation is the focus, some real-world datasets that used for benchmarking purpose should be considered. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject