AutoBayes: Automated Bayesian Graph Exploration for Nuisance-Robust Inference
Authors: Andac Demir
Toshiaki Koike-Akino
Ye Wang
Deniz Erdogmus
Venue:  (2021)

Abstract: Learning data representations that capture task-related features, but are invariant to nuisance variations remains a key challenge in machine learning. We introduce an automated Bayesian inference framework, called AutoBayes, that explores different graphical models linking classifier, encoder, decoder, estimator and adversarial network blocks to optimize nuisance-invariant machine learning pipelines. AutoBayes also enables learning disentangled representations, where the latent variable is split into multiple pieces to impose various relationships with the nuisance variation and task labels. We benchmark the framework on several public datasets, and provide analysis of its capability for subject-transfer learning with/without variational modeling and adversarial training. We demonstrate a significant performance improvement with ensemble learning across explored graphical models.

Under review as a conference paper at ICLR 2021
AUTOBAYES: AUTOMATED BAYESIAN GRAPH EXPLO- RATION FOR NUISANCE-ROBUST INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Learning data representations that capture task-related features, but are invariant nuisance variations remains a key challenge in machine learning. We introduce an automated Bayesian inference framework, called AutoBayes, that explores different graphical models linking classifier, encoder, decoder, estimator and adversarial net- machine learning pipelines. AutoBayes also enables learning disentangled representations, where the latent variable is split into multiple pieces to impose various relationships with the nuisance variation and task labels. We benchmark the framework several public datasets, and provide analysis of its capability for subject-transfer learning with/without varia- tional modeling and adversarial training. We demonstrate significant performance improvement with ensemble learning across explored graphical models.
work blocks to optimize
INTRODUCTION
The great advancement of deep learning techniques based on deep neural networks (DNN) has enabled more practical design of human-machine interfaces (HMI) through the analysis of the user's physiological data (Faust et 2018), such as electroencephalogram (EEG) (Lawhern et al. 2018) and electromyogram (EMG) (Atzori et al.) 2016). However, such biosignals are highly prone to variation depending the biological states of each subject (Christoforou et al. 2010). Hence, frequent calibration often required in typical HMI systems.
Toward resolving this issue, subject-invarian methods (Özdenizei et al. 2019b). employing adversar- ial training (Makhzani et al.) 20155] Lample et al. 2017; Creswell et al. (0177) with the Conditional Variational AutoEncoder (A-CVAE) (Louizos et al. 2015 Sohn et al. 2015) shown in Fig. (b) have emerged to reduce user calibration for realizing successfu HMI systems. Compared to a standard DNN classifier C in Fig. conditional decoder D. and adversary A networks offers excellent subject-invariant performance. The DNN structure may be potentially extended with more functional blocks and more latent nodes as shown in Fig. [ However, such a DNN architecture design may rely on human effort and insight to determine the block connectivity of DNNs. Automation of hyperparameter and architecture exploration in the context of AutoML (Ashok et al. 2017 Brock et al. (017;; Caa et al. 20 7 He
aa) integrating additional functional blocks for encoder E, nuisance-
al. Miikkulaneen al. 2019 Real et al.) (020) [Stanley Miikuulainen 2002] Zoph
2018
et al.
2018 can facilitate DNN design suited for nuisance-invarian invariant inference. Nevertheless, without
proper reasoning, most of the search space for link connectivity will be pointless.
In this paper, we propose a systematic automation framework called AutoBayes, which searches for the best inference graph model associated with a Bayesian graph model (also a.k.a. Bayesian network) well-suited to reproduce the training datasets. The proposed method automatically formulates various different Bayesian graphs by factorizing the joint probability distribution in terms of data, class label, subject identification (ID), and inherent latent representations. Given Bayesian graphs, some meaningful inference graphs are generated through the Bayes- -Bll algorithm (Shachter) (013) for pruning redundant links to achieve high-accurase estimation. In order to promote robustness against nuisance variations such as inter-subjecc/session factors, the explored Bayesian graphs can provide
For example of speech recognition, nuisance factors such as speaker's 's attributes and recording environment may change the task accuracy. For image recognition, ambient light conditions and image sensor conditions may become inherent nuisance factors. In the context of this paper, nuisance variations mainly refer to subject identities and biological states during recording sessions for physiological data learning.

---

Under review as a conference paper at ICLR 2021
C
(a) Standard Classifier Net.
D
S
D
X"
X
3
Y
Z2
X
E
C
N
S"
(c) Potentially Extended Classifier
(b) Adversarial CVAE-Based Classifier Net
Figure 1: Inference methods classify Y given data X under latent Z and semi-labeled nuisance S.
100
99.8
Accuracy (%)
70
99.6
99.4
99.2
98.0
Ensemble AutoBayes
AutoBayes
A-CVAE Clessiter
Standard Classifer
30
OMNIST
Streas
RSVP
Emp
FacesEasic
FacesNoisy
ASL
Figure Model accuracy across different datasets. AutoBayes offers significant gain.
reasoning to use adversarial training with/without variational modeling and latent disentanglement. We demonstrate that AutoBayes can achieve excellent performance across various public datasets, in particular with an ensemble stacking of multiple explored graphical models.
KEY CONTRIBUTIONS
At the core of methodology is the consideration of graphical models that capture the probabilistic relationship between random variables representing the data features X, task labels Y. nuisance variation labels S, and (potential) latent representations Z. The ultimate goal label Y from the measured data feature X, which is hindered by the presence of nuisance variations (e.g., inter-subject/seesion variations) that are (partially) labelled by S. One may use standard DNN to classify Y given X as shown in Fig. a)) without explicitly involving S or Z. Although A C.VAE in Fig. b may offer nuisance-rowe robust performance through adversarial disentanglement of S from latent Z, there is guarantee that such model can perform well across different datasets. is exemplified Fig. [ A-CVAE outperforms the standard DNN model for some datasets (QMNIST, Stress, EmP) while it does not for the other cases. This may be due to the underlying probabilistic relationship of the data varying across datasets. Our proposed framework can construct justifiable models, achieving higher performance for every dataset, demonstrated in Fig. 2 It verified that significant gain is attainable with ensemble methods of different Bayesian graphs which explored in AutoBayes. For example, method with relatively shallow architecture achieves 99.61 % accuracy which is close to state-of-the-art performance in QMNIST dataset.
infer the task
The main contributions of this paper over the existing works five-fold follows:
2

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Pseudocode for AutoBayes Framework
Require: Nodes set V/ [Y, S1, S2 ................................ Sn Z1, Zz...., Zm,, where Y denotes task labels, X is measurement data, S = [S,, S 22...,, Sn] are (potentially multiple) semi-supervised nuisance variations, and 2 (2 72,,.., Zm| (potentially multiple) latent vectors
Ensure: Semi-supervised training/validation datasets
1: for all permutations of node factorization from Y to X do
2: Let Bg be the corresponding Bayesian graph for the permuted full-chain factorization
3: for all combinations of link pruning on the full-chain Bayesian graph Bg do
4:
Let B be the corresponding pruned Bayesian graph
5:
Apply the Bayes-Ball algorithm on B to build a conditional independency list I for all permutations of node factorization from X to Y do
6:
7:
Let Fo be the factor graph corresponding to a full-chain conditional probability Prune all redundant links in Fo based on conditional independency I
8:
9:
Let J be the pruned factor graph
10:
Merge the pruned Bayesian graph B into the pruned factor graph F Attach an adversary network A to latent nodes for Zk 1 S E
11:
Assign an encoder network & for p( in the merged factor graph Assign a decoder network 'D for p(r) ...) in the merged factor graph
13:
Assign a nuisance indicator network N for p(S)....) the merged factor graph
15:
Assign a classifier network C for p(yl ..-) ) in the merged factor graph
Adversary train the whole DNN structure to minimize loss function (
17:
end for
At most ([V) 2)! combinations
18:
end for
D At most 2 11/2 combinations
19:
end for
At most ([V) 2) combinations
20:
return the best model having highest task accuracy in validation sets
AutoBayes automatically explores potential graphical models inherent to the data by combi- natorial pruning of dependency assumptions (edges) and then applies Bayes-Ball to examine various inference strategies, rather than blindly exploring hyperparameters of DNN blocks.
AutoBayes offers solid reason of how to connect multiple DNN blocks to impose condi- tioning and adversary censoring for the task classifier, feature encoder, decoder, nuisance indicator and adversary networks, based on an explored Bayesian graph.
The framework is also extensible to multiple latent representations and nuisances factors.
Besides fully-supervised training, AutoBayes can automatically build some relevant graphi- cal models suited for cemi-supervised learning.
Multiple graphical models explored in AutoBayes be efficiently exploited improve performance by ensemble stacking.
We note that this paper relates to some existing literature in AutoML, variational Bayesian infer-
ence (Kingma & Welling 2013) Sohn et al. 2015 Louizos et al.) 2015), adversarial training (Good
fellow et al. |l. 201ouluumuuln et al. 20166] Donahue et al.] [016]] Makhzani et al. [0015] Lample et al.
al. 2017 Creswell | (207)), and Bayesian network (Nie al. 2018) Njah et al. 2019] Rohekar et al.
2018 as addressed in Appendix iI more detail. Nonetheless, AutoBayes is a novel framework that diverges from AutoML,, which mostly employs architecture tuning at miero level. Our work focuses on exploring neural architectures at a macro level, which is not an arbitrary diversion. but necessary interlude. Our method focuses on the relationships between the connections in neural network's 's architecture and the characteristics of the data (Minsky & Papert 2017). In addition to the macro-level structure learning of Bayesian network, our approach provides a new perspective in how to involve the adversarial blocks and to exploit multiple models for ensemble stacking.
3 AUTOBAYES
AutoBayes Algorithm: The overall procedure of the AutoBayes algorithm is described in the The AutoBayes automatically constructs non-redundant inference factor
pseudocode of Algorithm [ graphs given hypothetical Bayesian graph assumption, through the use of the Bayes-Ball algorithm.
3

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(d)
(e)
(
(h)
Figure 3: Bayes-Ball algorithm basic rules
Shachter 2013). Conditional nodes are shaded.
(a) Bayesian Model
(b) Z-First Inference
(c) S-First Inference
Figure Full-chain Bayesian graph and inference models for Z-first or S-first factorizations.
Depending on the derived conditional independency and pruned factor graphs, DNN blocks for encoder E, decoder D. classifier C, nuisance estimator A and adversary A are reasonably connected. The entire network is trained with variational Bayesian inference and adversarial training.
The Bayes- Ball algorithm (Shachter 2013) facilitates automatic pruning of redundant links in inference factor graphs through the analysis of conditional independency. Fig. 3 shows ten Bayes-Ball rules to identify conditional independency. Given a Bayesian graph, we can determine whether two disjoint sets of nodes are independent conditionally on other nodes through graph separation criterion. Specifically, an undirected path is activated encountering stop symbol: - in Fig. 3 If there are no active paths between two nodes when some conditioning nodes are shaded, then those random variables are conditionally independent.
Bayes ball can travel along without
Graphical Models:
We here focus 4-node graphs. Let p(y, 8, z, denote the joint probability distribution underlying the datasets for the four random variables, i.e., Y, S, Z, and X. The chain rule yield the following factorization for a generative model from Y X (note that at most 4! factorization orders exist including useless ones such as with reversed direction from X to Y):
ply, 2, 2,) p(y))(sly)pp(H)
())() s,,),
which is visualized in Bayesian graph of Fig. 4 The probability conditioned on X can then be factorized, e.g., as follows (among 3! different orders of inference factorization for four-node graphs):
ply, 8, 2/1)) [p(2|=)p(s|s)) p)ulls 2,x), Z-first-inferecce
S-firs-innerence
which are marginalized to obtain the likelihood: p(y(z) = E.aae [p(y. s. 2)) The above two
scheduling strategies in (2) are illustrated in factor graph models as Figs. 4 4nd and [c] respectively.
The graphical models in Fig. do not impose any assumption of potentially inherent independency in datasets and hence most generic. However, depending the underlying independency in datasets, we may be able to prune some edges in those graphs. For example, if the data only follows the simple Markov chain of X, while being independent of S and Z, shown in Fig.
aa
all links except
one between X and Y will be unreasonable in inference graphs of Figs. bbb and
(c) that justifies the
standard classifier model Fig. [[ a) This implies that more complicated inference models such as A-CVAE can be unnecessarily redundant depending on the dataset. This motivates us to consider an extended AutoMI. framework which automatically explores the best pair of inference factor graph and corresponding Bayesian graph models matching dataset statistics besides the micro-scale hyperparameter tuning.
Methodology: AutoBayes begins with exploring any potential Bayesian graphs by cutting links of the full-chain graph in Fig. 4aa imposing possible (conditional) independence. We then adopt the Bayes-Bas Ball algorithm each hypothetical Bayesian graph examine conditional independence
4

---

Under review as a conference paper at ICLR 2021
(a) Model
(b) Model
(c) Model C
(d) Model D
(e) Model E
Model
g Model
(h) Model HI
Iodel
i Model J
(k) Model
Figure 5: Example Bayesian graphs for data generative models under automatic exploration. Blue arrows indicate generative graph for decoder networks. Thick circled S specifies the requirement of S-conditional decoder, which is less-convemient when learning unlabeled nuisance datasets.
Applying Bayes-Ball justifies the reasonable pruning of the links in the full-chain inference graphs, and also the potential adversarial censoring when Z is independent of S. This process automatically constructs the connectivity of inference, generative, and adversary blocks with sound reasoning.
different inference strategies, e.g., full-chain Z-IS-first inference graphs in Figs. b
Consider an example case when the data adheres the following factorization:
ply. 8, 2, x) p(y)p(slp))p()))) p)p(2)
(3)
where we explicitly indicate conditional independence by slash-cancellation from the full-chain case in (I). This corresponds to Bayesian graphical model illustrated in Fig. Applying the Bayes-Ball Ball algorithm to the Bayesian graph yields the following conditional probability:
ply. 8, 2|2) = p(2[=)p(() p))p(l,,, p.
for the Z-first inference strategy in (2 The corresponding factor graph is then given in Fig. G(c) Note that the Bayes-Ball also reveals that there no marginal dependency between Z and S. which provides the reason to use adversarial censoring to suppress nuisance information S in the latent space Z. consequence. by combining the Bayesian graph and factor graph, we automatically obtain A-CVAE model in Fig. bbb AutoBayes justifies A-CVAE under the assumption that the data follows the true generative model is unknown, AutoBayes explores different Bayesian graphs like in Fig. 5 to search for the most relevant model. Our framework is readily applicable to graphs with more than 4 nodes to represent multiple Y, S, and Z. Models and K in Fig. are such examples having multiple latent factors Zi and Z2. Despite the search space for Bayes will rapidly grow with the number of nodes, most realistic datasets not require a large number of neural network blocks for macro-level optimization. See Appendix A.2 for more detailed descriptions for some Bayesian graph models to construct factor graphs like in Fig Also discussions of graphical models suitable for semi-supervised learning in Appendix A.4
the Bayesian model E in Fig. e)
Training: Given a pair of generative graph and inference graph, the corresponding DNN structures will be trained. For example of the generative graph model K in Fig. k) one relevant inference graph Kz in Fig. [[ will result in the overall network structure as shown in Fig. where adversary network is attached as Zg is (conditionally) independent of S. This 5-node graph model justifies recent work on partially disentanged A A-CVAE by Han et al. (202)). Each factor block is realized
by a DNN, e.g.. parameterized by 0 for po(21 Z2 (r), and all the networks except for adversarial network are optimized to minimize corresponding loss functions including L(j. v) as follows:
min max E C(g. y) + (8) Cn .x) X,KLL(Z 22 (W I)) C(s((ss 0.
(5)
(21, 22) Po (z), = (21, z2), = po())
Y'= ((1)) 8' (2))
(6)
where X, denotes regularization coefficient, KL is the Kellback-Leiblle divergence, and the adver- sary network P,, (s' (z2) is trained to minimize C(3', s) in an alternating fashion (see the Adversarial Regularization paragraph below).

---

Under review as a conference paper at ICLR 2021
Z
(a) Model Dz
(b) Model Ds
(c) Model Ez
(d) Model Es
(e) Model Fz
Model Fs
X
X
X
X
8
2
7A
2
(g) Model (h) Model Gs
(i) Model
i Model
(k) Model
()) Model
Figure 6: Z-first and S-first inference graph models relevant for generative models D-G, J, and K. Green arrows indicate feature extraction graph for encoder networks. Thick circled specifies the end node of inference, which is convenient when learning unlabeled nuisance datasets.
Generative
Py (x 221 Ø2)
X'
L(t', a))
Z1
Pe (s(21)
S
8(8
po (2 22
Zz
( 21, 22)
((G 9)
Inference
Pr, (22)
S'
((S
Adversary
Figure Overall network structure for pairing generative model K and inference model Kz.
The training objective can be formally understood from likelihood maximization perspective, in manner that can be seen generalization of the VAE Evidence Lower Bound (ELBO) con- cept (Kingma & Welling 2013). Specifically, it can viewed as the maximization of a variational lower bound of the likelihood (2, y, s) that is implicitly defined and parameterized by the networks, where represents the collective parameters of the network modules (c.g., $ = (Ø, v μ)) in the example of equation 5 that specify the generative model po (2,, y. 8 8|2), which implies the likelihood po (z. y, s), as given by
f pa y, s/z)p(2) dz.
(0 y, 8) =
However, since this expression is generally intractable, we introduce go lz, y. variational approximation of the posterior Pa (2|2, implied by the generative model (Kingma & Welling
2013 Ranganath et al. 2014):
log (1,, yr, s1l2)) -og ge(2 r1, Vi. Si) p(zi)
+log qo (2e (2s V/s s:) pa (z 20 Wi 8;)
log pa [24, Mi. 8.)
n
log (2;, Wis
+ (LL(gg(= )
log po (I;; M/, 8x 2+) (go y, (I{()))
where the samples 2; (o(2) x, Vi, s;)
are drawn for each training tuple (z,, His st), and the final inequality follows from the non-negativity of KI. divergence.

---

Under review as a conference paper at ICLR 2021
Table 1: Parameters of Public Dataset Under Investigation
Datasct
Modality
Dimension
Nuisance ( [S)
Classes ({Y))
Samples
Reforence
QMNIST
Image
28 28
836
10
70,000
Yadav & Botton
2019
Stress
Temperalure elc.
7
20
4
24,000
Biannclatab cl a 20766
RSVP
ERG
16 128
10
4
41,400
Orrhan 2012
MI
EEG
64 x 480
106
4
9,540
Golcherger oi aly 2000
Emr
EEG
56
27
9,180
Faces Basic
ECoU
31 x 400
14
2
4,100
Mileer aL 201 2016
Faces Noksy
39
2,100
ASL
EMG
16 x 100
5
33
9,900
Gumay 2019
Ultimately, the minimization of our training loss function corresponds to the maximization of the lower bound in (7). which corresponds to maximizing the likelihood of our implicit generative model, while also optimizing the variational posterior (2|2z. y, s) toward the actual posterior for the latent representation po [z|as lalar, y. 8), since the gap in the bound is given by (go Further factoring of log po (2r, y, 8|2) yields the multiple loss-terms and network modules.
(=(= [=,,, y, 8) s)]ppp (=|2, y, 8)).
Adversarial Regularization: We can utilize adversarial censoring when Z and S should be marginally independent, e.g., such as in Fig. [b) and Fig. 7 in order to reinforce the learning of representation Z that is disentangled from the nuisance variations S. This is accomplished by introducing an adversarial network that aims to maximize parameterized approximation g(s[2) of the likelihood p(s|2). while this likelihood is also incorporated into the loss for the other modules with a negative weight. The adversarial network, by maximizing the log likelihood log (()[2), essentially maximizes bower-bound of the mutual information I(S; Z), and hence the main network is regu- larized with the additional term that corresponds to minimizing this estimate of mutual information. This follows since the log-likelihood maximized by the adversarial network is given by
Ellog (<{|2)] I(S; 2) - (H(S) (L(pssa))ll((sl)))
where the entropy H(S)) is constant.
Ensemble Learning: We further introduce ensemble methods to make best use of all Bayesian graph models explored by the AutoBayes framework without wasting power-performance models. Ensemble stacked generalization works by stacking the predictions of the base learners in higher level learning space, where a meta learner corrects the predictions of base learners (Wolpert) 1992). Subsequent to training base learners, assemble the posterior probability vectors of all base learners together to improve the prediction. We compare the predictive performance of a logistic regression (LR) and shallow multi-layer perceptron (MLP) inference models. See Appendix A.5 for more detailed description of the stacked generalization.
ensemble meta learner to aggregate all
4 EXPERIMENTAL EVALUATION
We experimentally demonstrate the performance of AutoBayes for publicly available datasets as listed Table [ Note that they cover a wide variety of data size, dimensionality, subject scale, and class levels as well as sensor modalities including image, EEG, EMG, and electrocorticography (ECoG). See more detailed information of each dataset Appendix A.6
Datasets:
Model Implementation: All models were trained with minibatch size of 32 and using the Adam optimizer with an initial learning rate of 0.001. The learning rate is halved whenever the validation loss plateaus. A compact convolutional neural network (CNN) with layers is employed encoder network to extract features from C' x T data. Each convolution is followed by batch normalization (BN) and rectified linear unit (ReLU) activation. The AutoBayes chooses either deterministic latent encoder or variational latent encoder under Gaussian prior. The original data is reconstructed by decoder network that applies transposed convolutions, All of our experiments were run for 20 epochs on NVIDIA Tesla K80 12GB GPU. See Appendix A.7 for more details.
an
Results: F aa) shows the performance of QMNIST across 39 different inference models explored in AutoBayes including 2 ensemble models. Over 37 base models, some outperforms the standard
7

---

Under review as a conference paper at ICLR 2021
Dondaard Classiter
Encamble Autoflayes
Focurdty
8
3 & 5 8
(a) QMNIST (1.0.33%)
(b) MI (1.1.11), EmP (17.0%)) ASL (4.12.0%
Figure 8: Task classification accuracy across different graphical models (with standard deviation).
90
Accurucy (%)
Accuncy
90
80
A-VAB
A-CVAE
Disemble Amollayes
AutoBayes
AutoBayes
A-VAF
Classifier
70
Standad Classilia
80
5038
Ex
of
10
Number of Parameters
(a) Subject Variation Robustness
(b) Accuracy vs. Space Complexity
Figure 9: Task classification accuracy for Stress dataset.
classifier model A, whereas the rest of the models underperform. We observe large gap of 1.0% between the best and worst models with a standard deviation of 0.23% across all Bayesian graph models. This indicates that we may have potential risk that one particular model may lose up to 1.0% accuracy if we do not explore different models.
Similar behaviors with huge deviation be for different datasets shown in Fig. 8b). was shown that the best inference strategy highly depends on datasets. Specifically, the best model at one dataset does not perform best for different datasets. This suggests that we must consider different inference strategies for each target dataset and our AutoBayes provides such an adaptive framework across datasets. More detailed results are found Appendix A.8
Remarkably, the ensemble of base learners further enhances the performance regardless of the choice from LR or MLP as the meta learner, illustrated in Fig. P across all the datasets. For some low- performing datasets such as ErrP, MI and Faces (Noisy), ensemble learning significantly improves the accuracy by 15.3%, 19.3% and 13.2% at the expense of more storage and computational resources.
Exploring different models has actually significant benefit in improving nuisance robustness as shown Fig. a), where box-whisker plots are present to show the quartile distribution of the subject variation for the Stress dataset having S 20 users. We can observe that the standard classification (Model A) has a wider distribution; the best subject achieves an accuracy grater than 96%, whereas the worst-case user has lower than 82% accuracy. Except for model A. the other models from B to Kz take the subject ID (S) into consideration to extract nuisance-role robusst feature, which leads to significant improvement for the worst-case performance not only for the mean median. The
8

---

Under review as a conference paper at ICLR 2021
Table 2: Task classification performance of AutoBayes compared state-of-the-art.
Method
QMNIST
Stress
Emr
Basic
Noisy
ASL
Ensemble AutoBayes (Meta-MLP)
99.61
98.98
99.99
76.71
91.21
78.36
89.71
55.06
Ensemble AutoBayes (Meta-LR)
99.55
98.96
99.98
77.14
88.54
73.68
88.40
54.98
Best AutoBayes
99.54
95.35
93.42
57.83
75.91
67.31
76.58
51.12
State the art (SOTA)
99.84
85.30
71.60
03.8
48.80
ensemble stacking further improves the subject variation robustness, achieving the worst-case user performance of at least 94%. Additional results per user are found in Appendix
Despite the performance gain, the nuisance-robust models tend to have higher complexity. Fig. b shows the trade-off between the accuracy and the space complexity. Here, varied the number of hidden layers and hidden nodes for the models A, B, and Is to adjust the space complexity. The Pareto front over the finite set of DNN configurations indicated with lines. It is observed that the standard classifier model A has superior performance only at low complexity regimes, while it does improve performance beyond 95% accuracy even with increased complexity. The Pareto front of AutoBayes is thus better than the individual models at higher accuracy regimes. See Appendix A. for an additional analysis of time complexity.
We finally compare the performance of AutoBaves with the benchmark competitor models from (Byerly et al. 2020; Han et al., 2020 Özdenizei et al. b019cblal in Table 2 It can be seen that AutoBayes outperforms the state-of-the-art in all datasets except QMNIST. Consequently, we can see great advantage of AutoBayes with exploring different graphical models. Even for QMNIST, AutoBayes meta-MLP model, achieving 99.61% % accuracy, ranks 17 in the published leaderboard. Note that performing better than 99. 99.84% nearly impossible, since some numbers are illegible or mislabeled. Also note that have not specifically designed AutoBayes architecture for image classification but for apatio-temport signal applications and apperrparameters were not fully optimized yet.
AutoBayes can be readily integrated with AutoML to optimize any hyperparameters of individual DNN blocks. Nevertheless, our primary objective to show proof-of-concep benefit from solely graphical model exploration of AutoBayes, we leave more rigorous analysis to optimize DNN parameters such network depths, widths, activation, augmentation, etc. as future work.
CONCLUSION AND FUTURE WORK
We proposed new concept called AutoBayes which explores various different Bayesian graph models to facilitate searching for the best inference strategy, suited for nuisance-robust deep Icarning. With the Bayes-Bar -all algorithm, among classifier, encoder, decoder, nuisance estimator and adversary DNN blocks. As a proof- of-concept analysis, we demonstrated the benefit of AutoBayes for various public datasets. We observed a huge performance gap between the best and worst graph models, implying that the use of one particular model without graph exploration can potentially suffer a poor classification result. In addition, the best model for one dataset does not always perform best for different data, which encourages us use AutoBayes for adaptive model generation given target datasets. We further improved the performance approaching the state-of-the the- accuracy by exploiting multiple graphical models explored in AutoBayes through the use of ensemble stacking. The ensemble AutoBayes offers significant gain in nuisance robustness by improving the worst-case user performance. Even though additional computations are required, we showed that AutoBayes can still achieve the superior Pareto front in the trade-off between complexity and accuracy. We are extending the AutoBayes framework to integrate AutoML to optimize hyperparameters of each DNN block. How to handle the exponentially growing search space of possible Bayesian graphs along with the number of random variables remains a challenging future work. It should require more sophisticated metrics like Bayesian information criterion for efficient graph exploration.
method can automatically construct reasonable link connections

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anubhav Ashok, Nicholas Rhinchart, Fares Beainy, and Kris M Kitani. N2N learning: Network to network compression via policy gradient reinforcement learning. arXiv preprint arXiv:1709.0600
2017.
Manfredo Atzori, Matteo Cognolato, and Henning Müller. Deep learning with convolutional neural networks applied to eleetromyography data: A resource for the classification of movements for prosthetic hands. Frontiers in neurorobotics, 10:9, 2016.
Justin Bayer, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. Evolving memory cell structures for sequence learning. In International Conference on Artificial Neural Networks, pp. 755-764. Springer, 2009.
Javad Birjandtalab, Diana Cogan, Maziyar Baran Pouyan, and Mehrdad Nourani. non-EEG biosig- nals dataset for assessment and visualization of neurological status. In 2016 IEEE International Workshop on Signal Processing Systems (SiPS), pp. 110-1 14. IEFF, 2016.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXiv:1708.05530 2017.
Adam Byerly, Tatiana Kalganova, and Ian Dear. A branching and merging convolutional network with homogeneous filter capsules. arXiv preprint arXiv:2001.091 2020.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Reinforcement learning for
architecture search by network transformation. arXiv preprint 1707v.177..04873 2017.
Luis de Campos. A scoring function for learning bayesian networks based on mutual information and conditional independence tests. Journal of Machine Learning Research, 7(0ct):2149-2183 2006.
Christoforos Christoforou, Robert M Haralick, Paul Sajda, and Lucas C Parra. The bilinear brain: towards subject-invariant analysis. In 2010 4th International Symposium on Communications, Control and Signal Processing (ISCCSP), pp. 1-6. IEEE, 2010.
Antonia Creswell, Anil A Bharath, and Biswa Sengupta. Conditional autoencoders with adversarial information factorization. arXiv preprint 17I:v:1111.0..1 2017.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. AutoAugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 113 123, 2019.
James Cussens, Matti Järvisalo, Janne H Korhonen, and Mark Bartlett. Bayesian network structure learning with integer programming: Polytopes, facets and complexity. Journal of Artificial Intelligence Research, 58:1 85-229, 2017.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 2016.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropictro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606 (60.000000000704 2016.
Oliver Faust, Yuki Hagiwara, Tan Jen Hong, Oh Shu Lih, and U Rajendra Acharya. Deep learning for healthcare applications based on physiological signals: A review. Computer methods and programs in biomedicine, 161:1-13, 2018
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E. Mietus, George B Moody, Chung-Kan Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23):e215-e22) 2000.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014.
10

---

Under review as a conference paper at ICLR 2021
Sezen Yagmur Günay, Mathew Yarossi, Dana H Brooks, Eugene Tunik, and Deniz Erdogmuy. Transfer learning using low- diwdmmmnsiioal subspaces for EMG- based classification hand posture. In 2019 9th International IEEE/EMBS Conference Neural Engineering (NER), pp. 1097-1100. IEEE, 2019.
Mo Han, Ozan Özdenizei, Ye Wang, Toshiaki Koike-Akino and Deniz Erdogmus. Disentangled ad- versarial autoencoder for subject-invariant physiological feature extraction. IEEE Signal Processing Letters, pp. 1565- 1569, 2020.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoML for model compression and acceleration mobile devices. In Proceedings of the European Conference Computer Vision (ECCV), pp. 784-800, 2018.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztion with Gumbel-softmax, arXiv preprint arXiv:1611.01144 2016.
Rafal Jozefowiez, Wojciech Zaremba, and lla Sutskever. An empirical exploration of recurrent network architectures. In International conference on machine learning, pp. 2342-2350, 2015.
Diederik Kingma and Max Welling.
Auto-encoding variational Bayes.
arXiv preprint
arXiv:1312.6114 2013.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc' Aurelio Ranzato. Fader networks: Manipulating images by sliding attributes. In Advances in Neural Information Processing Systems, pp. 5967-5976, 2017.
Vernon J Lawhern, Amelia Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. Journal of neural engineering, 15(5)0056133, 2018.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. arXiv preprint 15 2015.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint 1r 2015.
Perrin Margaux, Maby Emmanuel, Daligault Sébastien, Bertrand Olivier, and Mattout Jérémie. Objective and subjective evaluation of online error correction during P300-based spelling. Advances in Human- Commante Interaction, 2012, 2012.
Thomas Miconi. Neural networks with differentiable structure. arXiv preprint arXiv: 1606.0260 2016.
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293-312. Elsevier, 2019.
Kai J Miller, Dora Hermes, Nathan Witthoft, Rajesh PN Rao, and Jeffrey G Ojemann. The phys- iology perception in human temporal lobe specialized for contextual novelty. Journal of neurophysiology. 11((1):256-25) 263, 2015.
Kai Miller, Gerwin Schalk, Dora Hermes, Jeffrey G Ojemann, and Rajesh PN Rao. Spontaneous decoding of the timing and content of human object perception from cortical surface recordings reveals complementary information in the event-related potential and broadband spectral change.
PLoS computational biology, 12(1), 2016.
Kai Miller, Dora Hermes, Franco Pestilli, Gagan S Wig, and Jeffrey G Ojemann. Face percept formation in human ventral temporal cortex. Journal of neurophysiology. 1186))2614-2627,, 2017.
Marvin Minsky and Seymour A Papert. Perceptrons: An introduction to computational geometry. MIT press, 2017.
11

---

Under review as a conference paper at ICLR 2021
Siqi Nie, Meng Zheng, and Qiang Ji. The deep regression Bayesian network and its applications: Probabilistic deep learning for computer vision. IEEE Signal Processing Magazine, 35(1):101-111, 2018.
Hasna Njah, Salma Jamoussi, and Walid Mahdi. Deep bayesian network architecture for big data mining. Concurrency and Computation: Practice and Experience, 31(2):e4418 18, 2019.
Umut Orhan, Kenneth Hild, Deniz Erdoğmuş, Brian Roark, Barry Oken, and Melanie Pried-Oken. RSVP keyboard: An EEG based typing interface. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 645-648. IEEE, 2012.
Ozan Özdenizci, Barry Oken, Tab Memmott, Melanie Fried-Oken, and Deniz Erdogmus. Adversarial feature learning in brain interfacing: An experimental study on eliminating drowsiness effects. arXiv preprint arXiv: 2019a.
Ozan Özdenizei, Ye Wang, Toshiaki Koike-Akino, and Deniz Erdogmuş. Transfer learning in brain-computer interfaces with adversarial variational autoencoders. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 207-210. IEEE, 2019b.
Ozan Özdenizei, Ye Wang, Toshiaki Koike-Akino, and Deniz Erdogmus. Adversarial deep learning in eeg biometrics. IEEE Signal Processing Letters, 26(5):710-714, 2019c.
Daniel S Park, William Chan, Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. SpecAugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:190.0087 2019.
Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniivesiityreesss. 2000.
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Artificial Intelligence and Statisties, pp.
PMLR, 2014.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning-Voin Volume 70, pp. 2902-2911. 1. JMLR. org, 2017.
Esteban Real, Chen Liang, David R So, and Quoc V Le. AutoML-Zen -Zro: Evolving machine learning algorithms from scratch. arXiv preprint arXiv:2003.03884, 2020.
George Rebane and Judea Pearl. The recovery of causal poly-trees from statistical data. arXiv preprint urXiv:1304.2733 2013.
Raanan Y Rohekar, Shami Nisimov, Yaniv Gurwicz, Guy Koren, and Gal Novik. Constructing deep neural networks by Bayesian network structure learning. In Advances Neural Information Processing Systems, pp. 3047-3058, 2018.
Marco Scutari. Bayesian network constraint-based structure learning algorithms: Parallel and
optimised implementations in the bnlearn package. arXiv preprint 100, 2014.
Ross D Shachter. Bayes-Bale The rational pastime (for determining irrelevance and requisite information in belief networks and influence diagrams). arXiv preprint arXiv: /101:/330.741.. 2013.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pp. 3483 3491, 2015.
Kenneth O Stanley and Risto Mikkkulainen. Evolving neural networks through augmenting topologies. Evolutionarr computation, 10(2):99-127, 2002.
David H Wolpert. Stacked generalization. Neural networks, 5(2):241-259, 1992.
Chhavi Yadav and Léon Bottou. Cold case: The lost MNIST digits. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 2019.
12

---

Under review as a conference paper at ICLR 2021
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:161..0158, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 8697-8710, 2018.
13

---

Under review as a conference paper at ICLR 2021
APPENDICES
A.I
RELATED WORK
We note that this paper relates to some existing literature follows.
AutoML: Searching DNN models with hyperparameter optimization has been intensively investigated in a framework called AutoML (Ashok et al., 2017: Brook ct al. 2017| Cai
et al., 2017; He et al. 2018 8, Miikkulainen et al. [019] Real et adl 2020; Staanlyy
& Miikkulainen/ 2002 Zoph al.] 2018). The automated methods include architecture Search (Zoph et al. 201 8 Real et al. 2017 He et al. 2018 Real et al. 2020), learning rule design (Bayer et 2009 Jozefowicz al. 2015), and augmentation exploration (Cubuk 20199] Park et al. 2019). Most work used either evolutionary optimization or reinforce-
c al.
ment learning framework adjust hyperparameters or to construct network architecture from pre-selected building blocks. (Miconi (016) gradually increases the size of an RNN starting from only one node by incorporating structural parameters into model training, which are optimized along with the model weights. (Zoph & Le| (016) uses reinforcement learning to find the optimal neural network architecture based on actor-critic framework. The method uses an LSTM controller and critic to explore the hyperparameter configurations for each layer (number of filters, kernel size and stride) based on the validation error of the output architecture that corresponds to reward. The recent AutoMM- (Real et al.) 2020) considers an extension to preclude human knowledge and insights for fully automated designs from scratch.
Variational Bayesian Inference: The VAE (Kingma & Welling 2013) introduced vari- ational Bayesian inference methods, incorporating autoassociative architectures, where generative and inference models can be learned jointly. This method was extended with the CVAE (Sohn et al.] 2015), which introduces a conditioning variable that could be used represent nuisance variations, and a regularized VAE in (Louizos et al. 2015), which considers disentangling the nuisance variable from the latent representation.
The concept of adversarial networks was introduced with Generative Adversarial Networks (GAN) (Goodfellow et al., 2014), and has been adopted into myriad applications. The simultaneously discovered Adversarially Learned Inference (ALI) (Du moulin et al., 2016) and Bidirectional GAN (BiGAN) (Donahue et al. (016)) propose an adversarial approach toward training an autoencoder. Adversarial training has also been combined with VAF to regularize and/or disentangle the latent representations (Makhzani ct al. 2015 Lample et al. 2017 Creswell et al. 2017).
Adversarial Training:
Bayesian Network Structure Learning: Deep Bayesian network (Nie et al. 2018 Njah
et al. 2019: Rohekar et al. 2018) has been studied to learn probabilistic relationships between random variables. Learning model structure of Bayesian network is a problem that has long been studied, c.g., recovery algorithm (Rebane & Pearl 2013). scoring methods (Campos) 2006), and constraint methods (Scutari 2014]|Pearl et al.. 20000. Scoring methods commonly use the posterior probability of the Bayesian network given training data, such as Bayesian information criterion (BIC). Although the complexity of an exhaustive search is superexponential in the number of variables, recent approaches (Cussens et al.) 2017) showed capability to learn structure of Bayesian network with up to 100 variables using integer programming. Constraint baase methods use conditional independence tests between pairs of variables, commonly mutual information test or the Student's 's t-test for correlation. All constraint-based methods entail three phases: i.e., (i) learning Markov blankets of each variable, (ii) learning neighbors (parents and children) of each variable that identifies which ares are present in a Bayesian network, and (iii) establishing are directions.
Compared to the existing AutoML literature, our method provides more systematic framework to explore justifiable network architectures from macro view. Although related Bayesian network was studied to design DNN architecture, our method extends it to realize nuisance robustness by reasonably involving adversarial networks. In addition, ensemble stacking first introduced in AutoML framework where multiple architectures can be reused to improve the performance over every individual model.

---

Under review as a conference paper at ICLR 2021
A.2 BAYESIAN GRAPH AND INFERENCE MODELS
Given measurement data, we never know the true joint probability beforehand, and therefore we shall assume one of several possible generative models. AutoBayes aims explore such potential graph models to match the measurement distributions. As the maximum possible number of graphical models is huge even for four-node case involving Y. S, Z and X, we restrict our focus a few meaningful graphs-on of oftinterest shown in Fig. [S] Each Bayesian graph corresponds to the following assumption of the joint probability factorization (p(x) ...) term specifies generative model of X
Model-A
Model-B
Model-C
Model-D
Model-E
ply, 8,2,)
Model-F
(9)
Model-G
Model-H
Model-I
21 y), 22, 21, Model-K
Model-J
where explicitly indicate independence by slash-cancelled factors from the full-chain case in equa- tion I Blue-colored terms correspond to the blue arrows in Figs. generative graph of decoder networks. Depending the assumed Bayesian graph, the relevant inference strategy will vary as some variables may be conditionally independent, which enables pruning links in the inference factor graphs. As shown in Fig. 66] the reasonable inference graph model can be automatically generated by the Bayes-Ball Ball algorithm (Shachter 2013) on each Bayesian graph hypothesis inherent in datasets. Specifically, the conditional probability ply, s, 2 (r) can be obtained for each model as below.
Bayesian Graph Model A (Direct Markov): The simplest model between X and Y would be single Markov chain without any dependency of S and Z, shown Bayesian graph of Fig. a This model puts assumption that the data are nuisance-invarian. For this case, there is no reason to employ complicated inference models such as A-CVAE since most factors will independent as p(y,
p(2 ))(()) (v)) $, z).. We hence should use standard classification method, as in Fig. a to infer Y given X, based on the inference model p(y|z) without involving S and Z.
Bayesian Graph Model B (Markov Latent): Assuming latent Z can work in Markov chain of Y - Z X shown in Fig. bb)] we obtain simple inference model: p(y, s, 2(z)) =
p(z[#) ((())) f) 2, f). Note that this model assumes independence between Z and S, and thus adversarial censoring (Makhzani et al. 2015 Creswell et al., 2017) Lample et al. 2017) 7) can make more robust against nuisance. This model is hence based on A-VAE.
Bayesian Graph Model C (Subject-Dependent)
We may model the case when the data X
directly depends on subject and task Y, shown in Fig. C For this case, we may consider the corresponding inference models due to the Bayes-Ball Ball:
(14sla)p(al44.ddllssss.s.. ply 8, 2/x) = (p(yl=))(()) x *1
Model-Cs
(10)
Model-Cy
Note that this model does not depend on 7, and thus Z-first inference strategy reduces S-first model. As reference, we here consider additional Yfirsst inference strategy to evaluate the difference.
Bayesian Graph Model (Latent Summary): Another graphical model shown in Fig. where a latent space bridges all other random variables. Bayes-Ball yields the following models:
(p(z(z)p(s(z A, 2,)), Model-Dz
ply s, = (x))
(1I)
((())))((())))) (,,,,, 2, 8f Model-Ds
whose graphical models are depicted in Figs. a and respectively.
2

---

Under review as a conference paper at ICLR 2021
Bayesian Graph Model E (Task-Summary Latent): Another graphical model involving latent variables is shown in Fig. ee) where a latent space only summarizes Y. Bayes-Ball yields the following inference models:
ply 8, 2/2) p(t
Model-Ez
(12)
(1)) 2, Model-Es
which are illustrated in Figs. [ and (d)
Note that the generative model E has no marginal dependency between Z and S. which provides the reason to use adversarial censoring to suppress nuisance information S in the latent space Z. In addition, because the generative model of X is
dependent on both Z and S, is justified to employ the A-CVAE classifier shown in Fig. [b)
Bayesian Graph Model (Subject-Summany Latent): Consider Fig. DD
where a latent variable summarizes subject information S. The Bayes-Ball provides the inference graphs shown in Figs. and ( which respectively correspond to:
s, (p(al))((slz,))) 2) Model-FZ
(13)
((())r)p(z(s, 2) J. R) Model-Fss
Bayesian Graph Model G:
Letting the joint distribution follow the model in Fig. g the following inference models via the Bayes-Ball:
we obtain
(11)))p(sl,,))p(00 2, Model-Gz ((ssx))((22 (. (p(ss=)p(((ss_)))))) s. Model-Gs
ply. s, 2/2x)
(14)
whose graphical models are described in Figs.
and Note that the inference model Gs in
Fig. (h identical to the inference model Ds Fig. bb) Although the inference graphs Gs and Ds
are identical, the generative model of X is different as shown in Figs. [g and (d) Specifically, VAE
decoder for the model G should feed S along with variational latent space Z, and thus using CVAE
is justified for the model G but D. This difference of the generative models can potentially make
different impact on the performance of inference despite the inference graph alone is identical.
Bayesian Graph Models and I: Both the generative models H and shown in Figs. 5(h) and (i) have the fully-connected inference strategies as given in (2), whose graphs are shown in Figs.
and [c) respectively, since no useful conditional independency can be found with the Bayes-Ball. Analogous to the relation of models Ds and Gs, the inference graph can be identical for Bayesian graphs H and I, whereas the generative model X different as shown in Figs. and
Bayesian Graph Model (Disentangled Latent): We can also consider multiple latent vectors to generalize the Bayesian graph with more vertices. We here focus on two such examples of graph models with two-latent spaces as shown in Figs. FIj) and (k) Those models are identical class of the model D. except that a single latent Z is disentangled into two parts Zi and Zz, respectively associated with and Given the Bayesian graph of Fig. the Bayes-Ball yields some inference strategies including the following two models:
[p(2 22 (s)p))))p))))))))))))))))) 24, Ap 22, (pss==)p(21))))) (s, x) 2))(22) 21, 21, ()p(pppp.p...) f), Model-Js
Model-Jz
ply, 8, 21, 22|x)=
(15)
which are shown in Figs.
h and (1) Note that Zg is marginally independent of the nuisance variable
S, which encourages the use of adversarial training to be robust against subject/session variations.
Bayesian Graph Model K (Conditionally Disentangled Latent): Fig. k linking Z, and Zg yields the following inference models:
Another modified model in
22 Dz ;2,
Model-Kz Model-Ks
ply. s, 22(x)
(16)
(((s)))((21) (>2(l, 21, x) r))(v/ 21, 22, f),
as shown in Figs. k and (1) The major difference from the model lies the fact that the inference graph should use Z along with Z2 to infer Y.
3

---

Under review as a conference paper at ICLR 2021
A.3 BACKGROUND ON VARIATIONAL BAYESIAN INFERENCE
Variational AE AutoBayes may automatically construct autoencoder architecture when latent variables are involved, e.g., for the model E in Fig. [ For this case, Z represents stochastic node to marginalize out for X reconstruction and Y inference, and hence VAE will be required. In contrast vanilla autoencoders, VAE uses variational inference by assuming a marginal distribution for latent p(z). In variational approach, we reparameterize Z from a prior distribution such as the normal distirbution to marginalize. Depending on the Bayesian graph models, we can also consider reparametering semi-supervision on S (i.e., incorporating a reconstruction loss for S) as conditioning variable. Conditioning on Y and/or should depend on consistency with the graphical model assumptions. Since VAE is special case of CVAE, we will go into further detail about the more general CVAE below.
When X is directly dependent on S or Y along with Z in the Bayesian graph, the AutoBayes gives rise the CVAE architecture, e.g.. for the models E/F/G/H/I in F For those generative models, the decoder DNN needs to feed S or Y as a conditioning parameter. Even for other Bayesian graphs, the S-first inference strategy will require conditional encoder in CVAE, e.g.. the models Ds/Es//ss/Gs/JJ/KK in Fig. 6 where latent Z depends on S.
Conditional VAE
Consider the case when S plays as the conditioning variable in a data model with the factorization:
p(8, =) p(s)p(=)p(((ss,
(17)
where we leave p(s) arbitrary (since it will not be directly used). The CVAE is trained according to maximizing the likelihood of data tuples (s, x) with respect to p(æ]ss, which is given by
directly parameterize p(ar|s, =), set p(=) to something simple (e.g., isotropic Gaussian), and
p(x]+) = p(alss = =))(;)
(18)
which is intractable to compute exactly given the potential complexity of the parameterization of p(z]ss z). While it could be possible to approximate the integration with sampling of Z, the crux of the VAE approach is to utilize a variational lower-bound of the likelihood that involves a variational approximation of the posterior p(z|s, x) implied by the generative model. With q(2)s, x)) representing the variational approximation of the posterior, the Evidence Boond (ELBO) is given by
log (og|(= 2 Ezzg(= a..)) s.m) porl =]] - KL (1(2)s,))))ll
(19)
The parameterization of the variational posterior q 2z) may also be decomposed into parameterized components, c.g., (z]ss, .r = q(s) ( such as in the S-ffirtt models shown in Fig. 6 Such decomposition also enables the possibility of supe-supe training, which can be convenient when some of the variables, such as the nuisances variations, are not always labeled. For data tuples that include s, the likelihood ((s can also be directly optimized, and the given value for s is used an input to the computation of q(2 s,, æ).. However, for tuples where is missing, the component q( (x) can be used to generate an estimate of s to be input to q z). We further discuss semi supervisd learning and the sampling methods for categorical nuisance variables in Appendix below.
A.4
LEARNING: CATEGORICAL SAMPLING
Graphical Models for Semi-Supervised Learning Nuisance values S such as subject ID or session ID may not be always available for typical physiological datasets, in particular for the testing phase of an HMI system deployment with new users, requiring semi-supervised methods. We note that some graphical models are well-suited for such semi-supervised training. For example, among the Bayesian graph models in Fig. the models C/E/G/I require the nuisance S to reproduce
X. If no ground-truth labels of S are available, we need to marginalize across all possible categories for the decoder DNN 'D. Even for other Bayesian graphs, the corresponding inference factor graphs in Fig, may not be convenient for the semi-supervised settings. Specifically, for models Ez/Fz/Jz/Kz have an inference of S at the end node, whereas the other inference models use inferred S for subsequent inference of other parameters. If S is missing or unknown as a semi-supervised setting, those inference graphs having S in a middle node are inconvenient as need sampling over all possible nuisance categories. For instance, the model Kz shown in Fig. 7 does not need S marginalization, and thus readily applicable to semi-supervised datasets.
4

---

Under review as a conference paper at ICLR 2021
Variational Categorical Reparameterization In order to deal with the issue of categorical sam- pling. we can use the Gumbel-Softmai reparameterization trick (Jang et al., 2016). which enables differentiable approximation of one-hot encoding. Let [T] x2, ................................ ... ||ss denote target probability mass function for the categorical variable S. Let 91, -2. 9 9is be independent and identically
distributed samples drawn from the Gumbel distribution Gumbel(O, 1) Then, generate an |S|- dimensional vector
[$1. $2, ..., $|s] according to
8k exp((log(Ta) ( + 9k)/T)
(20)
exp((log(T:) + 9:)/T)
where T > 0 is a softmax temperature. As the softmax temperature T approaches O, samples from the Gumbel-Softmax distribution become one-hot and the distribution becomes identical to the target categorical distribution. The temperature 7 is usually decreased across training epochs as an annealing technique, e.g., with exponential decaying.
A.5 ENSEMBLE LEARNING: STACKED GENERALIZATION
To achieve higher predictive performance, we construct ensembles from the output posterior class probabilities of all graphical models. Let D, = { Un, Un, Sn)[n : :N denote data set, where In is a data instance, yn is the task label, Sn is the nuisance (subject) label and N is the number of samples in the dataset. We randomly split the data into training set Praii and validation set Diest- Given 37 graphical models, which we call base learners, we induce decision algorithm Mk. for 1, ................................ 37 by invoking the kth graphical model on the data in D Prain- For each Vp in Durain graphical model Ma generates a class probability vector for task and nuisance label prediction. Let (r,( = { (P(y [xn), (.a)...... P(y: x (zn) ......... P(yN, (w,,)) denote the posterior probability distribution over N, task labels and (T.)) = (a.
P(SN, [am)] denote the posterior probability distribution over N, nuisance labels produced by model MM given data instance In- Ensemble generalization works by stacking the predictions of the base learners in a higher level learning space, where meta learner, denoted as Mr. corrects the predictions of base learners (Wolpert 1992). Subsequent to training base learners, we assemble the posterior probability vectors of all base learners together: (z,,) = {Pky (xn)3 and (z(x = (Pks (In)), where k 37. Mr is trained using the predictions from all base learners as input attributes: Din = (E,, (x,))) and correct labels as output: Dout {vn: 8n)). where n = 1 Ntrain Hold-out Drest is used to measure the classification performance of both base and meta learners. To make best use of the base learners, we compare the predictive performance of a LR model and a shallow MLP meta learner in Table 2
A.6 DATASETS DESCRIPTION
We used publicly available physiological datasets well as benchmark MNIST follows. The parameters of datasets are also summarized in Table
QMNIST: A hand-written digit image MNIST with extended label information including
writer ID number (Yadav & Bottou 2019 Fl There are |S| 539 writers for classify-
ing Y = 10 digits from grayscale 28 x 28 pixel images over 60,000 training samples.
Additional 297 writers provide 10,000 test samples.
Stress: A physiological dataset considering neurological stress level (Birjandtalab et al. 2016 consists of multi-modal biosignals for |Y|= = 4 discrete stress states from [S] 20 healthy subjects, including plyysical/oommiiive/mmoooooo
stresses as well as relaxation. The data were collected by C' 7 sensors, i.c., electrodermal activity, temperature, three- dimensional acceleration, heart rate, and arterial oxygen level. For each stress status, corresponding task of 5 minutes long (i.e., T = 300 time samples with 1 Hz down-sampling) was assigned to subjects for total of 4 trials.
"The Gumbel 1) distribution can he sampled by drawing Exp(1) and computing g log(e). dataset: htw.://wwwaaaa////..o/ooo/o///oa Stress dataset: https////.///////...ooooo///////
5

---

Under review as a conference paper at ICLR 2021
RSVP: An EEG-based typing interface using rapid serial visual presentation (RSVP) paradigm (Orhan et al. 2012 | = 10 healthy subjects participated in the experi- ments at three sessions performed
different days. The dataset consists of 41,400 epochs of C = 16 channel REG data for T = 128 samples, which were collected by g. USBamp biosignal amplifier with active electrodes during RSVP keyboard operations. for emotion elicitation, resting-state, or motor imagery/execution task.
Y
4 labels
MI:
The PhysioNet EEG Motor Imagery (MI) dataset (Goldberger et al. 2000 Excluding irregular timestamp, the dataset consists of [S] 106 subjects' EEG data. During the experiments, subjects were instructed to perform cue-based motor execution/imagery tasks while = 64 channels were recorded at a sampling rate of 160 Hz. Focusing on motor imagery tasks, we the EEG data for three seconds of post-cue interval data (i.e., T 480 time samples). The subject performed |Y| = 4-class tasks; either right hand motor imagery, left hand motor imagery, both hands motor imagery, or both feet motor imagery. This resulted in a total of 90 trials per subject.
ErrP: An error-related potential (ErrP) of front-central EEG dataset (Margaux et al. 2012 The dataset consists of EEG data recorded from |s| = 16 healthy subjects participating in an offlin P300 spelling task, where visual feedback of the inferred letter is provided to the user at the end each trial for 1.3 seconds to monitor evoked brain responses for erroneous decisions made by the system. EEG data were recorded from C = 56 channels for epoched 1.25 seconds at sampling rate of 200 Hz (i.e., T = 250). Across five recording sessions, each subject performed a total of 340 trials. Since it was offline copy spell task, binary Y| = labels were provided as erroncous or correct feedback.
Faces Basic: An implanted electrocorticographh (ECoG) array dataset for visual stimulus experiments (Miller et al. 2015 2016 & ECoG arrays were implanted on the subtemporal cortical surface of [S] = 14 epilepsy patients. |Y| = 2 classes of grayscale images, either faces or houses, were displayed rapidly in random sequence for 400 ms each with black- screen intervals of 400 ms. The ECoG potentials were measured with respect to a scalp reference and ground, at a sampling rate of 1000 Hz. Subjects performed basic face and house discrimination task. There were 3 sessions for each patient, with 50 house pictures and 50 face pictures in each run, in total 4,100 samples. We use the first C = 31 channels to analyze for T 400. Reusing the public dataset requires the ethics statement information
Faces Noisy: The implanted ECoG arrays dataset for visual stimulus experiments (Miller et al. 2015 2017). The experiment is similar to Faces Basic dataset, while pictures of faces and houses are randomly scrambled. There are |S| = 7 subjects with C 39 channels. Refer ethics statement to reuse the dataset 10
ASL: An EMG dataset for finger gesture identification for American Sign Language (ASL) Gunna et al.) 2 II S = 5 healthy, right-handed, subjects participated in experiments with surface EMG (Delsys Inc. Trigno) recorded at 2 kHz from |C] 16 lower-arm muscles. Subjects shaped their right hand into letters and numbers of the ASL posture set presented pictures on computer screen (|Y = 33 postures, 3 trials per
*RSVP dataset:
6 MII dataset: http://///ee///oooo..oo////////a
8 Faccs dataset: hattpe://cxhibitss
Ethies statement: All patients participated in a purely voluntary manner, after providing informed written consent, under experimental protocols approved by the Institutional Review Board the University of Washing- ton (#12193). All patient data was anonymized according to IRB protocol, in accordance with HIPAA mandate. These data originally appeared in the manuscript "Spontaneous Decoding of the Timing and Content of Human Object Perception from Cortical Surface Recordings Reveals Complementary Information in the Event-RRelted Potential and Broadband Spectral Change" published in PLoS Computational Biology in 2016 (Miller et al.) 2016).
AIl patients participated in a purely voluntary manner, after providing informed written consent, under experimental protocols approved by the Institutional Review Board of the University of Washington (#12193). All patient data was anonymized according to IRB protocol, in accordance with HIPAA mandate. These data originally appeared in the manuscript "Face percept formation human ventral temporal cortex" published in Journal of Neurophysiology in 2017 (Miller et al. 2017).
ASL. Dataset: http: //h11. whandl e. net / / /2044//2222222233

---

Under review as a conference paper at ICLR 2021
Table 3: DNN model parameters in Fig. Conv(h, w)G denotes 2D convolution layer with kernel size of (h, w) for oulput channel of over group g. FC(h) denotes fully-connected layer with h output nodes. BN denotes batch normalization.
Classifier
Encoder &
Decoder D
Nuisance A
Adversary A
FC(
Conv(1, 15)30
FC(20T")
FC(
FC 2))
BN+ReLU
BN+ReLU
ReLU
BN+ReLU
BN+ReLU FC([S))
FC((D)
Conv(1, 7)50
Conv(C, 1)50
FC((S))
BN+ReLU
BN+ReLU
Conv(1,3)
Conv(1,
BN+ReLU
BN+ReLU
ConvíC, 1)
Conv(1,7)
FC([Z))
BN+ReLU
Conv(1, 15)50
posture). Dynamic letters 'J' and 'Z.' were omitted, along with the number '0', which is visually the same as the letter 'O', The participants were given 2 seconds to form the posture, 6 seconds to maintain it, and 2 seconds to rest between trials. The signal is decimated to be T 100.
A.7 DNN MODEL PARAMETERS
For 2D datasets, we use deep CNN for the encoder & and decoder 'D blocks. For the classifier C, nuisance estimator N, and adversary A, we use a multi-layer perceptron (MLP) having three layers, whose hidden nodes are doubled from the input dimension. We also use batch normalization (BN) and ReLU activation as listed in Table 3 Note that for a tabular data such as Stress datasets, CNN was replaced with 3-layer MLP having ReLU activation and dropout with ratio of 20%. Also the MLP classifier was replaced with CNN for 2D input dimension cases such as in the model A. The number of latent dimensions of X into the CNN encoder such as in the model Ds, dimension mismatch poses a problem. We address this issue by using linear layer project S into the temporal dimensional space of X and another linear layer to project it into the spatial dimensional space of X. The dot product of those two projected vectors is concatenated additional channel input. We use A, 0.01 for the regularization coefficient. We leave hyperparameter exploration to integrate AutoML and AutoBayes as remaining future work.
chosen Z 64. When we need to feed S along with 2D data
A.8 PERFORMANCE RESULTS
The additional results for the all datasets are listed in Table 4 The results suggest that the best inference strategy highly depends on datasets. Specifically, the best model at one dataset does not perform best for different datasets; e.g., the model varvaiiaaional Is was best for ASL dataset, while the model variational Ds was best for RSVP dataset. It suggests that we shall consider different inference strategies for each target dataset and AutoBayes provides such adaptive framework. Also note that reconstruction loss may not good indicator select the graph model. addition, huge performance gap between the best and worst models was observed for some datasets. For example, the task accuracy of 46 was achieved with model non-van vor-variaaional Dz for Faces (Noisy) dataset, whereas the model variational B offers 51.4%. This implics that may have potential risk that one particular model cannot achieve good performance if we do not explore different models.
A.9 SUBJECT VARIATION PERFORMANCE
For Stress dataset, there are SS| 20 subjects. As we have shown in Fig. 9).. we demonstrated that AutoBayes can improve robustness against the nuisance variation, i.e., subject ID S. In Fig. 10 we show that the task classification accuracy highly depends on the subject ID S. Here, the box-whisker plots shows the accuracy distribution over different models from A to Kz. The outliers are identified by whisker factor of 2.4 with respective an inter-quartile range. It is that some users (e.g..
7

---

Under review as a conference paper at ICLR 2021
Table 4: Performance of datasets: the reconstruction loss, the scores of nuisance classification and task classification in variationallnon-varaatoooo and addersarial/non-advveere setting.
Reconstraction (dB)
Classification (%)
Classification
Complesity
Non- Von-Variational
Variational
Nou- Variational
Variational
Nao- Vari-iaaato
Visiational
No. Parameters
Clock Time
Model
-51.33
99.02
290K
01 :
Model
65.66
61.62
98. 72
99.44
978K
U1
Model Cs
66. 38
13.12
99.32
3.56M
01 :0:000
Medel Cy
67.74
12.17
99.30
3.53M
01 4:48
Model DJ
57.14
41.43
10.50
9.90
99.35
99.23
3.43M
01 : 54 06
Model 1tx
66.04
66.74
0.44
0.46
99.16
99.27
1.03M
01
37
Model
-65.35
-66.88
11.27
10.51
99.
99.21
4.17M
01 55:
65.51
61.41
90.35
90.13
-3.39
99.34
02
Model
QMNIST
Model
65.68
-42.05
9.71
4.20M
Model
66,6600
44.32
15.94
3.49M
Model He
65.85
43.45
13.20
14.70
99.47
99.28
3.49M
02
Model Is
65.35
-4.41
16.96
18.57
99.46
99.32
4.28M
02:
Model lx
65.84
45.
14.97
15.45
99.54
99.28
4.28M
02 23
Model Js
39.02
-57.3
11.41
11.21
99.47
99.39
4.11M
02 28
Model Je
67.96
-61.51
6.44
5.02
98. 85
99.46
1.71M
02 14 :00
Model
65.61
63.35
11.59
1.16
99.49
99.10
4.12M
02 21
Model Kz.
67.33
-61.20
6.32
6.94
99.15
99.15
1.71M
02
Model A
56.31
86.87
32.7K
Model
-66.
59.
94.79
92.67
97.
00:
Model
-67.74
59.46
93.48
50.0K
Model Cy
-6.666
75.77
91.93
48.
00: 00 55
Model Ds
61.94
336004
59.90
28.37
93.26
83.70
90.3K
00 01
Medel Dz
66.02
48.40
81.17
36.21
94.22
79.78
99.0k
00: 01 03
Model Es
66.38
63.35
64.21
79.76
94.00
92.05
96.3K
00: 01 08
Model
64.73
59.25
90.85
91.92
96.02
30.00
99.7K
00: 01 46
Model Ps
64.73
38.00
68.45
40.74
94.07
87.90
94.4K
00:
Model
-66.94
33..57
83.25
5.18
94.02
87.24
1K
00:
Stress
67.96
25.58
93.61
97.3K
00:
Medel
-6.000
94.4K
00:
Model
Madel
66.74
47.94
79.51
74.38
94.74
91.94
96.4K
Model
-6.77.00 07.98
-47.90
84.46
68.63
94.90
90.52
103K
Model Is
-07.13
37.117
79.36
92.47
9.5.85
30.00
140K
00:
Model Jz
66.74
54.02
86.27
58.59
96.17
86.99
1358
00
Model Ks.
68.64
51.00
73.07
87.33
94.65
80.74
14GK
00 01
Model Kz.
66.58
51.94
85.00
61.84
94.33
86.34
141K
00 02
30.09
93.00
Model
01:
Model
31.33
437K
Model
331557
90.38
91.84
433K
Model
33.61
30.17
91.33
84.77
00
Model
335.27
335
92.42
86.84
92.44
92.71
1.87M
00 :8:
Madel Ea
33.61
31.44
91.74
90.46
03.23
92.99
2.02M
00: :4:
Model Ex
33.62
33.52
94.26
83.01
92.85
91.99
2.03M
01 18
Model Fs
33.60
30.17
91.03
90.38
92.15
93.27
2.06M
01 00 522
Model Pi
32.94
30.16
9.57
9.85
90.21
91.04
1.933
01 08
RSVP
Model Gs
35.78
31.24
92.17
92.90
89.83
86.82
2.03M
00 67 26
Model Gz
35.28
30.34
91.27
90.18
92.15
91.31
2.03M
00 : 32:220
-35400
30.18
03.89
91.31
93.05
91.22
2.06M
01 04:
35.39
92.65
92.76
Model
Model
Modil
36.10
36.09
Model
35.82
36.66
93.60
3.49M
Model
-35.65
33.005
90.93
92.86
93.19
90.54
3.65M
01
Model Ks.
-35.53
33.00
91.99
82.10
02.81
93.03
3.50M
00 58: 04
8

---

Under review as a conference paper at ICLR 2021
Table 4: Performance of datasets (continued)
Datase
Method
Recomstructon Loss (dB)
Nuisance Classification (%)
Tisk Classification (%)
Model Complesity
Non-Varitional
Variationl
Non-Variationl
Variationl
Nou-Variatioaal
Varistional
Clock Time
Model
-00.28
55.85
454K
66.32
47.61
6.29M
Model Cts
-32.12
35.99
62.65
5.80M
Model Cy
-32.15
43.60
52.98
5.84M
Model Ds
-31.34
200.00
74.15
1.14
24.26
24.89
10, 9M
03 25 51
Model Dz
32.14
96.92
4.82
9.01
65.26
01.80
6.30M
02 57
Model Es
-32.22
-30.90
61.95
0.74
44.74
24.85
11. 7M
03 25 11
Model Ex
33.52
-30.82
5.77
8.21
54.12
48.85
11. TM
04 43 56
Model Fs
-3036
200.35
38.60
0.66
48.90
51.91
11.2M
04 16
Model Fx
31.86
29.77
3.05
0.96
67.88
26.40
6.64M
D4 24
MI
Model Gs
-32,
-30.07
33.97
0.55
53.01
24.82
11.7M
03 : 30: 37
Ci
-32311
-30.06
0.96
62.61
26.40
03
Model Hs
-30.88
88.42
57.87
62.68
49.04
15:
Hz
-30.02
43.93
67.21
26.96
21.2M
-32.27
$4.99
55.00
24.26
04:
Medel to
-32.55
48.49
1.08
63.57
26.03
Mo
-30.10
49.19
41.54
24.93
Model Je
32.88
-35.
43.64
31.10
57.50
44.93
3M
03 59
Model Ks
330.79
-30.15
61.18
0.77
23.79
25.18
17 00.0
04 00
Model K>
32.27
332.4
29.26
28.31
48.12
48.79
12.3M
03 43
A
-31.04
69.89
301K
00 : 48
B
41.26
71.81
71.39
3.40M
D1
94.95
63.68
Model Cy
-41.11
98.98
70.07
1.06M
59.099000
Model Ds
98.66
4.04M
Model Dz
42.62
97.30
68.09
76.91
3.41M
Model Es
-99.49
-38.91
97.12
92.91
70.01
65.38
4.14M
00: 59:
Model Fo
-41.77
-11.1 11.98
47.18
99.64
70.91
72.42
4.15M
01 18:
Model Is
39.54
-30.00
98.32
6.73
71.45
70.07
4.13M
D1 09
Model Fe
-41.35
-30. 10
93.35
8.35
66.71
70.19
3.50M
01 17: 28
EnP
Model Gs
40.23
-33.96
97.00
0.42
70.85
70.31
4.14M
00: 56 37
Madel Gr
41.02
22.90
96.67
98.68
69.23
67.91
4.15M
00 : 57 12
Model Hs
40.03
22.32
98.14
98.02
67.85
29.93
4.13M
01 10. 47
Model He
-1..9
-9.00
96.81
97.12
68.81
69.11
4.13M
01 : 05:00.3
ls
35.09
-30.07
98.26
96.33
69.02
67.31
01 07: 48
40.54
96.21
96.33
70.25
66.95
10
40.33
08.37
05.08
7.21M
-41.2
72.54
69.29
6.54M
Modal Ks
-38.85
-37.71
98.86
5.77
68.63
69.29
7.22M
Model Kz
42.488
-40.5
98.32
95.01
72.36
69.85
6.50M
05

---

Under review as a conference paper at ICLR 2021
Table 4: Performance of datasets (continued)
Doiset
Method
Reconstraction Loss (di))
Naisance Clasudication (%6
Task Classifiction (98)
Model Complexity
Non: Variational
Variaticcal
Non Variatiocal
Variational
Non Variational
Variational
No. Porameters
Clock Time
Model
63.30
332K
22
Model
33.08
30.10
48.66
6.27M
Model
-32.18
80.45
64.50
960K
00
Medel Cly
32.96
87.26
65.62
954K
00
Model Ds
32.99
30.10
92.23
7.09
02.74
48.08
0.80M
00
Model Dy
331.68
23.37
86.70
7.77
66.99
49.28
5.28M
00.
Model Es
-31.98
-30.08
92.95
6.78
50.96
53.12
5.88M
00. 47
Model
31.84
30.03
38.94
97.60
50.96
51.36
6.91M
01 00
Model
33.92
30.11
96.07
5.09
61.14
62.82
5.93M
00 04
-32.95
-2.880
49.60
10.02
61.30
61.14
5.40M
01:
Faces Basic
32.66
00:
33.13
30.11
30.08
Model
31.63
30.11
97.92
94.39
61.94
6.01M
Model
333
-30.06
91.67
67.31
6.01M
Model Js
-33.28
-30.12
94.87
8.33
51,04
52.23
10.9M
00:
Model
-32.21
-29.50
93.83
7.29
65.79
51,28
10.3M
Model Ks
31.12
-29.88
88.94
7.45
51.92
53.85
10.9M
00.
Model Kz
32.69
-30.09
93.43
7.93
51.76
51.84
10.3M
00. 56 56:
Model
73.59
00:
Model
30.10
664K
90.56
662K
Model
27.90
15.28
5.53M
Model
-10.11
330.09
96.09
14.38
76.41
53.91
5.28M
00.
Modd Es
30.09
28.70
91.09
13.28
74.38
52.50
5.59M
00
Model Ez
30.47
28.58
21.41
93.75
70.94
52.97
5.61M
00
Model Is
30.14
30.08
95.82
13.75
71.88
75.62
5.68M
00 32
Faces Noisy
Model Fz
29.96
27.76
27.50
17.03
72.50
72.19
5.40M
00 40
Model
28.46
30.15
93.75
13.91
71.66
52.60
5.59M
00. 30.
Model
30.59
-30.09
94.53
80.94
75.00
75.18
5.61M
00
30.04
30.008
75.59
69.06
00: 31:
30.30
96.94
00:
30.04
97.97
00:
Model
Model Jz
30.07
30.03
14.22
10.3M
Model
300 29
-30.14
65.62
15.52
54.08
53.44
10.0M
Model
30..22
-28445
94.84
12.68
76.56
54.23
10.3M
Model A
24.22
41.69
588K
01: 18.
Model
23.89
-24.08
37.80
1.53M
01:
03.63
01:
01:
-4.07
01:
16:
Madel
24.07
24.07
94.00
1.63M
Model
24.90 69.96
24.10
43.16
85.45
37.23
1.63M
Modil Fs
24.07
- 24.08
93.93
97.58
38.75
42.27
2.00M
01:
Modil Fz
24.08
24.08
9.99
10.79
28.
42.36
1.89M
03. 00
Model Gs
-24.07
-24.08
94.45
93.81
38.81
39.83
1.65M
01:
Model Gx
22..50
24.81
98.89
94.76
47.43
43.32
1.63M
01: 27
Model Hts
25.10
-24.08
96.61
94.26
49.30
36.39
2.00M
01: 39
Model
24.87
24.08
94.77
94.20
48.31
37.33
2.00M
01: 4.6
Is
-24.87
-24.08
96.54
94.37
51.12
38.31
2.01M
01: 39:
01: 15:::45
Model
-24.07
-24.11
97.09
36.77
2.92M
01: 01:
24.09
24.11
2.93M
01:
Medel
24.22
-24.22
12.34
2.80M
01: :..00:40
10

---

Under review as a conference paper at ICLR 2021
S 8) have superior performance whereas classification task is harder for other users (e.g..
= 6). Our AutoBayes can well resolve the issues of such a nuisance variation by linking the adversarial block for S-independent latent variables Z to generate subject-invarian feature.
8
8
Accuracy (N)
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18 19
20
Subject
Figure 10: Task classification accuracy across subject ID for Stress dataset.
A.10 TIME COMPLEXITY ANALYSIS
In Fig. b)), we have shown the accuracy vs. the space complexity. In this section, we evaluate the time complexity in Figs. I a)) and (b), which show the task classification accuracy as function of computation time for training and testing, respectively, for the Stress dataset. As in the same setting of Fig. (b) explored different DNN configurations for the models A, B, and Js, by sweeping the number of hidden layers and hidden nodes. Some Pareto-front DNN configurations having lower complexity and higher accuracy are connected with lines. We used pytorch on NVIDIA Tesla K80 GPU with CUDA 1..1. It seen that the standard classifier model A outperforms the other models in lower complexity regimes, whereas our AutoBayes regimes. should also noted that the increase the time complexity is not so significant (by few folds) in comparison to that of the space complexity (by few magnitudes) in Fig, (b).
achieve better Pareto front for higher accuracy
90
90
Accuracy (%)
Accuracy (%)
80
80
Ensemble Autolkaye AunoBayes VAE
Ensemble AutoBayee
AutoBayes A VAE Standad Classifer
Standard Clasitier
50
100
150
200
250
300
10
15
20
Training Clock
Testing Clock
Figure 11: Task classification accuracy function of time complexity for Stress dataset.
11


Reviews:
[{"review_id": "ecOgK7MiCg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a framework, called AutoBayes, to automatically\ndetect the conditional relationship between data features (X), task\nlabels (Y), nuisance variation labels (S), and potential latent\nvariables (Z) in DNN architectures.  Assuming a Bayesian network (BN)\nwhich represents the (possibly) conditional independencies between the\naforementioned variables, the authors propose a learning algorithm\nwhich consists of applying Bayes-ball to detect and prune unnecessary\nedges in the graph (effectively finding a subgraph, independence map\nof the BN), train the resulting DNN architecture, and choose the network\nwhich achieves the highest validation performance.  This idea is\ninteresting, especially compared to hyperparameter optimization\napproaches for model tuning, and the results seem convincing.\n\nHowever, relevant previous work is not cited and discussed in the paper.\nSpecifically, BN structure learning and inference in BNs (both of\nwhich are well studied and have extensive literature) are fully\nrelevant, but are not discussed or mentioned at all.  For instance,\nthe paper uses undefined terms such as \"Bayesian graph model,\" \"Bayesian\ngraphs,\" and \"graph model,\" in place of Bayesian network (which is\nrigorously defined).  It is important that such related previous work\nbe discussed to delineate what is novel in the presented approach and\nplace its contributions within the greater context of this previous\nwork.  This inclusion would also help the presentation of concepts in\nthe paper.  For instance, the discussion surrounding equations 1\nand 2, i.e.:\n\"The chain rule can yield the following factorization for a generative\nmodel from Y to X (note that at most 4! factorization orders exist\nincluding useless ones such as reverse direction from X to Y )...,\" is\nthe concept of an elimination ordering in the elimination\nalgorithm for BNs (and graphical models in general).  Showcasing the\npresented work in this light, (i.e., as a natural\ncombination of BN structure learning with macro-level neural-architecture\noptimization) would be particularly novel and compelling.\n\nFinally, it is important to discuss the complexity of the presented\nalgorithm.  Given the Bayesian networks (BNs) in Algorithm 1,  each\nindependence map (and the underlying DNN\narchitecture) must be trained then validated.  This algorithm scales\nfactorially in the number of nodes in the BN.  It is great that the\nselected subgraph performs so well (Figure 2), but super-exponential\ncomplexity multiplied by DNN cross-validation training is going to be\nvery hard to do as m and n grow in Algorithm 1.\n\nOther comments:\n\n-The definition of nuisance and nuisance-variables are implicitly\nassumed throughout the paper.  An exact definition of what is meant by\nnuisance, in the context, of the paper would be very helpful.\n\n-Algorithm 1 is mostly one large block of text, and is very hard to\nparse on the first read.\n\n-In the main contributions enumerated from pages 2-3, contribution 3\nlooks redundant given contribution 1.\n\n-\"Besides fully-supervised training, AutoBayes can automatically build some relevant graphi-\ncal models suited for semi-supervised learning.\" <- please include a\nlink to where this is discussed.  The enumerated list of contributions\nwould be a perfect roadmap for the paper (just include references to\nsections after every contribution)\n\n-\"We note that this paper relates to some existing literature... as addressed in Appendix A.1. Nonetheless, AutoBayes\nis a novel framework that diverges from AutoML, which is mostly employed to architecture tuning\nat a micro level.  Our work focuses on exploring neural architectures at a macro level, which is not\nan arbitrary diversion, but a necessary interlude.\" <- Appendix A.1\nshould really be included in the\npaper, related work is a not an optional section.  For instance, the\nreader may not know what the authors mean in terms of micro versus\nmacro level.  Reading this without further explanation until later\nin the paper, it would\nseem that micro-level is more nuanced than macro-level and the former\nperhaps subsumes the latter; the authors should detail what they mean\nand distinguish their work from previous works in the main paper.\n\n-The terminology is very clumsy: \"Bayesian graph models,\" \"Bayesian\ngraph,\" and \"graph model\" are not established terms and, as such,\nshould be defined so the reader knows what type of ML method is being discussed.  The authors should specify that they have a Bayesian\nnetwork whose factorization describes the conditional relationship\nbetween (random) variables.\n\n-\"VAE Evidence Lower Bound (ELBO) concept\" <- please include citation\nfor the ELBO\n\n-\"How to handle the exponentially growing search space of possible\nBayesian graphs along with the number of random variables remains a\nchallenging future work.\" <- this is exactly structure learning in\nBayesian networks (see the Bayesian information criterion, i.e., BIC score).", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y6OJYJ9_pJp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a novel method dubbed AutoBayes that tries to find optimal Bayesian graph models for \"nuisance-robust\" deep learning. They employ the Bayes-Ball algorithm to construct reasonable inference graphs from a generative model given by iterative search. The corresponding DNN modules are then built/linked and trained using a form of variational inference with adversarial regularization where applicable. The authors also propose the use of an ensembling approach to further improve robustness of the \"best\" model.\n\nPros:\n- Interesting and novel approach\n- Experimental results are encouraging and seem to demonstrate a clear advantage of AutoBayes over other methods\n\nCons:\n- The organization of the paper is erratic; overall structure doesn't flow well.\n- Multiple figures have details (e.g. color or style of arrows, shading, etc.) which are never explained.\n- Little to no context is given for what nuisance-robust learning (and correspondingly, nuisance \"variation\" variables) actually is. Similarly for the datasets and baselines.\n- No discussion of time/space complexity or computational demands, which is surprising given the apparent combinatorial complexity of the nested for-loops in Algorithm 1.\n- Placement with respect to existing work is somewhat vague (the authors refer to \"similarities\" and \"relationships\" but do not concretely describe them).\n- The paper needs to be proof read several more times. There are numerous grammatical errors and poorly phrased sentences. I will give a few examples below, but this is largely up to the authors to sort out.\n\nOverall, the paper reads more like a rough draft of a technical report than a standalone research article. It's confusing, difficult to read (I found that I needed to jump around and re-read a lot in order to understand what the authors were trying to say), and fails to give necessary context to readers who aren't familiar with a very specific subset of the deep learning literature. While it's of course always fine to defer to references for details, it's important that the reader can broadly understand your method and the problem it's trying to solve without hunting down a dozen references. This paper seems to assume that the reader has read every relevant paper and is arriving at this work in sequence.\n\nIf the authors are willing to do some reorganizing and more diligent proof-reading, I'd be happy to reconsider my rating after seeing the revisions.\n\nA couple of examples of poorly phrased or grammatically incorrect sentences:\n\n\"It may be because the probabilistic ~relation~ *relationships* *in the* underlying data ~varies~ vary ~over~ across datasets\" (pg 2)\n\n\"The ~whole~ DNN blocks are trained with ~adversary~ *adversarial* learning ~in a~ *using* variational Bayesian inference.\" (pg.3)", "extra_scores": {}, "extra_text": {}}, {"review_id": "L5EWXWSFN63", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe paper presents AutoBayes: a new approach for nuisance-robust deep learning which explores different Bayesian graph models to search for the best inference strategy. It automatically builds connections between classifier, encoder, decoder, nuisance estimator and adversary DNN blocks. The approach also enables disentangling the learned representations in terms of nuisance variation and task labels. Different benchmark datasets have been used for evaluation.\n \n##################################################################\n\nStrenghts:\n- The idea of automatically exploring various graphical models to select the best performing one is interesting.  \n- Overall, the paper is well written and easy to read.\n- Ensemble learning is performed by stacking the explored graphical models allowing to improve performance.\n\n##################################################################\n\nWeaknesses:\n- My main concern about the paper is that even though quantitative evaluation has been performed on several datasets with different modalities to show the benefit of using AutoBayes, the experimental evaluation section is not convincing enough as it lacks interpretation and analysis of the obtained results. For instance, one major problem addressed in this paper is models robustness to nuisance factors, however this was not discussed in this section. Hence, it would be good to include an experimental evaluation on this point. \n- In Table 4 in Appendix, the simple Model B which assumes independence between Z and S performs remarkably well on task classification of the 5 first datasets since it outperforms state-of-the-art methods on all of them except QMNIST and achieves classification accuracies that are very close to the best ones (with either variational or non-variational inference), outperforming most of the presented graphical models. On the remaining datasets, Model A which is independent of S and Z, performs well compared to other graphical models especially on Faces Basics and Faces Noisy datasets. Considering these observations and perhaps the time consumption of AutoBayes, what would be the motivation of exploring all the presented graphical models (besides stacking them for ensemble learning) rather than using the simplest models A or B with good hyperparameter search? Could these results be explained by the fact that potentially some of these datasets do not present high nuisance variabilities?\n- One of the main contributions as presented in Section 2 is the extensibility of the proposed framework to multiple latent representations and nuisance factors. Although this was demonstrated theoretically, it would be interesting to demonstrate this experimentally.\n- One of the properties of AutoBayes is its ability to learn disentangled representations in terms of nuisance factors. In practice, how can this be evaluated?\n\nMinor comments: \n- Typo in Equation 6: xˆ  = p_μ(z1, z2) instead of xˆ  = p_μ(z1)?\n- Steps to derive Equation 7 are not straightforward and can be more clarified.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DXaRFaZHsJi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update**\nI have read the author's rebuttal, and happy to see that a discussion regarding parameters is added (Figure 9). Other than that, my personal concern is similar to Anon Review 3's -- it seems that the core idea of the paper is drowned in too many technical details (granted, many of these are needed in order to implement this correctly). I wonder if a clearer discussion can be made like this -- you have a variational inference problem with certain independence assumptions, so write this out in the most abstract manner possible. To come up with a concrete objective, the question then becomes \"given the factor graph, how do we add the networks (this basically lines 11 - 16 in Algorithm but not in the flow of the main text)\". I think a better presentation and clarity in the main paper would greatly help acceptance.\n\n**Overview**\nThe paper proposes AutoBayes, which enumerates all the plausible graphical models between data, label, and nuisance variables, remove redundant edges with d-separation rules, and learns neural networks to represent the parent-to-child information.\n\n**Strengths**\nEmpirical results are very strong for certain EEG datasets such as ErrP and RSVP. It appears that on the EEG datasets, different graph structures would have very different performances so there might not be a graph structure that works equally well on all of them, which motivates the need for searching for the optimal graph structure.\n\n**Weaknesses**\n\nScalability: unlike existing methods in AutoML, AutoBayes does not seem to attempt to optimize the process from which graphs are selected (i.e. pruning of graphs that are unlikely to work well), resulting in the need to enumerate all possible graphs (where the complexity is doubly-exponential with respect to the number of variables). This means that the method will have difficulty even scaling to a small amount of variables (e.g. 10).\n\nInterpretation of the learned structures: it seems that on some datasets, CVAE already provides comparable performance to the best AutoBayes architecture, and on others best AutoBayes architecture perform much better. Can we extract any insight from the AutoBayes procedure?\n\nFairness of experiments: for each component of the graph, the network structure is the same; therefore, compared to a structure X->Z->Y, we have fewer parameters if we use X->Y. It is unclear as to whether the empirical improvement can be gained simply by representing edges with larger networks.\n\nEmpirical comparison: it appears that model ensembles have a significant effect over the performance. However, since one can also ensemble models from different initializations but the same architecture, it is unclear what the performance gap would become if we also do ensemble on one model but with different parameters.\n\n**Minor suggestion**\nPerhaps spend some text describing what is special about EEG datasets, and why do we expect having to predict nuisance variables to improve performance.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a method for identifying appropriate graphical models through enumeration, pruning of redundant dependencies, and neural network conditionals. While structure learning is an interesting application and there are some promising results, there were a number of concerns around experimental evaluation, computational complexity of the method, clarity of the presentation, and connections to prior work. In particular, R1's concerns around the large field of structure learning in Bayesian Networks, and unwillingness to use the established terminology (and comparing to methods there) was not sufficiently addressed in the rebuttal.
Decision: Reject