AutoBayes: Automated Bayesian Graph Exploration for Nuisance-Robust Inference

1 INTRODUCTION . The great advancement of deep learning techniques based on deep neural networks ( DNN ) has enabled more practical design of human-machine interfaces ( HMI ) through the analysis of the user ’ s physiological data ( Faust et al. , 2018 ) , such as electroencephalogram ( EEG ) ( Lawhern et al. , 2018 ) and electromyogram ( EMG ) ( Atzori et al. , 2016 ) . However , such biosignals are highly prone to variation depending on the biological states of each subject ( Christoforou et al. , 2010 ) . Hence , frequent calibration is often required in typical HMI systems . Toward resolving this issue , subject-invariant methods ( Özdenizci et al. , 2019b ) , employing adversarial training ( Makhzani et al. , 2015 ; Lample et al. , 2017 ; Creswell et al. , 2017 ) with the Conditional Variational AutoEncoder ( A-CVAE ) ( Louizos et al. , 2015 ; Sohn et al. , 2015 ) shown in Fig . 1 ( b ) , have emerged to reduce user calibration for realizing successful HMI systems . Compared to a standard DNN classifier C in Fig . 1 ( a ) , integrating additional functional blocks for encoder E , nuisanceconditional decoder D , and adversary A networks offers excellent subject-invariant performance . The DNN structure may be potentially extended with more functional blocks and more latent nodes as shown in Fig . 1 ( c ) . However , such a DNN architecture design may rely on human effort and insight to determine the block connectivity of DNNs . Automation of hyperparameter and architecture exploration in the context of AutoML ( Ashok et al. , 2017 ; Brock et al. , 2017 ; Cai et al. , 2017 ; He et al. , 2018 ; Miikkulainen et al. , 2019 ; Real et al. , 2017 ; 2020 ; Stanley & Miikkulainen , 2002 ; Zoph et al. , 2018 ) can facilitate DNN design suited for nuisance-invariant inference . Nevertheless , without proper reasoning , most of the search space for link connectivity will be pointless . In this paper , we propose a systematic automation framework called AutoBayes , which searches for the best inference graph model associated with a Bayesian graph model ( also a.k.a . Bayesian network ) well-suited to reproduce the training datasets . The proposed method automatically formulates various different Bayesian graphs by factorizing the joint probability distribution in terms of data , class label , subject identification ( ID ) , and inherent latent representations . Given Bayesian graphs , some meaningful inference graphs are generated through the Bayes-Ball algorithm ( Shachter , 2013 ) for pruning redundant links to achieve high-accuracy estimation . In order to promote robustness against nuisance variations such as inter-subject/session factors , the explored Bayesian graphs can provide 1For example of speech recognition , nuisance factors such as speaker ’ s attributes and recording environment may change the task accuracy . For image recognition , ambient light conditions and image sensor conditions may become inherent nuisance factors . In the context of this paper , nuisance variations mainly refer to subject identities and biological states during recording sessions for physiological data learning . reasoning to use adversarial training with/without variational modeling and latent disentanglement . We demonstrate that AutoBayes can achieve excellent performance across various public datasets , in particular with an ensemble stacking of multiple explored graphical models . 2 KEY CONTRIBUTIONS . At the core of our methodology is the consideration of graphical models that capture the probabilistic relationship between random variables representing the data features X , task labels Y , nuisance variation labels S , and ( potential ) latent representations Z . The ultimate goal is to infer the task label Y from the measured data feature X , which is hindered by the presence of nuisance variations ( e.g. , inter-subject/session variations ) that are ( partially ) labelled by S. One may use a standard DNN to classify Y given X as shown in Fig . 1 ( a ) , without explicitly involving S or Z . Although A-CVAE in Fig . 1 ( b ) may offer nuisance-robust performance through adversarial disentanglement of S from latent Z , there is no guarantee that such a model can perform well across different datasets . It is exemplified in Fig . 2 where A-CVAE outperforms the standard DNN model for some datasets ( QMNIST , Stress , ErrP ) while it does not for the other cases . This may be due to the underlying probabilistic relationship of the data varying across datasets . Our proposed framework can construct justifiable models , achieving higher performance for every dataset , as demonstrated in Fig . 2 . It is verified that significant gain is attainable with ensemble methods of different Bayesian graphs which are explored in our AutoBayes . For example , our method with a relatively shallow architecture achieves 99.61 % accuracy which is close to state-of-the-art performance in QMNIST dataset . The main contributions of this paper over the existing works are five-fold as follows : Algorithm 1 Pseudocode for AutoBayes Framework Require : Nodes set V = [ Y , X , S1 , S2 , . . . , Sn , Z1 , Z2 , . . . , Zm ] , where Y denotes task labels , X is a measurement data , S = [ S1 , S2 , . . . , Sn ] are ( potentially multiple ) semi-supervised nuisance variations , and Z = [ Z1 , Z2 , . . . , Zm ] are ( potentially multiple ) latent vectors Ensure : Semi-supervised training/validation datasets 1 : for all permutations of node factorization from Y to X do 2 : Let B0 be the corresponding Bayesian graph for the permuted full-chain factorization 3 : for all combinations of link pruning on the full-chain Bayesian graph B0 do 4 : Let B be the corresponding pruned Bayesian graph 5 : Apply the Bayes-Ball algorithm on B to build a conditional independency list I 6 : for all permutations of node factorization from X to Y do 7 : Let F0 be the factor graph corresponding to a full-chain conditional probability 8 : Prune all redundant links in F0 based on conditional independency I 9 : Let F be the pruned factor graph 10 : Merge the pruned Bayesian graph B into the pruned factor graph F 11 : Attach an adversary network A to latent nodes Z for Zk ⊥ S ∈ I 12 : Assign an encoder network E for p ( Z| · · · ) in the merged factor graph 13 : Assign a decoder network D for p ( x| · · · ) in the merged factor graph 14 : Assign a nuisance indicator network N for p ( S| · · · ) in the merged factor graph 15 : Assign a classifier network C for p ( y| · · · ) in the merged factor graph 16 : Adversary train the whole DNN structure to minimize a loss function in ( 5 ) 17 : end for . At most ( |V| − 2 ) ! combinations 18 : end for . At most 2|V| ( |V|−1 ) /2 combinations 19 : end for . At most ( |V| − 2 ) ! combinations 20 : return the best model having highest task accuracy in validation sets • AutoBayes automatically explores potential graphical models inherent to the data by combinatorial pruning of dependency assumptions ( edges ) and then applies Bayes-Ball to examine various inference strategies , rather than blindly exploring hyperparameters of DNN blocks . • AutoBayes offers a solid reason of how to connect multiple DNN blocks to impose conditioning and adversary censoring for the task classifier , feature encoder , decoder , nuisance indicator and adversary networks , based on an explored Bayesian graph . • The framework is also extensible to multiple latent representations and nuisances factors . • Besides fully-supervised training , AutoBayes can automatically build some relevant graphi- cal models suited for semi-supervised learning . • Multiple graphical models explored in AutoBayes can be efficiently exploited to improve performance by ensemble stacking . We note that this paper relates to some existing literature in AutoML , variational Bayesian inference ( Kingma & Welling , 2013 ; Sohn et al. , 2015 ; Louizos et al. , 2015 ) , adversarial training ( Goodfellow et al. , 2014 ; Dumoulin et al. , 2016 ; Donahue et al. , 2016 ; Makhzani et al. , 2015 ; Lample et al. , 2017 ; Creswell et al. , 2017 ) , and Bayesian network ( Nie et al. , 2018 ; Njah et al. , 2019 ; Rohekar et al. , 2018 ) as addressed in Appendix A.1 in more detail . Nonetheless , AutoBayes is a novel framework that diverges from AutoML , which mostly employs architecture tuning at a micro level . Our work focuses on exploring neural architectures at a macro level , which is not an arbitrary diversion , but a necessary interlude . Our method focuses on the relationships between the connections in a neural network ’ s architecture and the characteristics of the data ( Minsky & Papert , 2017 ) . In addition to the macro-level structure learning of Bayesian network , our approach provides a new perspective in how to involve the adversarial blocks and to exploit multiple models for ensemble stacking . 3 AUTOBAYES . AutoBayes Algorithm : The overall procedure of the AutoBayes algorithm is described in the pseudocode of Algorithm 1 . The AutoBayes automatically constructs non-redundant inference factor graphs given a hypothetical Bayesian graph assumption , through the use of the Bayes-Ball algorithm . Depending on the derived conditional independency and pruned factor graphs , DNN blocks for encoder E , decoder D , classifier C , nuisance estimator N and adversary A are reasonably connected . The entire network is trained with variational Bayesian inference and adversarial training . The Bayes-Ball algorithm ( Shachter , 2013 ) facilitates an automatic pruning of redundant links in inference factor graphs through the analysis of conditional independency . Fig . 3 shows ten Bayes-Ball rules to identify conditional independency . Given a Bayesian graph , we can determine whether two disjoint sets of nodes are independent conditionally on other nodes through a graph separation criterion . Specifically , an undirected path is activated if a Bayes ball can travel along without encountering a stop symbol : in Fig . 3 . If there are no active paths between two nodes when some conditioning nodes are shaded , then those random variables are conditionally independent . Graphical Models : We here focus on 4-node graphs . Let p ( y , s , z , x ) denote the joint probability distribution underlying the datasets for the four random variables , i.e. , Y , S , Z , and X . The chain rule can yield the following factorization for a generative model from Y to X ( note that at most 4 ! factorization orders exist including useless ones such as with reversed direction from X to Y ) : p ( y , s , z , x ) = p ( y ) p ( s|y ) p ( z|s , y ) p ( x|z , s , y ) , ( 1 ) which is visualized in Bayesian graph of Fig . 4 ( a ) . The probability conditioned on X can then be factorized , e.g. , as follows ( among 3 ! different orders of inference factorization for four-node graphs ) : p ( y , s , z|x ) = { p ( z|x ) p ( s|z , x ) p ( y|s , z , x ) , Z-first-inference p ( s|x ) p ( z|s , x ) p ( y|z , s , x ) , S-first-inference ( 2 ) which are marginalized to obtain the likelihood : p ( y|x ) = Es , z|x [ p ( y , s , z|x ) ] . The above two scheduling strategies in ( 2 ) are illustrated in factor graph models as in Figs . 4 ( b ) and ( c ) , respectively . The graphical models in Fig . 4 do not impose any assumption of potentially inherent independency in datasets and hence are most generic . However , depending on the underlying independency in datasets , we may be able to prune some edges in those graphs . For example , if the data only follows the simple Markov chain of Y −X , while being independent of S and Z , as shown in Fig . 5 ( a ) , all links except one between X and Y will be unreasonable in inference graphs of Figs . 4 ( b ) and ( c ) , that justifies the standard classifier model in Fig . 1 ( a ) . This implies that more complicated inference models such as A-CVAE can be unnecessarily redundant depending on the dataset . This motivates us to consider an extended AutoML framework which automatically explores the best pair of inference factor graph and corresponding Bayesian graph models matching dataset statistics besides the micro-scale hyperparameter tuning . Methodology : AutoBayes begins with exploring any potential Bayesian graphs by cutting links of the full-chain graph in Fig . 4 ( a ) , imposing possible ( conditional ) independence . We then adopt the Bayes-Ball algorithm on each hypothetical Bayesian graph to examine conditional independence over different inference strategies , e.g. , full-chain Z-/S-first inference graphs in Figs . 4 ( b ) / ( c ) . Applying Bayes-Ball justifies the reasonable pruning of the links in the full-chain inference graphs , and also the potential adversarial censoring when Z is independent of S. This process automatically constructs the connectivity of inference , generative , and adversary blocks with sound reasoning . Consider an example case when the data adheres to the following factorization : p ( y , s , z , x ) = p ( y ) p ( s| y ) p ( z| s , y ) p ( x|z , s , y ) , ( 3 ) where we explicitly indicate conditional independence by slash-cancellation from the full-chain case in ( 1 ) . This corresponds to a Bayesian graphical model illustrated in Fig . 5 ( e ) . Applying the Bayes-Ball algorithm to the Bayesian graph yields the following conditional probability : p ( y , s , z|x ) = p ( z|x ) p ( s|z , x ) p ( y|z , s , x ) , ( 4 ) for the Z-first inference strategy in ( 2 ) . The corresponding factor graph is then given in Fig . 6 ( c ) . Note that the Bayes-Ball also reveals that there is no marginal dependency between Z and S , which provides the reason to use adversarial censoring to suppress nuisance information S in the latent space Z . In consequence , by combining the Bayesian graph and factor graph , we automatically obtain A-CVAE model in Fig . 1 ( b ) . AutoBayes justifies A-CVAE under the assumption that the data follows the Bayesian model E in Fig . 5 ( e ) . As the true generative model is unknown , AutoBayes explores different Bayesian graphs like in Fig . 5 to search for the most relevant model . Our framework is readily applicable to graphs with more than 4 nodes to represent multiple Y , S , and Z . Models J and K in Fig . 5 are such examples having multiple latent factors Z1 and Z2 . Despite the search space for AutoBayes will rapidly grow with the number of nodes , most realistic datasets do not require a large number of neural network blocks for macro-level optimization . See Appendix A.2 for more detailed descriptions for some Bayesian graph models to construct factor graphs like in Fig . 6 . Also see discussions of graphical models suitable for semi-supervised learning in Appendix A.4 . Training : Given a pair of generative graph and inference graph , the corresponding DNN structures will be trained . For example of the generative graph model K in Fig . 5 ( k ) , one relevant inference graph Kz in Fig . 6 ( k ) will result in the overall network structure as shown in Fig . 7 , where adversary network is attached as Z2 is ( conditionally ) independent of S. This 5-node graph model justifies a recent work on partially disentanged A-CVAE by Han et al . ( 2020 ) . Each factor block is realized by a DNN , e.g. , parameterized by θ for pθ ( z1 , z2|x ) , and all of the networks except for adversarial network are optimized to minimize corresponding loss functions including L ( ŷ , y ) as follows : min θ , ψ , µ max η E [ L ( ŷ , y ) + λsL ( ŝ , s ) + λxL ( x̂′ , x ) + λzKL ( z1 , z2‖N ( 0 , I ) ) − λaL ( ŝ′ , s ) ] , ( 5 ) ( z1 , z2 ) = pθ ( x ) , ŷ = pφ ( z1 , z2 ) , ŝ = pψ ( z1 ) , x̂ ′ = pµ ( z1 ) , ŝ ′ = pη ( z2 ) , ( 6 ) where λ∗ denotes a regularization coefficient , KL is the Kullback–Leibler divergence , and the adversary network pη ( s′|z2 ) is trained to minimize L ( ŝ′ , s ) in an alternating fashion ( see the Adversarial Regularization paragraph below ) . X Z S Y ( a ) Model Dz X S Z Y ( b ) Model Ds X Z S Y ( c ) Model Ez X Z S Y ( d ) Model Es X Z S Y ( e ) Model Fz X S Z Y ( f ) Model Fs The training objective can be formally understood from a likelihood maximization perspective , in manner that can be seen as a generalization of the VAE Evidence Lower Bound ( ELBO ) concept ( Kingma & Welling , 2013 ) . Specifically , it can be viewed as the maximization of a variational lower bound of the likelihood pΦ ( x , y , s ) that is implicitly defined and parameterized by the networks , where Φ represents the collective parameters of the network modules ( e.g. , Φ = ( φ , ψ , µ ) in the example of equation 5 ) that specify the generative model pΦ ( x , y , s|z ) , which implies the likelihood pΦ ( x , y , s ) , as given by pΦ ( x , y , s ) = ∫ pΦ ( x , y , s|z ) p ( z ) dz . However , since this expression is generally intractable , we introduce qθ ( z|x , y , s ) as a variational approximation of the posterior pΦ ( z|x , y , s ) implied by the generative model ( Kingma & Welling , 2013 ; Ranganath et al. , 2014 ) : 1 n n∑ i=1 log pΦ ( xi , yi , si ) = 1 n n∑ i=1 [ log pΦ ( xi , yi , si|zi ) −log qθ ( zi|xi , yi , si ) p ( zi ) +log qθ ( zi|xi , yi , si ) pΦ ( zi|xi , yi , si ) ] ≈ 1 n n∑ i=1 [ log pΦ ( xi , yi , si|zi ) ] −KL ( qθ ( z|x , y , s ) ‖p ( z ) ) + KL ( qθ ( z|x , y , s ) ‖pΦ ( z|x , y , s ) ) ≥ 1 n n∑ i=1 [ log pΦ ( xi , yi , si|zi ) ] −KL ( qθ ( z|x , y , s ) ‖p ( z ) ) , ( 7 ) where the samples zi ∼ qθ ( z|xi , yi , si ) are drawn for each training tuple ( xi , yi , si ) , and the final inequality follows from the non-negativity of KL divergence . Ultimately , the minimization of our training loss function corresponds to the maximization of the lower bound in ( 7 ) , which corresponds to maximizing the likelihood of our implicit generative model , while also optimizing the variational posterior qθ ( z|x , y , s ) toward the actual posterior for the latent representation pΦ ( z|x , y , s ) , since the gap in the bound is given by KL ( qθ ( z|x , y , s ) ‖pΦ ( z|x , y , s ) ) . Further factoring of log pΦ ( x , y , s|z ) yields the multiple loss-terms and network modules . Adversarial Regularization : We can utilize adversarial censoring when Z and S should be marginally independent , e.g. , such as in Fig . 1 ( b ) and Fig . 7 , in order to reinforce the learning of a representation Z that is disentangled from the nuisance variations S. This is accomplished by introducing an adversarial network that aims to maximize a parameterized approximation q ( s|z ) of the likelihood p ( s|z ) , while this likelihood is also incorporated into the loss for the other modules with a negative weight . The adversarial network , by maximizing the log likelihood log q ( s|z ) , essentially maximizes a lower-bound of the mutual information I ( S ; Z ) , and hence the main network is regularized with the additional term that corresponds to minimizing this estimate of mutual information . This follows since the log-likelihood maximized by the adversarial network is given by E [ log q ( s|z ) ] = I ( S ; Z ) −H ( S ) −KL ( p ( s|z ) ‖q ( s|z ) ) , ( 8 ) where the entropy H ( S ) is constant . Ensemble Learning : We further introduce ensemble methods to make best use of all Bayesian graph models explored by the AutoBayes framework without wasting lower-performance models . Ensemble stacked generalization works by stacking the predictions of the base learners in a higher level learning space , where a meta learner corrects the predictions of base learners ( Wolpert , 1992 ) . Subsequent to training base learners , we assemble the posterior probability vectors of all base learners together to improve the prediction . We compare the predictive performance of a logistic regression ( LR ) and a shallow multi-layer perceptron ( MLP ) as an ensemble meta learner to aggregate all inference models . See Appendix A.5 for more detailed description of the stacked generalization .

Summary: The authors present a novel method dubbed AutoBayes that tries to find optimal Bayesian graph models for "nuisance-robust" deep learning. They employ the Bayes-Ball algorithm to construct reasonable inference graphs from a generative model given by iterative search. The corresponding DNN modules are then built/linked and trained using a form of variational inference with adversarial regularization where applicable. The authors also propose the use of an ensembling approach to further improve robustness of the "best" model.