AutoBayes: Automated Bayesian Graph Exploration for Nuisance-Robust Inference

1 INTRODUCTION . The great advancement of deep learning techniques based on deep neural networks ( DNN ) has enabled more practical design of human-machine interfaces ( HMI ) through the analysis of the user ’ s physiological data ( Faust et al. , 2018 ) , such as electroencephalogram ( EEG ) ( Lawhern et al. , 2018 ) and electromyogram ( EMG ) ( Atzori et al. , 2016 ) . However , such biosignals are highly prone to variation depending on the biological states of each subject ( Christoforou et al. , 2010 ) . Hence , frequent calibration is often required in typical HMI systems . Toward resolving this issue , subject-invariant methods ( Özdenizci et al. , 2019b ) , employing adversarial training ( Makhzani et al. , 2015 ; Lample et al. , 2017 ; Creswell et al. , 2017 ) with the Conditional Variational AutoEncoder ( A-CVAE ) ( Louizos et al. , 2015 ; Sohn et al. , 2015 ) shown in Fig . 1 ( b ) , have emerged to reduce user calibration for realizing successful HMI systems . Compared to a standard DNN classifier C in Fig . 1 ( a ) , integrating additional functional blocks for encoder E , nuisanceconditional decoder D , and adversary A networks offers excellent subject-invariant performance . The DNN structure may be potentially extended with more functional blocks and more latent nodes as shown in Fig . 1 ( c ) . However , such a DNN architecture design may rely on human effort and insight to determine the block connectivity of DNNs . Automation of hyperparameter and architecture exploration in the context of AutoML ( Ashok et al. , 2017 ; Brock et al. , 2017 ; Cai et al. , 2017 ; He et al. , 2018 ; Miikkulainen et al. , 2019 ; Real et al. , 2017 ; 2020 ; Stanley & Miikkulainen , 2002 ; Zoph et al. , 2018 ) can facilitate DNN design suited for nuisance-invariant inference . Nevertheless , without proper reasoning , most of the search space for link connectivity will be pointless . In this paper , we propose a systematic automation framework called AutoBayes , which searches for the best inference graph model associated with a Bayesian graph model ( also a.k.a . Bayesian network ) well-suited to reproduce the training datasets . The proposed method automatically formulates various different Bayesian graphs by factorizing the joint probability distribution in terms of data , class label , subject identification ( ID ) , and inherent latent representations . Given Bayesian graphs , some meaningful inference graphs are generated through the Bayes-Ball algorithm ( Shachter , 2013 ) for pruning redundant links to achieve high-accuracy estimation . In order to promote robustness against nuisance variations such as inter-subject/session factors , the explored Bayesian graphs can provide 1For example of speech recognition , nuisance factors such as speaker ’ s attributes and recording environment may change the task accuracy . For image recognition , ambient light conditions and image sensor conditions may become inherent nuisance factors . In the context of this paper , nuisance variations mainly refer to subject identities and biological states during recording sessions for physiological data learning . reasoning to use adversarial training with/without variational modeling and latent disentanglement . We demonstrate that AutoBayes can achieve excellent performance across various public datasets , in particular with an ensemble stacking of multiple explored graphical models . 2 KEY CONTRIBUTIONS . At the core of our methodology is the consideration of graphical models that capture the probabilistic relationship between random variables representing the data features X , task labels Y , nuisance variation labels S , and ( potential ) latent representations Z . The ultimate goal is to infer the task label Y from the measured data feature X , which is hindered by the presence of nuisance variations ( e.g. , inter-subject/session variations ) that are ( partially ) labelled by S. One may use a standard DNN to classify Y given X as shown in Fig . 1 ( a ) , without explicitly involving S or Z . Although A-CVAE in Fig . 1 ( b ) may offer nuisance-robust performance through adversarial disentanglement of S from latent Z , there is no guarantee that such a model can perform well across different datasets . It is exemplified in Fig . 2 where A-CVAE outperforms the standard DNN model for some datasets ( QMNIST , Stress , ErrP ) while it does not for the other cases . This may be due to the underlying probabilistic relationship of the data varying across datasets . Our proposed framework can construct justifiable models , achieving higher performance for every dataset , as demonstrated in Fig . 2 . It is verified that significant gain is attainable with ensemble methods of different Bayesian graphs which are explored in our AutoBayes . For example , our method with a relatively shallow architecture achieves 99.61 % accuracy which is close to state-of-the-art performance in QMNIST dataset . The main contributions of this paper over the existing works are five-fold as follows : Algorithm 1 Pseudocode for AutoBayes Framework Require : Nodes set V = [ Y , X , S1 , S2 , . . . , Sn , Z1 , Z2 , . . . , Zm ] , where Y denotes task labels , X is a measurement data , S = [ S1 , S2 , . . . , Sn ] are ( potentially multiple ) semi-supervised nuisance variations , and Z = [ Z1 , Z2 , . . . , Zm ] are ( potentially multiple ) latent vectors Ensure : Semi-supervised training/validation datasets 1 : for all permutations of node factorization from Y to X do 2 : Let B0 be the corresponding Bayesian graph for the permuted full-chain factorization 3 : for all combinations of link pruning on the full-chain Bayesian graph B0 do 4 : Let B be the corresponding pruned Bayesian graph 5 : Apply the Bayes-Ball algorithm on B to build a conditional independency list I 6 : for all permutations of node factorization from X to Y do 7 : Let F0 be the factor graph corresponding to a full-chain conditional probability 8 : Prune all redundant links in F0 based on conditional independency I 9 : Let F be the pruned factor graph 10 : Merge the pruned Bayesian graph B into the pruned factor graph F 11 : Attach an adversary network A to latent nodes Z for Zk ⊥ S ∈ I 12 : Assign an encoder network E for p ( Z| · · · ) in the merged factor graph 13 : Assign a decoder network D for p ( x| · · · ) in the merged factor graph 14 : Assign a nuisance indicator network N for p ( S| · · · ) in the merged factor graph 15 : Assign a classifier network C for p ( y| · · · ) in the merged factor graph 16 : Adversary train the whole DNN structure to minimize a loss function in ( 5 ) 17 : end for . At most ( |V| − 2 ) ! combinations 18 : end for . At most 2|V| ( |V|−1 ) /2 combinations 19 : end for . At most ( |V| − 2 ) ! combinations 20 : return the best model having highest task accuracy in validation sets • AutoBayes automatically explores potential graphical models inherent to the data by combinatorial pruning of dependency assumptions ( edges ) and then applies Bayes-Ball to examine various inference strategies , rather than blindly exploring hyperparameters of DNN blocks . • AutoBayes offers a solid reason of how to connect multiple DNN blocks to impose conditioning and adversary censoring for the task classifier , feature encoder , decoder , nuisance indicator and adversary networks , based on an explored Bayesian graph . • The framework is also extensible to multiple latent representations and nuisances factors . • Besides fully-supervised training , AutoBayes can automatically build some relevant graphi- cal models suited for semi-supervised learning . • Multiple graphical models explored in AutoBayes can be efficiently exploited to improve performance by ensemble stacking . We note that this paper relates to some existing literature in AutoML , variational Bayesian inference ( Kingma & Welling , 2013 ; Sohn et al. , 2015 ; Louizos et al. , 2015 ) , adversarial training ( Goodfellow et al. , 2014 ; Dumoulin et al. , 2016 ; Donahue et al. , 2016 ; Makhzani et al. , 2015 ; Lample et al. , 2017 ; Creswell et al. , 2017 ) , and Bayesian network ( Nie et al. , 2018 ; Njah et al. , 2019 ; Rohekar et al. , 2018 ) as addressed in Appendix A.1 in more detail . Nonetheless , AutoBayes is a novel framework that diverges from AutoML , which mostly employs architecture tuning at a micro level . Our work focuses on exploring neural architectures at a macro level , which is not an arbitrary diversion , but a necessary interlude . Our method focuses on the relationships between the connections in a neural network ’ s architecture and the characteristics of the data ( Minsky & Papert , 2017 ) . In addition to the macro-level structure learning of Bayesian network , our approach provides a new perspective in how to involve the adversarial blocks and to exploit multiple models for ensemble stacking . 3 AUTOBAYES . AutoBayes Algorithm : The overall procedure of the AutoBayes algorithm is described in the pseudocode of Algorithm 1 . The AutoBayes automatically constructs non-redundant inference factor graphs given a hypothetical Bayesian graph assumption , through the use of the Bayes-Ball algorithm . Depending on the derived conditional independency and pruned factor graphs , DNN blocks for encoder E , decoder D , classifier C , nuisance estimator N and adversary A are reasonably connected . The entire network is trained with variational Bayesian inference and adversarial training . The Bayes-Ball algorithm ( Shachter , 2013 ) facilitates an automatic pruning of redundant links in inference factor graphs through the analysis of conditional independency . Fig . 3 shows ten Bayes-Ball rules to identify conditional independency . Given a Bayesian graph , we can determine whether two disjoint sets of nodes are independent conditionally on other nodes through a graph separation criterion . Specifically , an undirected path is activated if a Bayes ball can travel along without encountering a stop symbol : in Fig . 3 . If there are no active paths between two nodes when some conditioning nodes are shaded , then those random variables are conditionally independent . Graphical Models : We here focus on 4-node graphs . Let p ( y , s , z , x ) denote the joint probability distribution underlying the datasets for the four random variables , i.e. , Y , S , Z , and X . The chain rule can yield the following factorization for a generative model from Y to X ( note that at most 4 ! factorization orders exist including useless ones such as with reversed direction from X to Y ) : p ( y , s , z , x ) = p ( y ) p ( s|y ) p ( z|s , y ) p ( x|z , s , y ) , ( 1 ) which is visualized in Bayesian graph of Fig . 4 ( a ) . The probability conditioned on X can then be factorized , e.g. , as follows ( among 3 ! different orders of inference factorization for four-node graphs ) : p ( y , s , z|x ) = { p ( z|x ) p ( s|z , x ) p ( y|s , z , x ) , Z-first-inference p ( s|x ) p ( z|s , x ) p ( y|z , s , x ) , S-first-inference ( 2 ) which are marginalized to obtain the likelihood : p ( y|x ) = Es , z|x [ p ( y , s , z|x ) ] . The above two scheduling strategies in ( 2 ) are illustrated in factor graph models as in Figs . 4 ( b ) and ( c ) , respectively . The graphical models in Fig . 4 do not impose any assumption of potentially inherent independency in datasets and hence are most generic . However , depending on the underlying independency in datasets , we may be able to prune some edges in those graphs . For example , if the data only follows the simple Markov chain of Y −X , while being independent of S and Z , as shown in Fig . 5 ( a ) , all links except one between X and Y will be unreasonable in inference graphs of Figs . 4 ( b ) and ( c ) , that justifies the standard classifier model in Fig . 1 ( a ) . This implies that more complicated inference models such as A-CVAE can be unnecessarily redundant depending on the dataset . This motivates us to consider an extended AutoML framework which automatically explores the best pair of inference factor graph and corresponding Bayesian graph models matching dataset statistics besides the micro-scale hyperparameter tuning . Methodology : AutoBayes begins with exploring any potential Bayesian graphs by cutting links of the full-chain graph in Fig . 4 ( a ) , imposing possible ( conditional ) independence . We then adopt the Bayes-Ball algorithm on each hypothetical Bayesian graph to examine conditional independence over different inference strategies , e.g. , full-chain Z-/S-first inference graphs in Figs . 4 ( b ) / ( c ) . Applying Bayes-Ball justifies the reasonable pruning of the links in the full-chain inference graphs , and also the potential adversarial censoring when Z is independent of S. This process automatically constructs the connectivity of inference , generative , and adversary blocks with sound reasoning . Consider an example case when the data adheres to the following factorization : p ( y , s , z , x ) = p ( y ) p ( s| y ) p ( z| s , y ) p ( x|z , s , y ) , ( 3 ) where we explicitly indicate conditional independence by slash-cancellation from the full-chain case in ( 1 ) . This corresponds to a Bayesian graphical model illustrated in Fig . 5 ( e ) . Applying the Bayes-Ball algorithm to the Bayesian graph yields the following conditional probability : p ( y , s , z|x ) = p ( z|x ) p ( s|z , x ) p ( y|z , s , x ) , ( 4 ) for the Z-first inference strategy in ( 2 ) . The corresponding factor graph is then given in Fig . 6 ( c ) . Note that the Bayes-Ball also reveals that there is no marginal dependency between Z and S , which provides the reason to use adversarial censoring to suppress nuisance information S in the latent space Z . In consequence , by combining the Bayesian graph and factor graph , we automatically obtain A-CVAE model in Fig . 1 ( b ) . AutoBayes justifies A-CVAE under the assumption that the data follows the Bayesian model E in Fig . 5 ( e ) . As the true generative model is unknown , AutoBayes explores different Bayesian graphs like in Fig . 5 to search for the most relevant model . Our framework is readily applicable to graphs with more than 4 nodes to represent multiple Y , S , and Z . Models J and K in Fig . 5 are such examples having multiple latent factors Z1 and Z2 . Despite the search space for AutoBayes will rapidly grow with the number of nodes , most realistic datasets do not require a large number of neural network blocks for macro-level optimization . See Appendix A.2 for more detailed descriptions for some Bayesian graph models to construct factor graphs like in Fig . 6 . Also see discussions of graphical models suitable for semi-supervised learning in Appendix A.4 . Training : Given a pair of generative graph and inference graph , the corresponding DNN structures will be trained . For example of the generative graph model K in Fig . 5 ( k ) , one relevant inference graph Kz in Fig . 6 ( k ) will result in the overall network structure as shown in Fig . 7 , where adversary network is attached as Z2 is ( conditionally ) independent of S. This 5-node graph model justifies a recent work on partially disentanged A-CVAE by Han et al . ( 2020 ) . Each factor block is realized by a DNN , e.g. , parameterized by θ for pθ ( z1 , z2|x ) , and all of the networks except for adversarial network are optimized to minimize corresponding loss functions including L ( ŷ , y ) as follows : min θ , ψ , µ max η E [ L ( ŷ , y ) + λsL ( ŝ , s ) + λxL ( x̂′ , x ) + λzKL ( z1 , z2‖N ( 0 , I ) ) − λaL ( ŝ′ , s ) ] , ( 5 ) ( z1 , z2 ) = pθ ( x ) , ŷ = pφ ( z1 , z2 ) , ŝ = pψ ( z1 ) , x̂ ′ = pµ ( z1 ) , ŝ ′ = pη ( z2 ) , ( 6 ) where λ∗ denotes a regularization coefficient , KL is the Kullback–Leibler divergence , and the adversary network pη ( s′|z2 ) is trained to minimize L ( ŝ′ , s ) in an alternating fashion ( see the Adversarial Regularization paragraph below ) . X Z S Y ( a ) Model Dz X S Z Y ( b ) Model Ds X Z S Y ( c ) Model Ez X Z S Y ( d ) Model Es X Z S Y ( e ) Model Fz X S Z Y ( f ) Model Fs The training objective can be formally understood from a likelihood maximization perspective , in manner that can be seen as a generalization of the VAE Evidence Lower Bound ( ELBO ) concept ( Kingma & Welling , 2013 ) . Specifically , it can be viewed as the maximization of a variational lower bound of the likelihood pΦ ( x , y , s ) that is implicitly defined and parameterized by the networks , where Φ represents the collective parameters of the network modules ( e.g. , Φ = ( φ , ψ , µ ) in the example of equation 5 ) that specify the generative model pΦ ( x , y , s|z ) , which implies the likelihood pΦ ( x , y , s ) , as given by pΦ ( x , y , s ) = ∫ pΦ ( x , y , s|z ) p ( z ) dz . However , since this expression is generally intractable , we introduce qθ ( z|x , y , s ) as a variational approximation of the posterior pΦ ( z|x , y , s ) implied by the generative model ( Kingma & Welling , 2013 ; Ranganath et al. , 2014 ) : 1 n n∑ i=1 log pΦ ( xi , yi , si ) = 1 n n∑ i=1 [ log pΦ ( xi , yi , si|zi ) −log qθ ( zi|xi , yi , si ) p ( zi ) +log qθ ( zi|xi , yi , si ) pΦ ( zi|xi , yi , si ) ] ≈ 1 n n∑ i=1 [ log pΦ ( xi , yi , si|zi ) ] −KL ( qθ ( z|x , y , s ) ‖p ( z ) ) + KL ( qθ ( z|x , y , s ) ‖pΦ ( z|x , y , s ) ) ≥ 1 n n∑ i=1 [ log pΦ ( xi , yi , si|zi ) ] −KL ( qθ ( z|x , y , s ) ‖p ( z ) ) , ( 7 ) where the samples zi ∼ qθ ( z|xi , yi , si ) are drawn for each training tuple ( xi , yi , si ) , and the final inequality follows from the non-negativity of KL divergence . Ultimately , the minimization of our training loss function corresponds to the maximization of the lower bound in ( 7 ) , which corresponds to maximizing the likelihood of our implicit generative model , while also optimizing the variational posterior qθ ( z|x , y , s ) toward the actual posterior for the latent representation pΦ ( z|x , y , s ) , since the gap in the bound is given by KL ( qθ ( z|x , y , s ) ‖pΦ ( z|x , y , s ) ) . Further factoring of log pΦ ( x , y , s|z ) yields the multiple loss-terms and network modules . Adversarial Regularization : We can utilize adversarial censoring when Z and S should be marginally independent , e.g. , such as in Fig . 1 ( b ) and Fig . 7 , in order to reinforce the learning of a representation Z that is disentangled from the nuisance variations S. This is accomplished by introducing an adversarial network that aims to maximize a parameterized approximation q ( s|z ) of the likelihood p ( s|z ) , while this likelihood is also incorporated into the loss for the other modules with a negative weight . The adversarial network , by maximizing the log likelihood log q ( s|z ) , essentially maximizes a lower-bound of the mutual information I ( S ; Z ) , and hence the main network is regularized with the additional term that corresponds to minimizing this estimate of mutual information . This follows since the log-likelihood maximized by the adversarial network is given by E [ log q ( s|z ) ] = I ( S ; Z ) −H ( S ) −KL ( p ( s|z ) ‖q ( s|z ) ) , ( 8 ) where the entropy H ( S ) is constant . Ensemble Learning : We further introduce ensemble methods to make best use of all Bayesian graph models explored by the AutoBayes framework without wasting lower-performance models . Ensemble stacked generalization works by stacking the predictions of the base learners in a higher level learning space , where a meta learner corrects the predictions of base learners ( Wolpert , 1992 ) . Subsequent to training base learners , we assemble the posterior probability vectors of all base learners together to improve the prediction . We compare the predictive performance of a logistic regression ( LR ) and a shallow multi-layer perceptron ( MLP ) as an ensemble meta learner to aggregate all inference models . See Appendix A.5 for more detailed description of the stacked generalization .

Summary: The paper presents AutoBayes: a new approach for nuisance-robust deep learning which explores different Bayesian graph models to search for the best inference strategy. It automatically builds connections between classifier, encoder, decoder, nuisance estimator and adversary DNN blocks. The approach also enables disentangling the learned representations in terms of nuisance variation and task labels. Different benchmark datasets have been used for evaluation.