Irrelevance and Parameter Learning in Bayesian Networks Nevin Lianwen Zhang Department of Computer Science , University of Science & Technology , Hong Kong Abstract It is possible to learn the parameters of a given Bayesian network structure from data because those parameters influence the probability of observing the data . However , some of the parameters are irrelevant to the probability of observing a particular data case . This paper shows how such irrelevancies can be exploited to speedup various algorithms for parameter learning in Bayesian networks . Experimental results with one of the algorithms , namely the EM algorithm , are presented to demonstrate the gains of this exercise . 1 Introduction Of concern to this paper are problems of learning relationships among variables from data . Solution of such a problem is usually divided into two steps : -LRB- 1 -RRB- selecting a model for the relationships and -LRB- 2 -RRB- estimating the parameters in the model using data . Many models exist . There are statistical models such as regression and loglinear models ; and there are AI models such as decision trees and artificial neural networks -LRB- ANN -RRB- . Recently , there have been growing interests , from both statisticians and AI researchers , in learning with a new type of models called Bayesian networks . Bayesian networks -LRB- Pearl 1988 , also called belief networks or probabilistic networks -RRB- are directed acyclic graphs , where each node represents a random variable and is attached with a conditional probability of the node given its parents . They have been widely accepted as a suitable knowledge representation framework for reasoning and decision making under uncertainty . In applications , Bayesian networks are usually constructed from experts ' knowledge . Knowledge acquisition , especially assessment of numerical probabilities , is usually difficult and unreliable . There is a need to refine Bayesian networks using data -LRB- Spiegelhalter et al 1993 -RRB- , and even to automatically construct Bayesian networks from data -LRB- see Heckerman 1995 for an overview -RRB- . This is one cause for the growing interests in learning Bayesian networks . A second cause is the realization that Bayesian network learning can potentially serve as an alternative to -LRB- Russell et al 1995 -RRB- or complement -LRB- Jordan 1995 -RRB- neural network learning as a general learning paradigm . Since arcs in Bayesian networks can be interpreted as representing causal relationships , experts ' qualitative knowledge , which is often readily available and reliable -LRB- e.g. it can be from a domain textbook -RRB- , can be used to aid model selection and reduce the amount of training data . There is one other advantage of Bayesian network learning that has received little attention up to now . Suppose a Bayesian network structure -LRB- model -RRB- has been selected and we want to estimate the parameters from data . This is possible because the parameters influence the probability of observing the data . For a particular data case , however , there is , due to the given network structure , a subset of parameters that are irrelevant to the probability of observing the data case -LRB- Geiger et al 1990 , Lauritzen et al 1990 -RRB- . Consequently , the data case should not contribute anything to the estimation of those parameters . Such irrelevancies can be exploited to speedup learning . Exploiting irrelevancies are especially important when there is a large number of variables in the model and there are multiple data sets , each concerning a different subset of variables . In such a case , the number of parameters that are relevant to a data set can be much smaller than the total number of parameters . This paper examines three types of algorithms for parameter learning in a given Bayesian network model , namely the EM algorithm -LRB- Lauritzen 1995 -RRB- , gradient descent -LRB- Russell et al 1995 -RRB- , and sequential learning -LRB- Spiegelhalter and Cowell 1992 -RRB- . We show how irrelevancies can be exploited to speedup all those algorithms . Experimental results with the EM algorithm are reported to demonstrate the gains of this exercise . 2 Parameter learning in Bayesian networks Let x1 , x2 , : : : , and xn be all the variables in a Bayesian network structure and let ssi be the set of parents of xi . This paper assumes that all variables have a finite number of possible values . The network structure is a model about the relationships among the variables ; it states that the joint probability p -LRB- x1 ; x2 ; : : : ; xn -RRB- can be factorized into the multiplication of the conditional probabilities of the variables given their respective parents , i.e. p -LRB- x1 ; x2 ; : : : ; xn -RRB- = Y i p -LRB- xijssi -RRB- : -LRB- 1 -RRB- This paper is concerned with learning conditional probabilities for a given network structure . We shall use ? ijk to denote p -LRB- xi = jjssi = k -RRB- , where j is a value of variable xi and k is a combination of the values of the parents of xi . For convenience , we shall say that k is a value of ssi and and call ? ijk a parameter pertaining to variable xi . We shall also use ? to denote the vector of all the parameters ? ijk . The joint probability over all the variables determined by a parameter vector ? will be denoted by p -LRB- x1 ; x2 ; : : : ; xnj ? -RRB- . The parameter vector ? is to be estimated from a collection D of data cases D1 , D2 , : : : , Dm that are independent given ? . Each data case is a set of variable-value pairs . Different data cases might involve different variables . This implies that data cases might be incomplete in the sense that they might not contain values for all variables . If ? is the true parameter vector , then the probability of observing D is p -LRB- Dj ? -RRB- = Ql p -LRB- Dlj ? -RRB- . Viewed as a function of ? , p -LRB- Dj ? -RRB- is called the likelihood function of ? given data D . The maximum likelihood -LRB- ML -RRB- estimate of ? is a value of ? that maximizes the likelihood function . When there is a prior probability -LRB- density function -RRB- p -LRB- ? -RRB- about ? , one can compute the posterior probability using Bayes ' rule , p -LRB- ? jD -RRB- = cp -LRB- ? -RRB- p -LRB- Dj ? -RRB- ; where c is the renormalization constant . In this case , ? ijk can be estimated by using the posterior expectation R ? ijkp -LRB- ? ijkjD -RRB- d?ijk . Lauritzen -LRB- 1995 -RRB- and Russell -LRB- et al 1995 -RRB- have respectively explained how the EM algorithm -LRB- Dempster et al 1977 -RRB- and gradient descent can be used to compute an approximate ML estimate of ? , while Spiegelhalter and Lauritzen -LRB- 1990 -RRB- , Spiegelhalter and Cowell -LRB- 1992 -RRB- , and Titterington -LRB- 1976 -RRB- have described approximate methods for obtaining the posterior expectation of ? ijk for every i , j , and k. Although very different in appearances , those methods share one thing in common . In obtaining the next estimate of ? from the current estimate ? -LRB- t -RRB- , a major part of the computation is spent in calculating the conditional probability p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- for every variable xi and for one or every data case Dl . This conditional probability is used to revise the parameters ? -LRB- t -RRB- ijk -LRB- for all j and all k -RRB- pertaining to xi . 2.1 Purpose of this paper For any set Y of variables in a Bayesian network , the ancestral set an -LRB- Y -RRB- consists of variables in Y and ancestors of those variables , i.e. variables from which there are directed paths to at least one variable Y . It is evident that xi2an -LRB- Y -RRB- implies ssiaean -LRB- Y -RRB- . Let Yl be the set of variables whose values are observed in data case Dl . The purpose of this paper is to show Main result : In various parameter learning algorithms , there is , for any data case Dl , no need to calculate p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- for variables xi outside the ancestral set an -LRB- Yl -RRB- . The main result is interesting because it cuts down computation and hence speeds-up learning . The amount of computation is reduced not only because the conditional probability p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- is calculated only for variables xi in the ancestral set an -LRB- Yl -RRB- , but also because calculating p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- for variables xi outside the ancestral set can be much more difficult than for those in the ancestral set . If one uses the clique tree propagation -LRB- CTP -RRB- inference algorithm -LRB- Lauritzen and Spiegelhalter 1988 , Shafer and Shenoy 1990 , Jensen et al 1990 -RRB- , the main result ensures that it suffices to construct a clique tree for the restriction of the Bayesian network onto an -LRB- Yl -RRB- . Without the main result , one would need to construct a clique tree for the entire network . The maximum clique in the latter clique tree can be much larger than its counterpart in the former clique tree and hence inference can take much more time . It not rare that inference in the entire network is infeasible , while inference in a subnetwork can be done in real time . The next section will establish the main result for the EM algorithm and report experimental results to demonstrate the efficiency gains that the main result brings about . Section 4 will extend the main result to the gradient descent and the sequential learning algorithms . 3 The EM algorithm The EM algorithm -LRB- Dempster et al 1977 -RRB- is a general iterative algorithm for computing approximate ML estimates from incomplete data . Lauritzen -LRB- 1995 -RRB- has shown how it can be used for parameter learning in Bayesian networks . In this context , major computation at each iteration is spent on calculating the conditional probability p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- for all data cases Dl and all variables xi . This section describes a modification of the EM algorithm -LRB- for Bayesian networks -RRB- . For each data case Dl , the modified algorithm calculates p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- only for variables xi in the ancestral set an -LRB- Yl -RRB- and hence runs faster than the original algorithm . Moreover , the modified algorithm has a higher rate of convergence than the original algorithm . 3.1 A brief review of the EM algorithm Let us begin with a brief review of the EM algorithm . The loglikelihood function of ? given data D is defined by l -LRB- ? jD -RRB- = lnp -LRB- Dj ? -RRB- = X l lnp -LRB- Dlj ? -RRB- : It is evident that maximizing the likelihood function is the same as maximizing the loglikelihood function . It is usually more convenient to work with the latter . A data case Dl is complete if all variables are observed in the case . From equation -LRB- 1 -RRB- it is easy to see that when all the data cases are complete , the loglikelihood function is l -LRB- ? jD -RRB- = X i ; j ; k f -LRB- xi = j ; ssi = k -RRB- ln?ijk ; -LRB- 2 -RRB- where f -LRB- xi = j ; ssi = k -RRB- stands for the number of data cases where xi = j and ssi = k. Consequently , the ML estimate ?? of ? can be obtained by setting ?? ijk = f -LRB- xi = j ; ssi = k -RRB- Pj f -LRB- xi = j ; ssi = k -RRB- for all i , j , and k : -LRB- 3 -RRB- Often , data cases are incomplete . This is when the EM algorithm comes into play . The algorithm starts with an initial estimate ? -LRB- 0 -RRB- of ? and improves it iteratively . From the current estimate ? -LRB- t -RRB- , the next estimate ? -LRB- t +1 -RRB- is obtained in two steps : the expectation step -LRB- E-step -RRB- and the maximization step -LRB- M-step -RRB- . The E-step computes the current expected loglikelihood function of ? given data D , i.e. Q -LRB- ? j ? -LRB- t -RRB- -RRB- = X l X Xl lnp -LRB- Dl ; Xlj ? -RRB- p -LRB- XljDl ; ? -LRB- t -RRB- -RRB- ; -LRB- 4 -RRB- where Xl is the set of variables whose values are missing from data case Dl . The M-step chooses the next estimate ? -LRB- t +1 -RRB- by maximizing the current expected loglikelihood : Q -LRB- ? -LRB- t +1 -RRB- j ? -LRB- t -RRB- -RRB- > = Q -LRB- ? j ? -LRB- t -RRB- -RRB- for all ? : By making use of equation -LRB- 1 -RRB- , we obtain that Q -LRB- ? j ? -LRB- t -RRB- -RRB- = X i ; j ; k ft -LRB- xi = j ; ssi = k -RRB- ln?ijk ; where ft -LRB- xi ; ssi -RRB- = X l p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- : -LRB- 5 -RRB- Hence , the E-step reduces to computing the function ft -LRB- xi ; ssi -RRB- for each variable xi , while the M-step reduces to computing the next estimate of ? by setting , ? -LRB- t +1 -RRB- ijk = ft -LRB- xi = j ; ssi = k -RRB- Pj ft -LRB- xi = j ; ssi = k -RRB- for all i , j , and k : -LRB- 6 -RRB- Because of equation -LRB- 2 -RRB- , the distribution p -LRB- Dj ? -RRB- of complete data is from the regular exponential family with natural parameter ln?ijk . This fact implies that the loglikelihood l -LRB- ? -LRB- t -RRB- jD -RRB- of incomplete data increases with t and hence converges -LRB- Dempster et al 1977 , Wu 1983 -RRB- . From equation -LRB- 5 -RRB- , we see that at each iteration , the EM algorithm needs to calculate the conditional probability p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- for every data case Dl and every variable xi , and this is the dominant part of the computation . 3.2 Modifying the EM algorithm Let Y be a subset of variables in a Bayesian network . The joint probability P -LRB- an -LRB- Y -RRB- -RRB- of the variables in the ancestral set an -LRB- Y -RRB- is given by p -LRB- an -LRB- Yl -RRB- -RRB- = Y i s.t. xi2an -LRB- Yl -RRB- p -LRB- xijssi -RRB- ; -LRB- 7 -RRB- We say a data case Dl is upward complete if an -LRB- Yl -RRB- = Yl . By equation -LRB- 7 -RRB- , we have that when all the data cases are upward complete , the loglikelihood function is l -LRB- ? jD -RRB- = X i ; j ; k g -LRB- xi = j ; ssi = k -RRB- ln?ijk ; -LRB- 8 -RRB- where g -LRB- xi = j ; ssi = k -RRB- stands for the number of data cases Dl such that Yl contains xi and that xi = j and ssi = k. Consequently , the ML estimate of ? can be obtained by using equation -LRB- 3 -RRB- with the f-function replaced by the g-function . The rest of this subsection describes an EM style algorithm for situations where data cases are not upward complete . An intuitive way to understand the EM algorithm is that it makes an incomplete data case Dl complete by filling in missing data Xl . The algorithm to be described below makes a data case that is not upward complete upward complete by filling in missing data . Let ? -LRB- t -RRB- be the current estimate of ? . We obtain the next estimate in the following two steps . First , we compute the function Q0 -LRB- ? j ? -LRB- t -RRB- -RRB- = X l X Zl lnp -LRB- Dl ; Zlj ? -RRB- p -LRB- ZljDl ; ? -LRB- t -RRB- -RRB- ; -LRB- 9 -RRB- where Zl is the set of variables in the ancestral set an -LRB- Yl -RRB- whose values are missing from the data case Dl , i.e. Zl = an -LRB- Yl -RRB- nYl . Then , we choose the next estimate ? -LRB- t +1 -RRB- by maximizing the Q0 function . It follows from equation -LRB- 7 -RRB- that Q0 -LRB- ? j ? -LRB- t -RRB- -RRB- = X i ; j ; k gt -LRB- xi ; ssi -RRB- ln?ijk ; where gt -LRB- xi ; ssi -RRB- = X l s.t.xi2an -LRB- Yl -RRB- p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- : -LRB- 10 -RRB- Consequently , our first step reduces to computing gt -LRB- xi ; ssi -RRB- for all variables xi , and our second step reduces to obtaining the next estimate ? -LRB- t +1 -RRB- by setting , ? -LRB- t +1 -RRB- ijk = gt -LRB- xi = j ; ssi = k -RRB- Pj gt -LRB- xi = j ; ssi = k -RRB- for all i , j , and k : -LRB- 11 -RRB- Because of equation -LRB- 8 -RRB- , the distribution p -LRB- Dj ? -RRB- of upward complete data is from the regular exponential family with natural parameters ln?ijk . This fact enables one to prove the convergence of the modified EM algorithm by following the same line of reasoning as in the proof of the convergence of the original EM algorithm -LRB- Dempster et al 1977 , Wu 1983 -RRB- . 3.3 Implementation , experiments and discussions 3.3.1 Implementation We have implemented the modified EM and EM based on the version of clique tree propagation -LRB- CTP -RRB- by Shafer and Shenoy -LRB- 1990 -RRB- . When implementing the modified EM , we have assumed that data cases are grouped into data sets such that data cases in the same data set involve the same variables . We have also assumed that identical data cases are aggregated . In our implementation , there is a preprocessing module which computes , for each data set , the ancestral set an -LRB- Yl -RRB- of the set Yl of variables involved and constructs a clique tree structure for the restriction of the Bayesian network onto an -LRB- Yl -RRB- . The clique tree structures are kept static over iterations of the learning algorithm . Ancestral sets are found by backward search ; the complexity is linear in the number of arcs . Clique tree structures are constructed by first obtaing the moral graphs of the restricted Bayesian networks -LRB- Lauritzen and Spiegelhalter 1988 -RRB- , finding elimination orderings by using the minimum deficiency heuristic -LRB- Bertel?e and F. Brioschi 1972 , Kjaerulff 1990 -RRB- , and then constructing the clique tree structures from the orderings . The total complexity is O -LRB- kn2 -RRB- , where n is the number of nodes in a restricted Bayesian network and k is the maximum number of neighbors of a node in the moral graph . All the work in one iteration of the modified EM is carried out by the main module . It consists of three submodules : a initialization submodule , a CTP submodule , and a learning submodule . The initialization submodule initializes each clique tree structure by using the the current network parameters . For each node in the Bayesian network , its conditional probability is attached to one clique in the tree , if exists , that contains the node . If more than one conditional probability is attached to one clique , they are multiplied into one function . If nothing is attached to a clique , the identity function is attached to it . Unlike in the version of CTP by Lauritzen and Spiegelhalter -LRB- 1988 -RRB- and Jensen et al -LRB- 1990 -RRB- , marginal probabilities of the cliques are not computed . The CTP submodule implements the CTP algorithm . It is called once for each data case -LRB- with the corresponding clique tree -RRB- , and it updates the gt function according to equation -LRB- 10 -RRB- . Finally , the learning submodule uses equation -LRB- 11 -RRB- to obtain the network parameters for the next iteration . The implementation for EM is the same except there is only one clique tree for the entire network . To make a quick comparison , we see that the modified EM spends more time in the initialization submodule than EM since it needs to initialize potentially many clique tree structures while EM has only one to initialize . Hence EM can possibly outperform the modified EM in some situations . As will be seen in the next subsection , however , the modified EM almost always significantly outperforms EM . There are two reasons . Firstly , the clique trees encountered by the modified EM are smaller -LRB- in terms of both the number of cliques and clique sizes -RRB- than those encountered by EM . Secondly , the amount of computation in initialization is usually substantially less than the amount of computation in propagation , especially when there are many data cases in a data set . Because of the second reason , EM can seldom significantly outperform the modified EM . 3.3.2 Experiments Experiments have been performed with two CPCS networks provided by Pradhan to compare the modified EM and EM . One of the networks is a polytree consisting of 40 nodes . The other consists of 145 nodes and contain loops . Data sets were generated for each network by using original network parameters . The number of data sets , the number of data cases in a data set , and the number of variables in a data case were pre-determined . In generating a data set , a list of variables were first randomly generated . Then the joint probability of those variables were computed and data cases were randomly generated by using this joint probability . Since the complexity of computing the joint probability is exponential , we have restricted the number of variables to be no larger than ten . Both EM and the modified EM were called to re-learn the network parameters from the data sets . The algorithms were allowed to run until the loglikelihoods l -LRB- ? -LRB- t -RRB- jD -RRB- at two consecutive iterations do not differ by more than 1.0 . Four experiments were carried out . The setups are described in the following table : Experiment Bayesian network Number of Data sets Number of variables in each data set Number of data cases in each data set 1 The 40-node BN 15 5 75 2 The 40-node BN 15 10 500 3 The 145-node BN 50 5 75 4 The 145-node BN 15 10 500 For example , the first experiment was performed in the 40-node network . Fifteen data sets were generated with each data set involving 5 variables and containing 75 data cases . In all the experiments , the loglikelihoods l -LRB- ? -LRB- t -RRB- jD -RRB- of the parameter vectors ? -LRB- t -RRB- produced over time by either EM or the modified EM were recorded and they are summarized in Figure 1 We see that when the modified EM is used , the loglikelihood converges quickly . When EM is used , on the other hand , the loglikelihood converges much slower . In all experiments EM was able to complete only a few iterations by the time the modified EM had already reached the stopping condition . 3.3.3 Discussions A closer look reveals that two facts contribute to the better performance of the modified EM . First , it took the modified EM less time to complete one iteration than EM -LRB- Table 1 -RRB- . Second , it took the modified EM less iterations to converge than EM -LRB- Figure 2 -RRB- . -4500 -4000 -3500 -3000 -2500 -2000 -1500 -1000 -500 10 20 30 40 50 60 70 80 Loglikelihood CPU time in seconds Experiment 1 `` EM '' `` modifiedEM '' -40000 -35000 -30000 -25000 -20000 -15000 -10000 50 100 150 200 250 300 350 Loglikelihood CPU time in seconds Experiment 2 `` EM '' `` modifiedEM '' -60000 -55000 -50000 -45000 -40000 -35000 -30000 -25000 -20000 -15000 -10000 20 40 60 80 100 120 140 Loglikelihood CPU time in seconds Experiment 3 `` EM '' `` modifiedEM '' -50000 -45000 -40000 -35000 -30000 -25000 -20000 -15000 -10000 100 200 300 400 500 600 700 Loglikelihood CPU time in seconds Experiment 4 `` EM '' `` modifiedEM '' Figure 1 : Loglikelihoods over time . Table 1 : Time for completing one iteration . Experiment EM modified EM 1 0.950962 0.094330 2 4.681479 1.209952 3 13.697670 1.081957 4 181.303357 4.822140 The first fact that contribute the better performance of the modified EM can be explained by observing that for each data case Dl , the modified EM calculates the conditional probability p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- only for those variables xi in the ancestral set an -LRB- Yl -RRB- , while EM calculates it for all variables . To explain the second fact , we need to recall a well known fact about EM : The rate of convergence is closely related to the amount of information that is needed to make data cases complete . More specifically , the greater -4500 -4000 -3500 -3000 -2500 -2000 -1500 -1000 -500 10 20 30 40 50 60 70 80 Loglikelihood Number of iterations Experiment 1 `` EM '' `` modifiedEM '' -40000 -35000 -30000 -25000 -20000 -15000 -10000 10 20 30 40 50 60 70 Loglikelihood Number of iterations Experiment 2 `` EM '' `` modifiedEM '' -60000 -55000 -50000 -45000 -40000 -35000 -30000 -25000 -20000 -15000 -10000 10 20 30 40 50 60 70 80 90 100 Loglikelihood Number of iterations Experiment 3 `` EM '' `` modifiedEM '' -12000 -11000 -10000 -9000 -8000 -7000 -6000 -5000 -4000 -3000 5 10 15 20 25 30 35 40 45 50 Loglikelihood Number of iterations Experiment 4 `` EM '' `` modifiedEM '' Figure 2 : Loglikelihoods over iterations . this amount of information , the slower the rate of convergence -LRB- Dempster et al 1977 -RRB- . Similarly , the rate of convergence of the modified EM is closely related to the amount of information that is needed to make data cases upward complete : The greater this amount of information , the slower the rate of convergence . Since it take less information to make data cases upward complete than to make them complete , the modified EM algorithm has a higher rate of convergence than the original EM algorithm . Before leaving this section , let us point out that the differences in performance between EM and the modified EM in experiments 2 and 4 are not as significant as in experiments 1 and 3 . This is due to the fact that data involve more variables in experiments 2 and 4 than in experiments 1 and 3 . In general , the more variables in data , the lower the extent to which the modified EM outperforms EM . 4 Other algorithms We have seen that the main result can significantly improve the performance of the EM algorithm . This section extends the result to the gradient descent and the sequential learning algorithms . 4.1 Gradient Descent Finding an ML estimate of ? is a constrainted nonlinear optimization problem . Each parameter ? ijk must be between and 1 and for fixed i and k , Pj ? ijk must be 1 . The gradient projection method -LRB- Leunberger 1984 -RRB- can be used to solve this problem . In this method , the next estimate of ? is obtained from the current estimate ? -LRB- t -RRB- as follows : Find the gradient vector 5 ? of l -LRB- ? jD -RRB- at ? -LRB- t -RRB- ; project it onto the constraint surface to get the gradient vector 50 ? along the the constraint surface ; and take a step in the direction 50 ? , with the step-size determined by line minization -LRB- or rather maximization -RRB- . The gradient vector 5 ? contains a component 5 ? ijk for each parameter ? ijk , which is the partial derivative of l -LRB- ? jD -RRB- with respect to ? ijk at ? -LRB- t -RRB- . Russell et al -LRB- 1995 -RRB- has shown that 5 ? ijk = @l -LRB- ? jD -RRB- @ ? ijk j ? = ? -LRB- t -RRB- = ft -LRB- xi = j ; ssi = k -RRB- ? -LRB- t -RRB- ijk : -LRB- 12 -RRB- The projection 50 ? of 5 ? onto the constraint surface is 50 ? ijk = ft -LRB- xi = j ; ssi = k -RRB- ? -LRB- t -RRB- ijk ? 1 ni X j ft -LRB- xi = j ; ssi = k -RRB- ? -LRB- t -RRB- ijk ; -LRB- 13 -RRB- where ni stands for the number of possible values of xi . In each iteration , the conditional probability p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- needs to be computed for every data case Dl and for every variable xi , and this is a major part of the computation . The following proposition states that for each data case Dl , p -LRB- xi ; ssijDl ; ? -LRB- t -RRB- -RRB- needs to be computed only for variables xi in the ancestral set an -LRB- Yl -RRB- . Proposition 1 The quantity one the right hand side of equation -LRB- 13 -RRB- remains unchanged if the function ft is replaced by function gt . Lemma 1 Let Y be a subset of variables in a Bayesian network and D be a data case . For any xi = 2an -LRB- Y -RRB- , p -LRB- xi ; ssijD -RRB- = p -LRB- xijssi -RRB- p -LRB- ssijD -RRB- : Proof : The lemma follows from the fact that p -LRB- an -LRB- fxig -LSB- Y -RRB- -RRB- = p -LRB- xijssi -RRB- p -LRB- an -LRB- ssi -LSB- Y -RRB- -RRB- : 2 Proof of the proposition : By Lemma 1 , we have that for any data case Dl and any xi = 2an -LRB- Yl -RRB- , p -LRB- xi = j ; ssi = kjDl ; ? -LRB- t -RRB- -RRB- ? -LRB- t -RRB- ijk ? 1 ni X j p -LRB- xi = j ; ssi = kjDl ; ? -LRB- t -RRB- -RRB- ? -LRB- t -RRB- ijk = p -LRB- xi = jjssi = k ; ? -LRB- t -RRB- -RRB- ? -LRB- t -RRB- ijk p -LRB- ssi = kjDl ; ? -LRB- t -RRB- -RRB- ? 1 ni X j p -LRB- xi = jjssi = k ; ? -LRB- t -RRB- -RRB- ? -LRB- t -RRB- ijk p -LRB- ssi = kjDl ; ? -LRB- t -RRB- -RRB- = p -LRB- ssi = kjDl ; ? -LRB- t -RRB- -RRB- ? 1 ni X j p -LRB- ssi = kjDl ; ? -LRB- t -RRB- -RRB- = : Hence , the proposition follows . 2 Russell et al -LRB- 1995 -RRB- uses an adaptation of the conjugate gradient algorithm , instead of the gradient projection method . There is not enough information in the paper for us to determine whether replacing ft with gt there would affect the algorithm behavior in terms of convergence ; and if yes , whether the effect is positive or negative . 4.2 Sequential learning Sequential learning methods -LRB- Spiegelhalter and Lauritzen 1990 , Spiegelhalter and Cowell 1992 , and Titterington 1976 -RRB- approximate the posterior expectation R ? ijkp -LRB- ? ijkjD -RRB- d?ijk for every i , j , and k . They absorb data cases one by one . Suppose ? -LRB- l -RRB- is the estimate after absorbing the first l data cases . In absorbing data case Dl +1 , they compute the conditional probability p -LRB- xi ; ssijDl +1 ; ? -LRB- l -RRB- -RRB- for every variable xi . This section shows that the conditional probability needs to be computed only for variables in the ancestral set an -LRB- Yl +1 -RRB- . To understand the sequential learning methods , let us inductively define pl -LRB- ? -RRB- = cp -LRB- Dlj ? -RRB- pl ? 1 -LRB- ? -RRB- ; l = ; 1 ; : : : ; m ; where c is the renormalization constant and when l = 0 , p0 -LRB- ? -RRB- is simply the prior probability p -LRB- ? -RRB- . Then pm -LRB- ? -RRB- = p -LRB- ? jD -RRB- . Due to the large number of parameters encapsulated in the parameter vector ? , straightforward computation of pm is unrealistic . Simplifying assumptions , and hence approximations , are usually made . Let ? i denote the collection of all the parameters pertaining to variable xi , i.e. ? i = f?ijk ; for all j and kg . The global independence assumption states that the parameter groups ? i for different i 's are independent under probability pl , i.e. pl -LRB- ? -RRB- = Qi pl -LRB- ? i -RRB- . When it is not the case , pl -LRB- ? -RRB- is approximated by Qi pl -LRB- ? i -RRB- . For any i and k , define ? ik to be the collection of parameters f?ijk ; for all jg . The local independence assumption states that for any fixed i , the parameter groups ? ik for different k 's are independent under probability pl , i.e. pl -LRB- ? i -RRB- = Qk pl -LRB- ? ik -RRB- . When it is not the case , pl -LRB- ? i -RRB- is approximated by Qk pl -LRB- ? ik -RRB- . The third assumption , which might be called the Dirichlet distribution assumption , states that the marginal probability pl -LRB- ? ik -RRB- is of a Dirichlet distribution D -LSB- li1k ; : : : ; linik -RSB- , where ni stands for the number of possible values of xi . When it is not the case , pl -LRB- ? ik -RRB- is approximated by a Dirichlet distribution . The parameters lijk -LRB- j = 1 ; : : : ; ni -RRB- in the Dirichlet distribution can be thought of as representing counts of past cases where xi = j and ssi = k . The estimate ? -LRB- l -RRB- can be obtained from the counts by ? -LRB- l -RRB- ijk = lijklik ; -LRB- 14 -RRB- where lik = Pj lijk is called the \ precision '' underlying our belief about ? -LRB- l -RRB- ijk . The quantity lijklik is actually the mean of ? -LRB- l -RRB- ijk . Under those three assumptions , the probability pl -LRB- ? -RRB- is completely determined by the counts lijk . Hence , computing pl +1 -LRB- ? -RRB- from pl -LRB- ? -RRB- reduces to obtaining the counts l +1 ijk from the counts lijk . We shall refer to this task as count updating . Spiegelhalter and Lauritzen -LRB- 1990 -RRB- have shown that pl +1 -LRB- ? ik -RRB- = X j pl -LRB- ? ikjxi = j ; ssi = k -RRB- p -LRB- xi = j ; ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- + pl -LRB- ? ik -RRB- -LRB- 1 ? p -LRB- ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- -RRB- : -LRB- 15 -RRB- where , by the standard conjugate Bayesian updating , pl -LRB- ? ikjxi = j ; ssi = k -RRB- is of the Dirichlet distribution D -LSB- li1k ; : : : ; lijk +1 ; : : : ; linik -RSB- : Equation -LRB- 15 -RRB- reduces count updating to approximating the distribution pl +1 -LRB- ? ik -RRB- by a Dirichlet distribution . This can be done in several ways . In fractional updating method -LRB- Titterington 1976 , Smith and Markov 1978 -RRB- , pl +1 -LRB- ? ik -RRB- is approximated by the Dirichlet distribution with counts l +1 ijk = lijk + p -LRB- xi = j ; ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- for all j : The probabilistic editor method -LRB- Spiegelhalter and Cowell 1992 -RRB- approximates pl +1 -LRB- ? ik -RRB- by the Dirichlet distribution with counts l +1 ijk = l +1 ik mijk for all j ; -LRB- 16 -RRB- where mijk = lijk + plijk lik + 1 ; l +1 ik = -LRB- lik + 1 -RRB- 2 Pj m2ijk -LRB- 1 ? mijk -RRB- ? Pj mijkplijk -LRB- 1 ? plijk -RRB- -LRB- lik + 1 -RRB- Pj m2ijk -LRB- 1 ? mijk -RRB- + Pj mijkplijk -LRB- 1 ? plijk -RRB- ; and where plijk is a shorthand for p -LRB- xi = j ; ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- =p -LRB- ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- . A common theme in those methods is that the conditional probability p -LRB- xi = j ; ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- is computed for very variable xi and all the counts are updated . The following proposition states that one needs only update the counts pertaining to variables in the ancestral set an -LRB- Yl +1 -RRB- , and consequently p -LRB- xi = j ; ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- needs to be computed only for those variables . Proposition 2 For any xi = 2an -LRB- Yl +1 -RRB- , pl +1 -LRB- ? ik -RRB- = pl -LRB- ? ik -RRB- : Proof : By inspecting the pdf of a Dirichlet distribution , one can easily see that pl -LRB- ? ikjxi = j ; ssi = k -RRB- = pl -LRB- ? ik -RRB- liklij k ? ijk : By Lemma 1 , we get p -LRB- xi = j ; ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- = p -LRB- xi = jjssi = k ; ? -LRB- l -RRB- -RRB- p -LRB- ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- = lijklik p -LRB- ssi = kjDl +1 ; ? -LRB- l -RRB- -RRB- : Thus , the proposition follows from the fact that Pj ? ijk = 1 . 2 . We would like to point out that the proposition remains true even without the local independence and the Dirichlet distribution assumptions . 5 Conclusions and future directions It is possible to learn parameters for a Bayesian network structure from data because those parameters influence the probability of observing the data . For a particular data case , one can determine , from the given network structure , a subset of parameters that are irrelevant to the probability of observing the data case . Consequently , the data case should not contribute anything in learning those parameters . This paper has shown how such irrelevancies can be exploited to speedup various algorithms for parameter learning in Bayesian networks . Efficiency gains have been empirically demonstrated with the EM algorithm . The amount of efficiency gains is closely related to the number of variables in data . The fewer the number of variables in data , the more significant the efficiency gains . It would be interesting to investigate if and when a data set that involves many variables can be approximated by data sets with less variables . It would also be interesting to determine whether irrelevance has a role to play in structure learning . Acknowledgement We thank M. Pradhan for providing the CPCS networks and the anonymous reviewers for insightful comments and suggestions . This paper has benefited from the AI and Statistics group at Hong Kong University of Science and Technology . Special thanks go to D. Y. Yeung . Research was supported by Hong Kong Research Grant Council grant HKUST658/95E . References -LSB- 1 -RSB- U. Bertel?e and F. Brioschi -LRB- 1972 -RRB- , Nonserial dynamic programming , Mathematics in Science and Engineering , Vol . 91 , Academic Press . -LSB- 2 -RSB- A. P. Dempster , N. M. Laird , and D. B. Rubin -LRB- 1977 -RRB- , Maximum likelihood from incomplete data via the EM algorithm , Journal of the Royal Statistical Society , 39 -LRB- Series B -RRB- , 1-38 . -LSB- 3 -RSB- D. Geiger , T. Verma , and J. Pearl -LRB- 1990 -RRB- , d-separation : From theorems to algorithms , in Uncertainty in Artificial Intelligence 5 , pp. 139-148 . -LSB- 4 -RSB- D. Heckerman -LRB- 1995 -RRB- , A tutorial on learning Bayesian networks , Technical Report MSR-TR-95-06 , Microsoft Research , Microsoft Corporation . -LSB- 5 -RSB- F. V. Jensen , K. G. Olesen , and K. Anderson -LRB- 1990 -RRB- , An algebra of Bayesian belief universes for knowledge-based systems , Networks , 20 , pp. 637 - 659 . -LSB- 6 -RSB- M. I. Jordan -LRB- 1995 -RRB- , Why the logistic function ? A tutorial discussion on probabilities and neural networks , Technical Report 9503 , Computational Cognitive Science , MIT . -LSB- 7 -RSB- U. Kjaerulff -LRB- 1990 -RRB- , Triangulation of Graphs - Algorithms giving small total state space , R 90-09 , Institute for Electronic Systems , Department of Mathematics and Computer Science , Strandvejen , DK 9000 Aalborg , Denmark . -LSB- 8 -RSB- S. L. Lauritzen -LRB- 1995 -RRB- , The EM algorithm for graphical association models with missing data , Computational Statistics and Data Analysis , Vol . 19 , 191-201 . -LSB- 9 -RSB- S. L. Lauritzen and D. J. Spiegelhalter -LRB- 1988 -RRB- , Local computations with probabilities on graphical structures and their applications to expert systems , Journal of Royal Statistical Society B , 50 : 2 , pp. 157 - 224 . -LSB- 10 -RSB- S. L. Lauritzen , A. P. Dawid , B. N. Larsen , and H. G. Leimer -LRB- 1990 -RRB- , Independence Properties of Directed Markov Fields , Networks , 20 , pp. 491-506 . -LSB- 11 -RSB- D. G. Leunberger -LRB- 1984 -RRB- , Linear and Nonlinear Programming , Second edition , Addison-Wesley Publishing Company . -LSB- 12 -RSB- J. Pearl -LRB- 1988 -RRB- , Probabilistic Reasoning in Intelligence Systems : Networks of Plausible Inference , Morgan Kaufmann Publishers , Los Altos , CA . -LSB- 13 -RSB- S. J. Russell , J. Binder , D. Koller , and K. Kanazawa -LRB- 1995 -RRB- , Local learning in probabilistic networks with hidden variables , IJCAI ' 95 , 1146 - 1152 . -LSB- 14 -RSB- G. Shafer and P. Shenoy -LRB- 1990 -RRB- , Probability propagation , Annals of Mathematics and Artificial Intelligence , 2 , pp. 327-352 . -LSB- 15 -RSB- D. J. Spiegelhalter and R. G. Cowell -LRB- 1993 -RRB- , Learning in probabilistic expert systems , in Bayesian Statistics 4 , J. M. Bernardo , J. O. Berger , A. P. Dawid , and A. F. M. Smith -LRB- eds . -RRB- , 447-465 . -LSB- 16 -RSB- D. J. Spiegelhalter , A. P. Dawid , S. L. Lauritzen , and R. G. Cowell -LRB- 1993 -RRB- , Bayesian analysis in expert systems , Statistical Science , Vol . 8 , No. 3 , 219-283 . -LSB- 17 -RSB- D. M. Titterington -LRB- 1976 -RRB- , Updating a diagnostic system using unconfirmed cases , Applied Statist. , 25 , pp. 238-247 . -LSB- 18 -RSB- C. F. J. Wu -LRB- 1983 -RRB- , On the convergence properties of the EM algorithm , The Annals of Statistics , Vol . 11 , No. 1 , 95-103 .

B O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

bayesian networks
efficiency
irrelevance
parameter learning