## 1 Introduction  
Bayesian networks [1] (BNs) have played a central role in machine learning research since the early days of the field as expert systems [2, 3], graphical models [4, 5], dynamic and latent variables models [6], and as the foundation of causal discovery [7] and causal inference [8]. They have also found applications as diverse as comorbidities in clinical psychology [9], the genetics of COVID-19 [10], the Sustainable Development Goals of the United Nations [11], railway disruptions [12] and industry 4.0 [13]. Machine learning, however, has evolved to include a variety of other models and reformulated them into a very general information-theoretic framework. The central quantities of this framework are Shannon's entropy and the Kullback-Leibler divergence. Learning models from data relies crucially on the former to measure the amount of information captured by the model (or its complement, the amount of information lost in the residuals) and on the latter as the loss function we want to minimise. For instance, we can construct variational inference [14], the Expectation-Maximisation algorithm [15], Expectation Propagation [16] and various dimensionality reduction approaches such as t-SNE [17] and UMAP [18] using only these two quantities. We can also reformulate classical maximum-likelihood and Bayesian approaches to the same effect, from logistic regression to kernel methods to boosting [19, 20]. Therefore, the lack of literature on how to compute the entropy of a BN and the Kullback-Leibler divergence between two BNs is surprising. While both are mentioned in [PERSON] and [PERSON] [5] and discussed at a theoretical level in [PERSON] et al. [21] for discrete BNs, no resources are available on any other type of BN. Furthermore, no numerical examples of how to compute them are available even for discrete BNs. _We fill this gap in the literature by:_  
* _Deriving efficient formulations of Shannon's entropy and the Kullback-Leibler divergence for Gaussian BNs and conditional linear Gaussian BNs._
* _Exploring the computational complexity of both for all common types of BNs._* _Providing step-by-step numeric examples for all computations and all common types of BNs._ Our aim is to make apparent how both quantities are computed in their closed-form exact expressions and what is the associated computational cost. The common alternative is to estimate both [PERSON]'s entropy and the Kullback-Leibler divergence empirically using Monte Carlo sampling. Admittedly, this approach is simple to implement for all types of BNs. However, it has two crucial drawbacks:  
1. Using asymptotic estimates voids the theoretical properties of many machine learning algorithms: Expectation-Maximisation is not guaranteed to converge [5], for instance. 2. The number of samples required to estimate the Kullback-Leibler divergence accurately on the tails of the global distribution of both BNs is also an issue [22], especially when we need to evaluate it repeatedly as part of some machine learning algorithm. The same is true, although to a lesser extent, for [PERSON]'s entropy as well. In general, the rate of convergence to the true posterior in Monte Carlo particle filters is proportional to the number of variables squared [23]. Therefore, efficiently computing the exact value of [PERSON]'s entropy and the Kullback-Leibler divergence is a valuable research endeavour with a practical impact on BN use in machine learning. To help its development, we implemented the methods proposed in the paper in our _bnlearn_ R package [24]. The remainder of the paper is structured as follows. In Section 2, we provide the basic definitions, properties and notation of BNs. In Section 3, we revisit the most common distributional assumptions in the BN literature: discrete BNs (Section 3.1), Gaussian BNs (Section 3.2) and conditional linear Gaussian BNs (Section 3.3). We also briefly discuss exact and approximate inferences for these types of BNs in Section 3.4 to introduce some key concepts for later use.