# Beyond Correlation: Mutual Information to Detect Damage in Nonlinear Systems
## 3 Mutual Information  
A fundamental SHM axiom is that damage increases the complexity of the system [45]. Shannon's entropy [46] is a natural choice as a measure of complexity. The entropy \(H\) of a random variable \(X\) is given by the expected value of \(-\log p(x)\), where \(p\) describes the probability distribution (mass or density function). Joint and conditional entropy of two or more random variables are defined analogously, using the joint and conditional probability distributions, respectively. Mutual information \((I)\) between \(X\) and \(Y\) quantifies the average amount of information shared by \(X\) and \(Y\) and is defined in terms of joint and marginal entropies as follows:  
\[I(X;Y)=H(X)+H(Y)-H(X,Y), \tag{6}\]  
where \(H(X)\) and \(H(Y)\) are marginal entropies of \(X\) and \(Y\), respectively, and \(H(X,Y)\) is their joint entropy. Mutual information \(I(X;Y)\) can also be interpreted as the Kullback-Leibler divergence [15] between the joint distribution \(p(x,y)\) and the product of the marginal distributions \(p(x)p(y)\). Thus, it characterizes the deviation from independence. That is \(I(X;Y)=0\) if and only if \(X\) and \(Y\) are independent random variables. Figure 2 shows a Venn-like diagram, showing the relationship between entropy and mutual information of two random variables \(X\) and \(Y\). Marginal entropies of \(X\) and \(Y\) are represented by the areas enclosed by the red and blue ellipses, respectively. The joint entropy \(H(X,Y)\) is represented by the union of the two ellipses. Figure 2: Information diagram of random variables \(X\) and \(Y\). While mutual information is not the only similarity metric that can measure the strength of nonlinear relationships, it offers the following advantages [47]: Compared to the copula-based alternatives such as [PERSON]'s \(\tau\) and [PERSON]'s \(\rho\), which measure monotonic relationships, mutual information captures all types of nonlinear dependencies. As opposed to the Hilbert-Schmidt Independence Criterion, mutual information is invariant under scaling and other invertible transformations, which is important in signal-processing applications. Compared to distance correlation and the maximal information coefficient, mutual information has lower computational complexity, making it suitable for the analysis of large datasets. Mutual information between two discrete random variables \(X\) and \(Y\) is defined as follows:  
\[I(X;Y)=\sum_{x}\sum_{y}p(x,y)\log\biggl{(}\frac{p(x,y)}{p(x)p(y)}\biggr{)}, \tag{7}\]  
where \(p(x,y)\) is the joint probability mass function and \(p(x)\) and \(p(y)\) are the marginal probability mass functions of \(X\) and \(Y\), respectively. The logarithm can be evaluated using any base. Mutual information between continuous random variables \(X\) and \(Y\) is defined similarly, replacing the sums with integrals and replacing the probability mass functions with probability density functions. In practice, the joint and marginal probability distributions required to compute the mutual information are unknown, and they need to be estimated from the observed data using a nonparametric method. A detailed description of multivariate density estimation methods can be found in [48]. As opposed to the correlation coefficient \(R\), which measures the strength of linear association between a pair of measurements, mutual information measures the strength of functional dependence between them. In other words, it measures the amount by which the knowledge of one quantity decreases the uncertainty about the other. The \(R^{2}\) value is commonly used as a measure of explanatory power in bivariate regression and is termed the coefficient of determination when used in this context. When the MI score is normalized to the \([0,1]\) interval, the resulting normalized MI score becomes comparable to \(R^{2}\) in terms of the values it can take: a score of 1 indicates complete dependence, and a score of 0 indicates independence.