# Dual total correlation

In information theory, dual total correlation, information rate, excess entropy, or binding information is one of several known non-negative generalizations of mutual information. While total correlation is bounded by the sum entropies of the n elements, the dual total correlation is bounded by the joint-entropy of the n elements. Although well behaved, dual total correlation has received much less attention than the total correlation. A measure known as "TSE-complexity" defines a continuum between the total correlation and dual total correlation.

## Definition

For a set of n random variables $\{X_{1},\ldots ,X_{n}\}$, the dual total correlation $D(X_{1},\ldots ,X_{n})$ is given by
    $D(X_{1},\ldots ,X_{n})=H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right),$
where $H(X_{1},\ldots ,X_{n})$ is the joint entropy of the variable set $\{X_{1},\ldots ,X_{n}\}$ and $H(X_{i}\mid \cdots )$ is the conditional entropy of variable $X_{i}$, given the rest.

## Normalized

The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value $H(X_{1},\ldots ,X_{n})$,
    $ND(X_{1},\ldots ,X_{n})={\frac {D(X_{1},\ldots ,X_{n})}{H(X_{1},\ldots ,X_{n})}}.$

## Relationship with Total Correlation

Dual total correlation is non-negative and bounded above by the joint entropy $H(X_{1},\ldots ,X_{n})$.
    $0\leq D(X_{1},\ldots ,X_{n})\leq H(X_{1},\ldots ,X_{n}).$
Secondly, Dual total correlation has a close relationship with total correlation, $C(X_{1},\ldots ,X_{n})$, and can be written in terms of differences between the total correlation of the whole, and all subsets of size $N-1$:
    $D({\textbf {X}})=(N-1)C({\textbf {X}})-\sum _{i=1}^{N}C({\textbf {X}}^{-i})$
where ${\textbf {X}}=\{X_{1},\ldots ,X_{n}\}$ and ${\textbf {X}}^{-i}=\{X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\}$
Furthermore, the total correlation and dual total correlation are related by the following bounds:
    ${\frac {C(X_{1},\ldots ,X_{n})}{n-1}}\leq D(X_{1},\ldots ,X_{n})\leq (n-1)\;C(X_{1},\ldots ,X_{n}).$
Finally, the difference between the total correlation and the dual total correlation defines a novel measure of higher-order information-sharing: the O-information:
    $\Omega ({\textbf {X}})=C({\textbf {X}})-D({\textbf {X}})$.
The O-information (first introduced as the "enigmatic information" by James and Crutchfield is a signed measure that quantifies the extent to which the information in a multivariate random variable is dominated by synergistic interactions (in which case $\Omega ({\textbf {X}})<0$) or redundant interactions (in which case $\Omega ({\textbf {X}})>0$.

## History

Han (1978) originally defined the dual total correlation as,
    ${\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\;.\end{aligned}}$
However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following:
    ${\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\\=&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]+(1-n)\;H(X_{1},\ldots ,X_{n})\\=&H(X_{1},\ldots ,X_{n})+\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})-H(X_{1},\ldots ,X_{n})\right]\\=&H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right)\;.\end{aligned}}$