Conditional Independence — The Backbone of Bayesian Networks | by Aerin Kim | Towards Data Science

Sometimes the explanation in Wikipedia is not the easiest to understand.
Let’s say A is the height of a child and B is the number of words that the child knows. It seems when A is high, B is high too.
There is a single piece of information that will make A and B completely independent. What would that be?
The child’s age.
The height and the # of words known by the kid are NOT independent, but they are conditionally independent if you provide the kid’s age.
A: The height of a childB: The # of words that the child knowsC: The child's age
A better way to remember the expression:
Conditional independence is basically the concept of independence P(A ∩ B) = P(A) * P(B) applied to the conditional model.
But I’ve seen the definition of not only P(A ∩ B|C) but also P(A|B ∩ C)!
Why is P(A|B ∩ C) = P(A|C) when (Aᅭ B)|C?
Here goes the proof.
The gist of conditional independence: Knowing C makes A and B independent.
P(A,B|C) = P(A|C) * P(B|C)
Why does the conditional independence even matter?
Because it is a foundation for many statistical models that we use. (e.g., latent class models, factor analysis, graphical models, etc.)
A Bayesian network represents a joint distribution using a graph. Specifically, it is a directed acyclic graph in which each edge is a conditional dependency, and each node is a distinctive random variable. It has many other names: belief network, decision network, causal network, Bayes(ian) model or probabilistic directed acyclic graphical model, etc.
It looks like so:
In order for the Bayesian network to model a probability distribution, it relies on the important assumption: each variable is conditionally independent of its non-descendants, given its parents.
For instance, we can simplify P(Grass Wet|Sprinkler, Rain) into P(Grass Wet|Sprinkler) since Grass Wet is conditionally independent of its non-descendant, Rain, given Sprinkler.
Using this property, we can simplify the whole joint distribution into the formula below:
Conditional independence between variables can greatly reduce the number of parameters.
This reduces so much of the computation since we now only take into account its parent and disregard everything else.
Let’s take a look at the numbers.
Let’s say you have n binary variables (= n nodes).
The unconstrained joint distribution requires O(2^n) probabilities.
For a Bayesian Network, with a maximum of k parents for any node, we need only O(n * 2^k) probabilities. (This can be carried out in linear time for certain numbers of classes.)
n = 30 binary variables, k = 4 maximum parents for nodes• Unconstrained Joint Distribution: needs 2^30 (about 1 million) probabilities -> Intractable!• Bayesian Network: needs only 480 probabilities 
We can have an efficient factored representation for a joint distribution using Conditional independence.
Let’s say I’d like to estimate the engagement (clap) rate of my blog. Let p be the proportion of readers who will clap for my articles. We’ll choose n readers randomly from the population. For i = 1, ..., n, let Xi = 1 if the reader claps or Xi = 0 if s/he doesn’t.
In a frequentist approach, we don’t assign the probability distribution to p. p would be simply ‘sum (Xi) / n’. And we would treat X1, ..., Xn as independent random variables.
On the other hand, in Bayesian inference, we assume p follows a distribution, not just a constant. In this model, the random variables X1, ..., Xn are NOT independent, but they are conditionally independent given the distribution of p.
“Correlation is not causation” means that just because two things correlate does not necessarily mean that one causes the other.
Here is a hilarious example of taxi accidents:
A study has shown a positive and significant correlation between the number of accidents and taxi drivers’ wearing coats. They found that coats might hinder the driver’s movements and cause accidents. A new law was ready to ban taxi drivers from wearing coats while driving.
Until another study pointed out that people wear coats when it rains... 😂
P(accidents, coats | rain) = P(accidents | rain) * P(coats | rain)
Correlations between two things can be caused by a third factor that affects both of them. This third factor is called a confounder. The confounder, which is rain, was responsible for the correlation between accident and wearing coats.
P(accidents | coats, rain) = P(accidents | coats)
Note that this does NOT mean accidents are independent of rain. What it means is: given drivers wearing coats, knowing rain doesn’t give any more information about accidents.
Marginal independence is just the same as plain independence,
Sometimes, two random variables might not be marginally independent. However, they can become independent after we observe some third variable.
Amount of speeding fine ᅭ Type of car | Speed
Lung cancer ᅭ Yellow teeth | Smoking
Child’s genes ᅭ Grandparents’ genes | Parents’ genes
Is the car’s starter motor working? ᅭ Is the car’s radio working?| Battery
Future ᅭ Past | Present (This is the Markov assumption!)
They are all in the same form as A ᅭ B | C. A and B look related if we don’t take C into account. However, once we include C in the picture, then the apparent relationship between A and B disappears. As you see, any causal relationship is potentially conditionally independent. We will never know for sure about the relationship between A & B until we test every possible C (confounding variable)!