Finding Causal Models is Hard
By: Marin Vlastelica Pogančić
Tags: Optimization, Artificial Intelligence, Machine Learning, Data Science, Statistics

Finding Causal Models is Hard

Why is it so hard to find Structrual Causal Models? A DAG perspective.

The link between correlation and causation has been discussed in science for quite a bit. Alas, we are all quite certain that correlation can be misleading. Hence the scientific community gave birth to causal inference and causal discovery. Previously I wrote an article on statistics versus causality, which is a bit more general than this one. In this article, I want to make it clear what exactly makes it so difficult to find causal models, in the context of graphical models, and yes, it is difficult.

[1]

Firstly, to understand the power of causal models, consider graphical models in general for describing probability distributions. Bayesian networks are such models (coincidentally introduced by the same person that introduced do calculus, Judea Pearl). In the case of Bayesian networks, we want to model a probability distribution. More concretely, we want to model a joint probability distribution. But not just in any way, we want to do it in a smart way, such that its computational overhead is minimal. Another way to put it, we want a sparse factorization of the joint probability distribution.

To be clearer, let's write it down in simple terms. Let us assume we have n random variables, so the joint probability distribution that we are interested in is

Now, for those of you familiar with the Bayes rule, we can write out this joint probability distribution as the following:

As you can see, there are quite a few conditionals in this factorization depending on n. There are two things to notice here:

Calculating those conditionals can be computationally expensive The statistics may be misleading

Thankfully, we can use the basic concept of independence in statistics to make this factorization sparser. The simple rule is if two variables are independent, then we can write their joint distribution as

This means that we can remove certain variables from certain conditionals, which makes the computation easier and this can be nicely formulated in terms of a graphical model where the children are only dependent on their parents. As an example, consider the following Bayesian network (graphical model):

question.In the absence of arrows, assume that the top nodes are always the parents of the bottom ones. From the graph, we can read out the presented factorization of the joint distribution:

Finding the correct directed acyclic graph (DAG) is a hard problem. There are many such factorizations that may explain the data but are incorrect in the sense that they ignore certain independencies. So keep in mind that, even before we start thinking about causality, we have a hard problem to identify the correct dependencies!

We know that graphical models are nice in the sense that they can save computation time. But we still have the problem of the interpretation of a graphical model. Can we actually be sure that the parents cause the children variables? Not at all, general Bayesian networks don’t really care about the causal direction. There are various ways that we can factorize the joint distribution such that it explains the data. Now, for comparison, causal models have a few traits that other graphical models don’t. The causal direction manifests itself in the form of a directed edge in the graph. The edge itself symbolizes the causal mechanism. In general, we require a bit more from our causal graphs than the vanilla Bayesian networks. We require certain invariances to hold in order to claim that the arrows represent causal mechanisms.

[1]

Causal Inference and Causal Discovery

When we are looking for a causal model, we can be looking for different things. We may have models that explain the data, we have a graphical model, but we do not know the direction of the causal mechanism. Or we may search for the whole graphical models in the first place. The thing that we can be sure about is that the causal model is minimal, or is the sparsest possible factorization that explains the data.

Another necessary assumption about causal models is the independence of causal mechanisms (ICM principle), meaning that if we have information of one causal mechanism, we cannot get any information about the others. Interestingly, the same assumption is made by the human brain in vision, called the generic viewpoint assumption.

The classical example of this is the Beuchet chair. When we look at something, our brain assumes that the object that we’re looking at (in this case, the chair), is independent of the viewpoint. This assumption may not hold, since the viewpoint in this case determines if the object is a chair or not (in our perception). This can be seen in the image below.

[2]

The mechanism causing something to be a chair is completely independent of the mechanism that allows us to see (namely light coming into the eye). In the case that this assumption doesn’t hold, we couldn’t say much about the cause and effect, since weird statistical dependencies would arise. Even in the case of fitting a model in computer vision, if the dataset is plagued with these kind of examples, we couldn’t really guarantee good performance of our model.

This is just the tip of the iceberg of the assumptions that are needed to be made to discover causal models. You have to appreciate the hardness of the problem as such.

Not to get into too much detail, an example of a Structural Equation Model is a simple two-variable model which can be written in two lines:

The causal link is the function f which is a function of X1 and some noise variable N2. To make the graphical model from above causal, we would simply put arrows on it to indicate the causal direction:

Well, that’s nice, but the problem is that we are really not sure about the arrows. We can reverse the arrows as an example like in the image below. We can probably find models that explain the correlation in the anti-causal direction, which is not correct:

Searching for causal models is difficult. Again, think about the number of arrow arrangements that we can do in such a model. Each edge has 2 possible directions, if a causal model has E edges, that means that there exist 2^E possible direction arrangements. The search space (just for the edge directions) is exponential in the number of edges. If we think about that we do not know where the edges are in the first place, and that we have V vertices (variables). We have this nice formula for the possible edge configurations in the graph:

This means that for each graph from the above formula, we have additional 2^E possible arrangements of its edges… This is a real mess for high dimensional problems. And then when we consider that we might actually want to learn the mechanisms themselves, i.e. to fit the models… What a mess. This is to show that…

Causal discovery is a difficult problem!

It is an open research question to find methods to identify causal direction between variables efficiently. But in some cases, the causal direction is simply not identifiable. This depends on the type of functions that describe the causal mechanism, or type of noise distributions that are used within the mechanism. Consequently, the assumption of identifiability is made when we want to find causal directions in the data. A big portion of the research is causal inference is deriving identifiability results stating under which assumptions can we find the underlying causal model. Of course, the more we can relax these assumptions, the better.

[1]

As it turns out, sometimes we indeed want to learn the anti-causal direction (i.e. to predict the cause from the effect), since there are cases where this is perfectly predictable, but this would also mean that our model is not invariant towards changes in distribution. The usefulness of causal inference in machine learning is quite a difficult topic also, everyone is speaking about “the next level of AI required causal reasoning”, but actually nobody really realizes what this encapsulates. The research is flooded with wrong uses of the word “causal”. Put, to end with a positive outlook, that leaves a lot of room for contributions!