A Basic Intro to AIC and BIC
By: Kevin Macias-Matsuura
Tags: Aic, Bic

Photo by Engin Akyurt

Why do academics worldwide fawn over formulas such as E=MC²? Besides the real world impact of harnessing nuclear power there is an inherent balance of simplicity and complexity. Consider the following quote from the Chinese philosopher Lao Tsu:

See simplicity in the complicated.

Achieve greatness in little things. In the universe the difficult things are done as if they are easy.

In the universe great acts are made up of small deeds.

The sage does not attempt anything very big,

And thus achieves greatness. Tao Te Ching — Lao Tsu — Chapter 63

Coding can quickly become an exercise in piling on and accounting for contingencies. I know I am guilty of getting in a “flow”, getting up to get a glass of water, and returning to find that my code has become more tangled than an amateur knitter’s first scarf. Put simply: in coding, as in life, often times less is more. This is the driving force behind the values of AIC and BIC, otherwise known as the Akaike Information Criterion and Bayesian Information Criterion. The former was developed by the statistician Hirotugu Akaike while the latter was developed by statistician Gideon E. Schwartz.

Both of these formulas essential do the same thing but in a slightly different way. Let’s take Akaike’s formula first to build an understanding which will seamlessly transfer to the BIC. The formula is written as follows:

In this formula k is equal to number of parameters in your model, ln is the natural log (not to be confused with the base 10 logarithm), and L-hat is the Maximum Likelihood Estimate (MLE). Here again we have simplicity meeting complexity. With only 3 elements, we can begin to understand the AIC.

The 2k part of the formula raises your score as a punishment for overly complex models. The more features you add, the more your score goes up. The second half of the formula is where the heart of it lies. Simply put, it drives your score down as your model better fits your data. Not so simply put, the maximum likelihood estimate deals with a probability density estimation. “Density estimation involves selecting a probability distribution function and the parameters of that distribution that best explain the joint probability distribution of the observed data (X) . . . In a more precise sense, the PDF is used to specify the probability of the random variable falling within a particular range of values, as opposed to taking on any one value.” (Jason Brownlee, A Gentle Introduction to Maximum Likelihood Estimation for Machine Learning). If that didn’t quite sink in, what The AIC is basically doing is taking a distribution at different points along a set of values and taking note as to where it can be placed in order to best capture the most values in a range of values. In my head I think of it like net fishing. A basic fisherman would randomly throw a net into a lake and hope for the best. A data-science-fisherman would cast his net, record the amount of fish he caught, record the location, then use the data to select the best place to cast his net in order to catch the most fish. With AIC just replace the area of the pond with your best predictive features.

With some basic understanding we can now talk about the BIC. It goes as follows:

It is essentially the same as the AIC with a slight twist. In BIC, instead of multiplying our parameters (k) by 2, we multiply them by ln(n) which is the natural log of the number of observations in our data as represented by n. While slightly different, the idea is the same. As the parameters and observations go up, the BIC score goes up, punishing us for overly complicated models with large amounts of data. Remember that adding more parameters will allow your model to explain more of the data but can lead to overfitting and poor performance overall. The second half of the formula functions the same way as in AIC. The better our model fits our data, the lower the score will go.

Every method comes with its issues. AIC and BIC come with underlying assumptions and its use of the Maximum Likelihood Estimation carries with it, ideas inherent in both Frequentist and Bayesian mathematics. They are likelihood estimations which is different from strict probability. MLE “can be considered a special case of maximum a posteriori (MAP) estimation, where we use a naïve prior and never bother to update it.” (Jonathan Balaban, A Gentle Introduction to Maximum Likelihood Estimation). Basically, MAP allows prior information akin to updating based on known information in Bayesian probability. If you want a rabbit hole to jump down you can look up a wealth of info on probability versus likelihood and likelihood estimates.

Now that we know what AIC and BIC are and some of the issues, how should we think about them in terms of our linear regression models? Best practice seems to be using them in concert with each other. You may want to find a model that falls somewhere in between a good score for AIC and BIC. For example if a 3 feature model gives you a good AIC and poor BIC, and a 6 feature model gives you a poor AIC but a good BIC, you would want to select a model with 4 or 5 features.

When running a linear regression model you may want to keep these values in mind as you refine and select your final model. Restraint and selective moves can often yield better results than piling things on top of each other. If all else fails and you find your fists clenched like rocks falling from a mountainside onto your keyboard as you stare at a floating abyss of alphanumeric characters; you can pause and think back to this quote, again, from Lao Tsu: