PyMC3 and Bayesian Inference Towards Non-Linear Models: Part 1 | by Adam Watts | Towards Data Science

Probabilistic programming languages such as Stan have provided scientists and engineers accessible means into the world of Bayesian statistics. As someone who is comfortable programming in Python, I have been trying to find the perfect Bayesian library that complements Python’s wonderful and succinct syntax.
One of the biggest challenges in learning new material and a new library is finding sufficient resources and examples. I couldn’t find an accessible article on using Python and Bayesian inference for parameter and model uncertainty quantification (UQ) towards non-linear models. Therefore, I hope this article helps others trying to apply these methods towards their own research or work.
This will be 2 part article to make it accessible to a wide audience but also make navigation and reading the articles easier.
Part 1 will quickly discuss two common libraries for Bayesian inference: PyStan, and PyMC3. We will also set up a non-linear function to be used for Bayesian inference. This will include parameter optimization though the SciPy library. We will use the optimal parameters as a starting point for our Bayesian Inference and check our Bayesian method. If you’re already comfortable with parameter optimization, feel free to skip directly to the PyMC3 section in part 2.
Part 2 is where we begin using PyMC3 and dive into the syntax and results. For validation, we will also see how the Bayesian methods compare to the frequentist approach for parameter uncertainty quantification.
I am not going to divine much into Baye’s theorem and the entire Bayesian framework, there are plenty of articles on Medium to satisfy your curiosity. The primary outcome for this article is for the reader to understand how to utilize PyMC3 for quantifying the uncertainty in the parameter(s) and model used in linear, and in particular, non-linear models. The Jupyter Notebook can be found here.
Stan is available for Python in the form of PyStan, however, the syntax is rather offputting. Let’s take a look at an example of creating a model in PyStan which is assigned to a variable of type string. This model will then be compiled to C++ with the help of PyStan.
Above is a simple example of a quadratic model with two parameters, α, and β with the response y. The model is very verbose and difficult to debug when one starts to use more complicated models. There must be something more Pythonic...
I believe the PyMC3 is a perfect library for people entering into the world of probabilistic programming with Python. PyMC3 uses native Python Syntax making it easier to debug and more intuitive. Let’s dive into an example and see the prowess of the library.
I will be using an everyday example of a simple non-linear equation for this example with the hopes that the concepts will be easier for comprehension and retention.
Let’s begin with a cooling of a hot object using Newton’s law of cooling. We are assuming no thermal gradients exist in the object (lumped capacitance).
For those who are rusty at ordinary differential equations (ODE), fear not. The solution is below.
T(t) represents the temperature of the drink at a given time t. τ is our parameter of interest which governs the cooling rate for this problem. T_env is our environmental temperature and T_0 is the temperature of the drink at time t=0. We are assuming that T_env is invariant with time.
In Python:
Let us also assume that we know ahead of time, a priori, that the environmental temperature is 30C and that the object started at 100C. Finally, we will assume that the unknown parameter τ=15.
We will first plot the theoretical solution for the cooling of a hot drink, shown as the solid blue curve. Here we can get an intuition for the equations and parameters.
We will also generate some data with artificial noise to simulate real-life data, shown here as red scatter points. We will assume that the data is IID. The response, temperature of the drink, is Gaussian, i.e., T(t)~ N(μ, σ2). The mean, μ, will follow the solution above. We will address shortly the problem that we don’t know τ from the data itself. The standard deviation in the response is σ=1.5. That is how we generated the fake noise, we will also be able to estimate this from the data too.
From this simulated data, we want to estimate the cooling parameter τ and estimate the uncertainty in its value. Ideally, our estimate parameter and the uncertainty will encapsulate the true value. Similarly, we also would like our uncertainty in the standard error of the model to encapsulate σ.
To clarify what I mean by the term encapsulate, assume we estimate a parameter γ=100 ± 10. We want the true parameter to be within the range of 90–110 which is the same as 100 ± 10.
Before we can estimate the uncertainty in τ, we must first optimize the value of τ that fits the data the best. In more technical terms, we wish to find τ such that it minimizes the residual sum of squares.
Where the function within the left-hand side of the bracket is our data and the function within the right-hand side of the bracket is the solution to our ODE.
We find τ_opt = 14.93, pretty close estimate to the real parameter of τ=15. Returning back to our initial problem, remember that the response, temperature of the drink, is Gaussian, i.e., T(t)~ N(μ, σ2). We finally have an estimate for τ, and we can check for normality.
Remember that if our response came from a Gaussian distribution, then our residuals (data minus the predictions) must also be Gaussian.
residuals = T_data -  newton_cooling(tau_opt, T_0, T_env, time_data)
Think, linear transformation, or even more fundamental, the shift one makes for a Z-score.
The residual, o=difference between the data and our model, looks great with a mean of roughly 0. They appear stochastic, homoscedastic, with roughly most of the data between the two red dash lines.
Let’s see how we did for estimating σ
sigma_error_square = (1/(len(time_data)-1))*ss_minsigma_error = np.sqrt(sigma_error_square)print(sigma_error)
We find that the estimate for σ is 1.42, pretty close to 1.5
If you’re still with me, fantastic! Now that we have estimates for τ and σ, we can finally begin our dive into Bayesian statistics and PyMC3. Check out part 2 of the series here.