Higher Order Numeric Differential Equations(Python) | by Rahul Tarak | Towards Data Science

By the nature of the topic, it is expected to have an understanding of calculus and basic programming to achieve this.
Install anaconda from https://www.anaconda.com/distribution/
import numpy as npimport matplotlib.pyplot as pltfrom mpl_toolkits.mplot3d import Axes3D # For 3D Plotting of Systems
I’d encourage you to play around with the code on Google colab, for which the link is here or https://colab.research.google.com/drive/14KT0sLGFphRF_QeL8e3by7tfNY0xkzq5
To provide some background as to why we care about truncation error, and why we need to use higher-order methods to achieve lower truncation errors
If you are confused about the Taylor Expansion, I’d recommend 3blue1brown’s video on the subject
From calculus, we know that the truncation error is the next term in the Taylor series, the (n+1)th term of the expansion. That is they are errors from approximate an infinite sum, as partial sums.
In terms of a computer, there is a term called machine epsilon or double precision. This is the accuracy beyond which a computer cannot tell the difference between two numbers. That is a truncation error of machine epsilon, while in mathematics is not considered an accurate result and still an approximation, in scientific computing it is considered accurate.
In the above equation, h or dx is the time step. The equation has a truncation error of ~O(h), which means to decreases the error from e-8 to e-10, the new time step would have to e-2 smaller than the old-time step. Which makes the increase in accuracy for Euler’s Method extremely slow.
Now one might ask the question, why does the speed of converge matter. Well apart from the fact that higher speed of converge will decrease the wait time of large computations, there is another reason. The reason is another error we face called the round-off error.
A round off error,[1] also called rounding error,[2] is the difference between the result produced by a given algorithm using exact arithmetic and the result produced by the same algorithm using finite-precision, rounded arithmetic.
This essentially means there is a slight error in every computation we do, in every time step. And the more time steps or, the more computation we perform, the more the round-off error adds up. There is a point where the round-off error will exceed the truncation error, and performing more computations will make the resulting value inaccurate.
The above example highlights the truncation error and round-off errors of two different approximations, one with a truncation error ~h2 represented in purple and one with truncation error ~h4 represented in yellow.
Notice in the above graph, that the purple graph with a truncation error of order 2 ~h2, which would be similar to Euler’s method reaches a much lower accurate(higher truncation error) than the higher-order method in yellow.
This is because the error only scales in a quadratic manner, that is if you reduce the time step by half, the error reduces by a fourth. While the higher-order method of order 4, for the same reduction of time step by half the resulting error would reduce by a sixteenth.
This method works for any order; however, in this post, we will focus on 4th order methods, specifically RK4.
If we expand the Taylor expansion to the order we desire, we will get the higher-order methods.
However, it is normally difficult to differentiate the differential equations to get d/dx(f(x)), but different mathematician have found different 4th order methods, the one that this post will focus on, was discovered by Runge, it is called RK4.
This function has truncation error ~O(h4), which means to move from e-8 to e-10, the difference in time step would be ∜(e-2) which would make the number of computations much lower.
In this post, I am not going too deeply into the mathematics, as I believe there are a lot of practical applications in computer science that can be achieved without a deep understanding of the underlying mathematics. I have linked to resources below to help understand the math better.
Finally, the fun part begins, so we have to choose which system of differential equations we going to solve. I’m showing an example of a more complex system of differential equations, as this will cover the skills required to solve simpler systems.
The system we are solving for today is the Lorenz System, more commonly known as the butterfly effect.
The above snippet performs the operation of a single time step, that is the dt move in [dx,dy,dz] as a vector. Looking at this as a vector operation makes it much easier to understand, here is where NumPy shines, allowing you to solve this time step as a vector function.
The above function is a general rk4, time step which is essential to solving higher order differential equations efficiently, however, to solve the Lorenz System, we need to set up some other functions to use this formula.
First, let’s set up the functions dx, dy, dz with the constants of the Lorenz System
# Constants of the Lorenz Systemb = 8/3s = 10r = 28# Functions dx,dy,dz of the Lorenz Systemdef dx(t,x,y,z):    return s*(y-x)def dy(t,x,y,z):    return x*(r-z) -ydef dz(t,x,y,z):    return x*y - b*z
Now to make a function to calculate the points of the lorenz system
Finally, we can start with the exiting part, graphing and visualizing the data.
Here we will be using the Axes3D module we imported earlier.
#Calling the previous function arbitrary start pointsx,y,z = lorenz(0,1,1.05,0,0.01)fig = plt.figure(figsize=(12,10))# This is how we set the 3D plotax = fig.gca(projection='3d')# Plotting the valuesax.plot(x, y, z, lw=0.5,alpha=0.7) # Plotting the final valuesax.scatter(x[-1],y[-1],z[-1],color=(1,0,0))#Very slightly different starting valuesx_1,y_1,z_1 = lorenz(0,0.9,1.04,0,0.01)#Plotting slightly different starting valuesax.scatter(x_1[-1],y_1[-1],z_1[-1],color=(0,0,0))#Setting Axis Titlesax.set_xlabel("X Axis")ax.set_ylabel("Y Axis")ax.set_zlabel("Z Axis")ax.set_title("Lorenz Attractor")
This highlights the phenomenon of chaos inherent to the Lorenz System, of how slightly different starting values create vastly different endpoints. Again this post is not a resource about the Lorenz system, which you can learn more about here https://en.wikipedia.org/wiki/Lorenz_system
Create your differential equations as functions
Using RK4 solve the set of differential equations as vector
Create a list of all the points from rk4
Plot the points in 3D using Axis3D and matplotlib
Once again, I would highly encourage you to play around with the code for yourself, which you can do at https://colab.research.google.com/drive/14KT0sLGFphRF_QeL8e3by7tfNY0xkzq5