Solving your first linear program in Python | by Bhaskar Agarwal | Towards Data Science

You might have come across the term 'linear programming' at some point in data science or research. I will try to explain what it is and how one can implement a linear program in Python.
A linear program finds an optimum solution for a problem where the variables are subject to numerous linear relationships. Furthermore, the problem could require one to maximise or minimise a certain condition, for example minimise the cost of a product, or maximise the profit.
An often discussed example of a linear program is that of the traveling salesman. Starting from his hometown a salesman needs to travel all cities of a district but in order to minimise traveling costs he must take the shortest possible path that crosses each city and ends in his hometown. Generally the best possible path is one that crosses each city only once, thereby resembling a closed loop (starting and ending in the hometown).
Personally I have applied linear programs in a multitude of applications.
Transport Industry: For route optimisation where various depots had to be visited while minimising the operational costs.
Energy industry: Optimise the electricity consumption for a household with a solar panel, while predicting the load pattern.
Logic Problems: Solving logic problems/puzzles using a linear program where all logical constraints have to be satisfied 'in parallel' (topic of the next post).
Bar talk: Think high-school mathematics, linear equations and solving simultaneous linear equations. Basically that is the idea, but add to this an extra condition on the variables that needs to be minimised or maximised.
Technical talk: A linear program optimises an objective function (or cost function) where a set of linear equalities (and inequalities) needs to be satisfied.
I will demonstrate what all this means with a real life example.
I need to bake a cake with four ingredients: flour, eggs, butter and sugar. However, I remember only bits and parts of the recipe as stated below.
total weight of all ingredients was exactly 700 grams
amount of butter was half that of sugar
weight of flour and eggs together was at most 450 grams
weight of eggs and butter combined was at most 300 grams
the weight of eggs plus butter was at most that of flour plus sugar
Is it possible to figure out the optimum values for each of the ingredients from the information above?
The answer is yes. This is what a linear program is designed to do.
We can define each of the variables as
flour: f
eggs: e
butter: b
sugar: s
We can then write each of the conditions as
Note that the equations above follow a pattern. All variables appear on the left and all constants on the right. The inequalities are expressed as less than or equal to. Once we are able to express a problem in the manner above, a linear program can be constructed. The last piece of the puzzle is an objective function. A cost function or objective function is basically another linear relationship (between the variables we are solving for) that requires to be maximised or minimised. This actually further simplifies the question.
An example of a health-conscious cost function could be the following
minimise the butter and sugar consumption
or in other words
We are almost ready to code our linear problem in Python. However, we can formulate the problem in a slightly more code-friendly way. Mathematically we can express a set of linear equations in matrix form which helps us visualise the problem computationally.
We define the following matrices
where the first row contains all our variables that we are solving for in the matrix X and our cost function as c. The equality matrices are in the second row, and matrices describing the inequality conditions are in the last row. The matrices are defined such that they follow the rules of matrix multiplication. This formulation allows us to write our equations as
Thus, using linear algebra our problem takes the form
Note that we use the transpose (superscript T) of matrix c so that we can multiply it with our solution matrix X.
We can now start coding this problem in Python.
My Python setup
Python 3.8.2
SciPy 1.18.1
Numpy 1.4.1
Cvxopt 1.2.3 (optional)
Using SciPy
SciPy in Python offers basic linear programming capabilities. To implement the above program using SciPy, we need to define all matrices accordingly. Below is a working example of the equations above that I implemented using SciPy's optimize library.
Which returns the following output
WITHOUT BOUNDS      con: array([ 5.27055590e-08, -3.78719278e-11])      fun: 249.99999998121916  message: 'Optimization terminated successfully.'      nit: 5    slack: array([3.39247208e-08, 6.95661413e+01, 7.24656159e+01])   status: 0  success: True        x: array([302.8994746 , 147.10052537,  83.33333333, 166.66666665])
The success field in the output above tells us if the optimisation was successful. In fact this message is explicitly returned in the field message or encoded as an integer value in the status filed which can take 5 integer values ranging form 0 to 4. Finally the x field contains the variables we were solving for, returned in the exact order we defined them with our matrix formulation. Thus the solution is
flour = 302.89g
eggs = 147.10g
butter = 83.33g
sugar = 166.66g
The result satisfies all of the conditions we imposed earlier. Total weight of all ingredients is still exactly 700g, the amount of butter and sugar is the minimum possible, butter is still half of sugar and so on.
Note: If you did not get the values above, it might have something to do with the version of SciPy you used and/or the default method used to execute the linear program, i.e. simplex vs. interior-point
Using SciPy and adding bounds for variables
I can make the problem a little more refined by adding some estimates for my variables. For example I remember that the flour was not more than 300g, and that the one time I had a 100g stick of butter I was able to make the cake. I make reasonable guesses for the rest of the variables.
Which returns
WITH BOUNDS      con: array([ 4.53132998e-09, -3.25428573e-12])      fun: 249.9999999983819  message: 'Optimization terminated successfully.'      nit: 6    slack: array([2.91322522e-09, 5.30261661e+01, 3.93856656e+01])  status: 0  success: True        x: array([286.35949946, 163.64050053,  83.33333333, 166.66666667])
The output now implies
flour = 286.35g
eggs = 163.64g
butter = 83.33g
sugar = 166.66g
Note that now the individual amounts of butter and sugar are higher than before. This is perfectly acceptable because the linear program is trying to simultaneously satisfy all the conditions we encoded, including our newly imposed bounds for each of the ingredients. The total is still 700g, butter is still half of sugar, butter is still less than 100g (a bound we imposed), flour +butter is still more than or equal to eggs + sugar and so on. This is the beauty of a linear program. It aims at satisfying every single condition it was passed, therefore returning the most optimum solution.
I wrote a simple parser for the result which returns a more readable output in the form of a dictionary.
With the output
Result (no bounds):  {'Flour': 303.0, 'Eggs': 147.0, 'Butter': 83.0, 'Sugar': 167.0}Result (with bounds):  {'Flour': 286.0, 'Eggs': 164.0, 'Butter': 83.0, 'Sugar': 167.0}
Using another linear programming library
In Python there are many libraries (CVXOPT, PULP, CVXPY, ECOS, etc.) that basically come with their own LP solvers or act as wrappers around other LP solvers. Thus one can choose a wrapper-solver combination depending on the problem at hand.
Below is the same problem preamble (the matrices and equations) implemented using the CVXOPT libray with a GLPK solver. The main difference here is that one needs to define the equations in cvxopt's own matrix framework. Note that CVXOPT is rather flexible and allows one to use a number of solvers depending on the problem. I did not pass any bounds for the variables.
Which returns the output
solution found [ 2.67e+02] [ 1.83e+02] [ 8.33e+01] [ 1.67e+02]
This is between the two solutions we found using SciPy's own linear optimisation routine above. While the amount of butter and sugar remains consistent, the flour and eggs have different values. Why is that? One reason is that our problem might not be well constructed (especially for values of flour and eggs), but the three solutions are equally acceptable as they satisfy all the constraints. Note that in the last two solutions the sum of flour and eggs remains the same, however the individual values differ by 20 grams. One might need to add another constraint to break this degeneracy.
We can break this degeneracy by adding the equation or knowledge that the weight of flour plus sugar is exactly 500 grams, or in other words
Could you try and add this to the relevant matrices (and code). Then re-run the SciPy solver without bounds, and CVXOPT without bounds to see if the values of the ingredients match?
The entire notebook and the addition of this equation is hosted here.
In general, the solution that a particular solver finds comes down to how a given solver actually solves the linear program. This is the topic of another post and I will refrain from going into the details here.