Integer Programming in Python | by Freddy Boulton | Towards Data Science

Integer Programming (IP) problems are optimization problems where all of the variables are constrained to be integers. IP problems are useful mathematical models for how to best allocate one’s resources. Let’s say you’re organizing a marketing campaign for a political candidate and you’re deciding which constituents to send marketing materials to. You can send each constituent a catchy flyer, a detailed pamphlet explaining your agenda, or a bumper sticker (or a combination of the three). If you had a way to measure how likely someone is to vote for your candidate based on the marketing materials they received, how would you decide which materials to send while also being mindful to not exceed your supply?
Traditional optimization algorithms assume the variables can take on floating point values, but in our case, it isn’t reasonable to send someone half a bumper sticker or three quarters of a pamphlet. We’ll use a special python package called cvxpy to solve our problem such that the solutions make sense.
I’ll show you how to use cvxpy to solve the political candidate problem, but I’ll start first a simpler problem called the knapsack problem to show you how the cvxpy syntax works.
Let’s pretend you’re going on a hike and you’re planning which objects you can take with you. Each object has a weight in pounds w_i and will give you u_i units of utility. You’d like to take all of them but your knapsack can only carry P pounds. Suppose you can either take an object or not. Your goal is to maximize your utility without exceeding the weight limit of your bag.
A cvxpy problem has three parts:
Creating the variable: We will represent our choice mathematically with a vector of 1’s and 0’s. A 1 will mean we’ve selected that object and a 0 will mean we’ve left it home. We construct a variable that can only take 1’s and 0’s with the cvxpy.Bool object.Specifying the constraints: We only need to make sure that the sum of our objects doesn’t exceed the weight limit P. We can compute the total weight of our objects with the dot product of the selection vector and the weights vector. Note that cvxpy overloads the * operator to perform matrix multiplication.Formulating the objective function: We want to find the selection that maximizes our utility. The utility of any given selection is the dot product of the selection vector and the utility vector.
Creating the variable: We will represent our choice mathematically with a vector of 1’s and 0’s. A 1 will mean we’ve selected that object and a 0 will mean we’ve left it home. We construct a variable that can only take 1’s and 0’s with the cvxpy.Bool object.
Specifying the constraints: We only need to make sure that the sum of our objects doesn’t exceed the weight limit P. We can compute the total weight of our objects with the dot product of the selection vector and the weights vector. Note that cvxpy overloads the * operator to perform matrix multiplication.
Formulating the objective function: We want to find the selection that maximizes our utility. The utility of any given selection is the dot product of the selection vector and the utility vector.
Once we have a cost function and constraints, we pass them to a cvxpy Problem object. In this case, we’ve told cvxpy that we’re trying to maximize utility with cvxpy.Maximize. To solve the problem, we just have to run the solve method of our problem object. Afterwards, we can inspect the optimal value of our selection vector by looking at its value attribute.
print(selection.value)matrix([[1.],        [1.],        [1.],        [1.],        [0.],        [1.],        [0.],        [0.],        [0.],        [0.]])
We’ve selected the first four items and the sixth one. This makes sense because these have high ratio of utility to weight without weighing too much.
Now that we’ve covered basic cvxpy syntax, we can solve the marketing optimization problem for our political candidate. Let’s say we had a model that takes in a constituent’s attributes and predicts the probability they will vote for our candidate for each combination of marketing materials we send them. I’m using fake data here but let’s pretend the model outputs the following probabilities:
print(test_probs[0:5])array([[0.0001, 0.0001, 0.3   , 0.0001, 0.2   , 0.0001, 0.2   , 0.3   ],       [0.1   , 0.1   , 0.1   , 0.2   , 0.2   , 0.2   , 0.1   , 0.0001],       [0.1   , 0.0001, 0.2   , 0.0001, 0.1   , 0.2   , 0.0001, 0.4   ],       [0.3   , 0.0001, 0.0001, 0.2   , 0.1   , 0.2   , 0.2   , 0.0001],       [0.2   , 0.3   , 0.1   , 0.0001, 0.2   , 0.1   , 0.1   , 0.0001]])
There are eight total probabilities per constituent because there are eight total combinations of materials we could send an individual. Here’s what the entries of each 1 x 8 vector represent:
[1 flyer, 1 pamphlet, 1 bumper sticker, flyer and pamphlet, flyer and bumper sticker, pamphlet and bumper sticker, all three, none].
For example, the first constituent has a 0.0001 probability of voting for our candidate if he received a flyer or pamphlet, but a 0.3 probability of voting for our candidate if we sent him a bumper sticker.
Before we get into the cvxpy code, we’ll turn these probabilities into costs by taking the negative log. This makes the math work out nicer and it has a nice interpretation: if the probability is close to 1, the negative log will be close to 0. This means that there is little cost in sending that particular combination of materials to that constituent since we are certain it will lead them to vote for our candidate. Vice versa if the probability is close to 0.
#clipping so that we don't take log of 0 or 1test_probs = np.clip(test_probs, 0.0001, 0.9999)#turning into costsmodel_costs = -np.log10(test_probs)
Finally, let’s say that we cannot send more than 150 flyers, 80 pamphlets, and 25 bumper stickers.
supply = np.atleast_2d([150, 80, 25])
Now that we’ve done all the set-up, we can get to the fun part:
Creating the variable: We’ll use a cvxpy.Bool object again because we can only make binary choices here. We’ll specify that it must be the same shape as our probability matrix:
Creating the variable: We’ll use a cvxpy.Bool object again because we can only make binary choices here. We’ll specify that it must be the same shape as our probability matrix:
selection = cvxpy.Bool(*test_probs.shape)
2. Specifying the constraints: Our selection variable will only tell us which of the 8 choices we’ve made for each constituent, but it won’t tell how many total materials we’ve decided to send to them. We need a way to turn our 1 x 8 selection vector into a 1 x 3 vector. We can do so by multiplying by the selection vector by the following matrix:
TRANSFORMER = np.array([[1,0,0],                        [0,1,0],                        [0,0,1],                        [1,1,0],                        [1,0,1],                        [0,1,1],                        [1,1,1],                        [0,0,0]])
If this part is a little confusing, work out the following example:
print(np.dot(np.array([0,0,0,1,0,0,0,0]), TRANSFORMER))array([1, 1, 0])
The fourth entry of our decision vector represents sending a flyer and pamphlet to the constituent and multiplying by TRANSFORMER tells us just that! So we’ll tell cvxpy that multiplying our selection matrix by transformer can’t exceed our supply:
supply_constraint = cvxpy.sum_entries(selection * TRANSFORMER, axis=0) <= supply
I’m summing over the rows with cvxpy.sum_entries to aggregate the total number of materials we’ve sent across all constituents.
We also need to ensure we’re making exactly once choice per constituent, otherwise the solver can attain zero cost by not sending anyone anything.
# We must make our choice per constituent# remember that the last column is for "no materials"feasibility_constraint = cvxpy.sum_entries(selection, axis=1) == 1constraints = [supply_constraint, feasibility_constraint]
3. Formulating the objective function: The total cost of our assignment will be the sum of the costs we’ve incurred for each constituent. We’ll use the cvxpy.mul_elemwise function to multiply our selection matrix with our cost matrix, this will select the cost for each constituent, and the cvxpy.sum_elemwise function will compute the total cost by adding up the individual costs.
cost = cvxpy.sum_entries(cvxpy.mul_elemwise(model_costs, selection))
The final step is to create the cvxpy.Problem and solve it.
# Solving the problemproblem = cvxpy.Problem(cvxpy.Minimize(cost), constraints=constraints)problem.solve(solver=cvxpy.GLPK_MI)
That’s it! Below is a snapshot of what our final assignments look like. We decided not to send any materials to the first constituent. This makes sense because the probability of them voting for our candidate is 0.3 whether we send them a bumper sticker or nothing at all.
It also turns out that the optimal assignment exhausts our supply of pamphlets and bumper stickers but only uses 83 of the total 150 flyers. We should tell our candidate that her flyers aren’t as persuasive as she thinks.
print(hard_assignments.value[0:5])matrix([[0., 0., 0., 0., 0., 0., 0., 1.],        [0., 0., 0., 1., 0., 0., 0., 0.],        [0., 0., 0., 0., 0., 0., 0., 1.],        [1., 0., 0., 0., 0., 0., 0., 0.],        [1., 0., 0., 0., 0., 0., 0., 0.]])print(np.dot(hard_assignments.value, TRANSFORMER).sum(axis=0))matrix([[83., 80., 25.]])
Here is all the code packaged together:
I hope you’ve enjoyed learning about integer programming problems and how to solve them in Python. Believe it or not, we’ve covered about 80% of the cvxpy knowledge you need to go out and solve your own optimization problems. I encourage you to read the official documentation to learn about the remaining 20%. CVXPY can solve more than just IP problems, check out their tutorials page to see what other problems cvxpy can solve.
To install cvxpy, follow the directions on their website. I would also install cvxopt to make sure all the solvers that come packaged with cvxpy will work on your machine
We’ve specified that cvxpy should use the GLPK_MI solver in the solve method. This is a special solver designed for IP problems. Before you solve your own problem, consult this table to see which prepackaged cvpxy solver is best suited to your problem.