Gurobi Machine Learning:

Incorporate your Machine
Learning Models into
Optimization
Usage and model tuning
Zed DeanTechnical Account Manager

Alison Cozad
Gurobi Expert Team NA Manager

April 2023

© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved

AgendaOpen Source,sponsored by Gurobi
Collaborate, Experiment, and Innovate with
the Gurobi Team

Gurobi Machine Learning
PackageWhy, What, How
Types of Model Errors/Residuals

Tuning in Machine Learning
Package
Logistic regression with piece-wise
approximations (PWA)
Decision Tree, Random Forest & Gradientboosting.
Neural NetworkFinal thoughts© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 2

github.com/Gurobi
Open Source Projects

Sponsored by Gurobi

We aim to foster a collaborative
community around Gurobi by openly
developing various optimization
projects and tools, making themmore accessible and user-friendly.
Our users can experiment with ourinnovative tools, providing direct
feedback to our developers
responsible for creating thesepackages.

© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 3

Combining Machine Learning and Optimization
010203Training aML model

Use the ML predictions
to define the Opt. Model

Embed a ML model
inside the Opt. model

ML Algorithm

ML Modelthe lasso, best subset regression

demandforecast,inventory levels

Opt.
Algorithm
least squares
regression

Optimization
Model

numerical
prediction(ex: 5)revenue management

ML ModelOpt.
Modelsales as afunction ofpricefacilitylocation© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 4

AgendaOpen Source,sponsored by Gurobi
Collaborate, Experiment, and Innovate withthe Gurobi TeamGurobi Machine Learning
PackageWhy, What, HowTypes of Model Errors/Residuals

Tuning in Machine Learning
Package
Logistic regression with piece-wise
approximations (PWA)
Decision Tree, Random Forest & Gradientboosting.
Neural Network

Final thoughts© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 5

Gurobi Machine Learning
Our Goals• Simplify the process of importing a trained machine learning model built with apopular ML package into an optimization model.
• Improve algorithmic performance to enable the optimization model to explore asizable space of solutions that satisfy the variable relationships captured in theML model.
• Make it easier for optimization models to mix explicit and implicit constraints.
Other similar packages:• Janos (Bergman et. al, 2019)• ReLU_MIP (Lueg et. al, 2021)• OptiCL (Maragno et.al, 2021)• OMLT (Ceccon et. al, 2022)© 2022 ©
Gurobi
LLC. Confidential,
All Rights|62023 Optimization,
Gurobi Optimization,
LLC. Confidential,
AllReserved
Rights Reserved|6Gurobi Machine Learning
Regression Models Understood by Gurobi ( and which has controllable errors )• Linear/Logistic regression• Decision trees• Neural network with ReLU activation• Random Forests• Gradient Boosting trees• Transformations:• Simple scaling of features• Polynomial features of degree 2• Pipelines to combine them

• Dense layers• ReLU layers• Object Oriented, functional orsequential• Dense layers• ReLU layers• Only torch.nn.Sequential models© 2022 ©
Gurobi
LLC. Confidential,
All Rights|72023 Optimization,
Gurobi Optimization,
LLC. Confidential,
AllReserved
Rights Reserved|7Gurobi Machine Learning
Generic Predictor Constraint

The package has four main functions:• add_predictor_constr(gp_model, predictor, input_vars, output_vars=None, **kwargs)• print_stats(abbrev=False, file=None)• remove()• get_error()© 2022 ©
Gurobi
LLC. Confidential,
All Rights|82023 Optimization,
Gurobi Optimization,
LLC. Confidential,
AllReserved
Rights Reserved|8Types of errors between the real systemand the optimal solution
Real system

Measurement
error

Machine Learning
ModelData

Model
error

Optimization
Model

In some cases, errors canbe introduced whentranslating the ML modelto the optimization model© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 10

Source of Modeling Errors
In this session we will try to explain how to minimize the errors by using trained
Machine Learning Predictors in Gurobi.
The source of the errors:• Models for logistic regression use a piecewise linear approximation and can haveapproximation error (controlled by parameters).
• Models for decision tree, Random Forest and Gradient Boosting can also
introduce small Modelling errors at threshold values of node splitting (can becontrolled).
• Models for neural networks doesn’t introduce error.

© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 11

AgendaOpen Source,sponsored by Gurobi
Collaborate, Experiment, and Innovate with
the Gurobi Team

Gurobi Machine Learning
PackageWhy, What, How
Types of Model Errors/Residuals

Tuning in Machine Learning
Package
Logistic regression with piece-wise
approximations (PWA)
Decision Tree, Random Forest & Gradientboosting.
Neural Network

Link to the example

Final thoughts© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 12

Motivational Example: Student Enrollment• We show how to reproduce the model of student enrollment from [BHB+22] with
Gurobi Machine Learning.
• This model was developed in the context of the development of Janos, a toolkit
similar to Gurobi Machine Learning to integrate ML models and MathematicalOptimization.
• This example illustrates how to use the logistic regression and tune thepiecewise-linear approximation of the logistic function.
• This example illustrates how to use Gradient Boosting and tune the epsilon tominimize the error.
• We also show how to deal with fixed features in the optimization model using
pandas data frames.

© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 13

The objective:

The objective is to maximize of the enrolled students

The constraints:

Student Enrolment

Budget of 50000
Scholarship per student is max 2500

ExampleUseful Data form last year:• Problem Description• Formulation & Mathematical Model• Implementation• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 14

Using the Data from last year we could build a logistic
function ( using scikit) that predict the possibility ofenrollment if a student was giving certain merit(scholarship).

Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model• Implementation• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 15

Student Enrolment

Example• Problem Description• The logistic function

Variables:• Probability• Merit• Formulation & Mathematical Model• Implementation• Feature Discussion

Probability is a predictor variable.
Merit is an optimization decision variable that wasembedded into the logistic function.

© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 16

Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model• Implementation• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 17

The student enrolment notebook( Logistic Regression) :Code Highlights# classify our features between the ones that are
fixed and the ones that will be# part of the optimization problem
features = ["merit", "SAT", "GPA"]
target = "enroll"# Run our regression
scaler = StandardScaler()regression = LogisticRegression(random_state=1)
pipe = make_pipeline(scaler, regression)
pipe.fit(X=historical_data.loc[:, features],y=historical_data.loc[:, target])Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model• Implementation: notebook examples• gurobimachinelearning/student_admission.ipynb atmain · Gurobi/gurobi-machinelearning(github.com)• gurobi-machinelearning/Decision Tree.ipynb
at main · Gurobi/gurobi-machinelearning(github.com)• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 18

The student enrolment notebook( Logistic Regression) :Code Highlights
pred_constr = add_predictor_constr(m, pipe, studentsdata, y,output_type="probability_1")
pred_constr.print_stats()

Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model• Implementation: notebook examples• gurobimachinelearning/student_admission.ipynb atmain · Gurobi/gurobi-machinelearning(github.com)• gurobi-machinelearning/Decision Tree.ipynb
at main · Gurobi/gurobi-machinelearning(github.com)• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 19

The student enrolment notebook( Logistic Regression) :Code Highlights
print("Maximum error in approximating the regression{:.6}".format(np.max(pred_constr.get_error())))
Maximum error in approximating the regression
0.00715885

Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model• Implementation: notebook examples• gurobimachinelearning/student_admission.ipynb atmain · Gurobi/gurobi-machinelearning(github.com)• gurobi-machinelearning/Decision Tree.ipynb
at main · Gurobi/gurobi-machinelearning(github.com)• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 20

The student enrolment notebook( Logistic Regression) :Code Highlights
pred_constr.remove()pwl_attributes = {"FuncPieces": -1,"FuncPieceLength": 0.01,"FuncPieceError": 1e-5,"FuncPieceRatio": -1.0,}
pred_constr = add_predictor_constr(m, pipe, studentsdata, y,
output_type="probability_1",
pwl_attributes=pwl_attributes)
m.optimize()print("Maximum error in approximating the regression{:.6}".format(np.max(pred_constr.get_error())))
Maximum error in approximating the regression
4.47141e-06

Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model• Implementation: notebook examples• gurobimachinelearning/student_admission.ipynb atmain · Gurobi/gurobi-machinelearning(github.com)• gurobi-machinelearning/Decision Tree.ipynb
at main · Gurobi/gurobi-machinelearning(github.com)• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 21

The student enrolment notebook( Decision Tree) :Code Highlights# classify our features between the ones that are
fixed and the ones that will be# part of the optimization problem
features = ["merit", "SAT", "GPA"]
target = "enroll"# Run our regression
regression = DecisionTreeRegressor(max_depth=10,max_leaf_nodes=50, random_state=1)

Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model

regression.fit(X=historical_data.loc[:, features],y=historical_data.loc[:, target])• Implementation: notebook examples• gurobimachinelearning/student_admission.ipynb atmain · Gurobi/gurobi-machinelearning(github.com)• gurobi-machinelearning/Decision Tree.ipynb
at main · Gurobi/gurobi-machinelearning(github.com)• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 22

The student enrolment notebook( Decision Tree) :Code Highlights# classify our features between the ones that are
fixed and the ones that will be# part of the optimization problem
features = ["merit", "SAT", "GPA"]
target = "enroll"# Run our regression
regression = DecisionTreeRegressor(max_depth=10,max_leaf_nodes=50, random_state=1)

Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model

regression.fit(X=historical_data.loc[:, features],y=historical_data.loc[:, target])• Implementation: notebook examples• gurobimachinelearning/student_admission.ipynb atmain · Gurobi/gurobi-machinelearning(github.com)• gurobi-machinelearning/Decision Tree.ipynb
at main · Gurobi/gurobi-machinelearning(github.com)• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 23

The student enrolment notebook( Decision Tree) :Code Highlights# Add Trained constraint
pred_constr = add_predictor_constr(m, regression,studentsdata, y)
print("Error in approximating the regression{:.6}".format(np.max(np.abs(pred_constr.get_error()))))
Error in approximating the regression 1.0

Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model• Implementation: notebook examples• gurobimachinelearning/student_admission.ipynb atmain · Gurobi/gurobi-machinelearning(github.com)• gurobi-machinelearning/Decision Tree.ipynb
at main · Gurobi/gurobi-machinelearning(github.com)• Feature Discussion© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 24

The student enrolment notebook( Decision Tree) :Code Highlights# Remove pred_constr
pred_constr.remove()# Add new constraint setting epsilon to 1e-5
pred_constr = add_predictor_constr(m, regression,studentsdata, y, epsilon=1e-5)
m.optimize()print("Error in approximating the regression{:.6}".format(np.max(np.abs(pred_constr.get_error()))))

Student Enrolment

Example• Problem Description• The logistic function• Formulation & Mathematical Model• Implementation: notebook examples• gurobimachinelearning/student_admission.ipynb atmain · Gurobi/gurobi-machinelearning(github.com)• gurobi-machinelearning/Decision Tree.ipynb
at main · Gurobi/gurobi-machinelearning(github.com)• Feature Discussion

Error in approximating the regression 5.54244e16© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 25

Tunning Decision Tree:Decision Tree Regression and node splitting.
• We add one binary decisionvariable for each node of the tree(and each input vector).
• We differentiate between
splitting nodes and leafs of thetree• By definition, if a node is on thedecision path, then we proceed .
• We use an indicator constraint tomodel that if it moves is on thedecision path, the output value ofthe output is fixed to with anindicator constraint.
This all cause a minimal error.

© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 26

Decision Tree Regression•It only corresponds to a small perturbation in the values of the input variables•

The default value for � is 0.

•

Adding small epslison value could easily address the error•

pred_constr = add_predictor_constr(m, regression, studentsdata, y,epsilon=1e-5)

Mixed Integer Formulations — Gurobi Machine Learning documentation (gurobimachinelearning.readthedocs.io)© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 27

Tunning Logistic Regression :Logistic Regression and Piecewise-LinearApproximation PWA• Function constraints in Gurobi• Allow to state 𝑦 = 𝑓 𝑥••

𝑓 is a predefined function
𝑦 and 𝑥 are one-dimensional variables

𝑝 𝑥 =11 + 𝑒 !(#)• Gurobi automatically performs a piecewise-linear
approximation of 𝑓 in the domain of 𝑥.

• Added logistic function to our set of predefined 𝑓.
• Why it is important• Probability = Odds / (1 + Odds)• # y = 1 / (1 + exp(-x))• gc = model.addGenConstrLogistic(x, y)© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 28Why it is important ? : Making probability alinear Equation
Odds
12000

Delta Log
Odds

Log Odds-9.210-6.908-4.605-2.303
0.000
2.303
4.605
6.908
9.210

10000

Odds on Success Prob of Success Delta Probability
0.0001
0.01%
#N/A
2.303
0.001
0.10%
0.09%
2.303
0.01
0.99%
0.89%
2.303
0.1
9.09%
8.10%
2.303
1
50.00%
40.91%
2.303
1090.91%
40.91%
2.303
100
99.01%
8.10%
2.303
1000
99.90%
0.89%
2.303
10000
99.99%
0.09%

8000
6000
4000
2000
00246Log Odds

81015.000

Probabilities

10.000120.00%100.00%5.000

80.00%
0.000

60.00%

040.00%246810-5.000

20.00%-10.000

0.00%
0246810-15.000© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 29How Gurobi PWA simplifies model development
process
Piecewise Linear Constraint – Without Gurobi

They consistently use piecewise linear constraints in our model.

• One such example is to calculate the annual
tax payable given a taxable income.

Tax Payable

Annual Federal Tax

Taxable Income

Source:Optimizing Your Financial Future: A Goals-BasedApproach to Financial Planning - Gurobi Optimization30© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 30

How Gurobi PWA simplifies model development
process
Piecewise Linear Constraint – Without GurobiThis piecewise linear function can be modelled with the following constraints:• 𝒃𝟏 + 𝒃𝟐 + 𝒃𝟑 = 𝟏𝑡% = 0, 1, 2, 3 and 𝑓% = 0, 2, 6, 15• 𝟎 ≤ 𝒔𝒊 ≤ 𝒃𝒊 𝒇𝒐𝒓 𝒊 = 𝟏, 𝟐, 𝟑
• 𝑡 = 𝑡% 𝑏% + (𝑡& − 𝑡% )𝑠% + 𝑡& 𝑏& + 𝑡' − 𝑡& 𝑠& + 𝑡' 𝑏' + (𝑡( − 𝑡' )𝑠'• 𝑡 = 0 : 𝑏% + (1 − 0)𝑠% + 1𝑏& + (2 − 1)𝑠& + 2𝑏' + (3 − 2)𝑠'
• 𝒕 = 𝒔𝟏 + 𝒃𝟐 + 𝒔𝟐 + 𝟐𝒃𝟑 + 𝒔𝟑

Generic Number (not actual tax amount)
16• 𝑓 = 𝑓% 𝑏% + (𝑓& −𝑓% )𝑠% + 𝑓& 𝑏& + 𝑓' − 𝑓& 𝑠& + 𝑓' 𝑏' + (𝑓( − 𝑓' )𝑠'• 𝑓 = 0 : 𝑏% + (2 − 0)𝑠% + 2𝑏& + 6 − 2 𝑠& + 6𝑏' + (15 − 6)𝑠'
• 𝒇 = 𝟐𝒔𝟏 + 𝟐𝒃𝟐 + 𝟒𝒔𝟐 + 𝟔𝒃𝟑 + 𝟗𝒔𝟑• Let 𝑡) be the taxable income, and 𝑓) be the tax payable.
• Let 𝑏% , 𝑏& , …, 𝑏*+% be binary variables such that 𝑏) ∈ {0, 1}• Let 𝑠% , 𝑠& , …, 𝑠*+% be segment variables such that 𝑠) ∈ 𝑅

Tax Payable, f

14
12
10
8
6
4
2
0
0

0.5

1

1.5

2

2.5

3

3.5

Taxable Income, t31© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 31How Gurobi PWA simplifies model development
process
Piecewise Linear Constraint – With Gurobi

Gurobi abstracts allthese cumbersome
works, all we need todo is add theconstraint to ourmodel with (Python):model.addGenConstrPWL(xvar=t, yvar=f, xpts=[0, 1, 2, 3], ypts=[0, 2, 6, 15])32

© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 32

Parameters: Is it easy to handle?
The details of the approximation are controlled using thefollowing four attributes (or using the parameters with the samenames):
FuncPieces, number of pieces.
FuncPiecesLength, desired width of each piece.
FuncPieceError, equal to the maximum absolute approximation
you are willing to tolerate, Gurobi will do pieces accordingly.
For details, check the General Constraint discussion© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 33

Tunning the student enrolment modelError Before TuningError After Tunning

Logistic Regression

0.00715885

4.47141e-06

Decision Tree

1.0

5.54244e-16© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 34

Tunning Neural network• Models for neural networks don’t introduce errors.
• However, get_error() will still report errors if the model suffer numerical issues,Guidelines for Numerical Issues - Gurobi Optimization© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 35

FAQ- Discussions• Which ML models would be more useful to integrate with optimization© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 36

Final thoughts
On the Gurobi ML package

Benchmark all ML regressors before making a final
decision which one to use.

Gurobi Machine Learning Package may generate
minimal controllable errors that could be handledeasilyLogistic regression error could be handled by adjusting
PWA (piecewise-linear approximation) parameters

github.com/Gurobi

Decision Tree, Random Forest and Gradient Boostingerrors can be handled by changing an epsilon default
value from 0 to very small value© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 37

Thank You

Link to the example

For more information: gurobi.com
Zed DeanTechnical Account Manager

Alison Cozad
Gurobi Expert Team NA Manager

dean@gurobi.com

cozad@gurobi.com© 2023 Gurobi Optimization, LLC. Confidential, All Rights Reserved