(MI)NLPLib 2
Stefan Vigerske

16th July 2015
ISMP, Pittsburgh

Model instance collections
Collecting optimization problems has been a popular “hobby” for long time, e.g.,
first release library
problem types
1985
NetlibLinear Programming
1992
MIPLIB
Mixed-Integer Programming
1993
CUTE
Nonlinear Programming
1998
SDPLib
Semidefinite Programming
1999
CSPLibConstraint Satisfaction Programming
199x
MacMINLP
Mixed-Integer Nonlinear Programming
2001
GAMS World LP, MIP, NLP, MINLP, SOCP, MPEC
2003
COCONUT
Nonlinear and Constraint Satisfaction Programming
2008
mintOC
Mixed-Integer Optimal Control
2009
minlp.org
MINLP, General Disjunctive Programming
2011
POLIP
Mixed-Integer Polynomial Programming
2014
CBLIB
Conic Programming
I for solver developers, access to a wide set of interesting problem instanceswith different characteristics has always been important
I commercial solver vendors test their solver on thousands of test problems
before releasing a new software version
I the evaluation of algorithmic improvements (w.r.t. robustness and efficiency)
requires well-balanced test sets of significantly many real-world instances 2 / 64

MINLPLib and GLOBALLibI

http://www.gamsworld.org

Initiated in 2001 (as part of GamsWorld/MinlpWorld/GlobalWorld):
M. Bussieck, A. Drud, and A. Meeraus
MINLPLib – A Collection of Test Models for Mixed-Integer Nonlinear Programming
INFORMS Journal on Computing 15, 114–119 (2003)

I“white-box” NLPs (GLOBALLib) and MINLPs (MINLPLib)I

frequently used for testing, but also benchmarking
3 / 64MINLPLib and GLOBALLib InstancesI

scalar GAMS format
Variables x1,x2,b3,b4,b5,objvar;
Positive Variables x1,x2;
Binary Variables b3,b4,b5;
Equations e1,e2,e3,e4,e5,e6;
e1.. - 2*x1 - 3*x2 - 1.5*b3 - 2*b4 + 0.5*b5 + objvar =E= 0;e2.. sqr(x1) + b3 =E= 1.25;
e3.. x2**1.5 + 1.5*b4 =E= 3;e4..
x1 + b3 =L= 1.6;
e5..
1.333*x2 + b4 =L= 3;
e6.. - b3 - b4 + b5 =L= 0;

4 / 64MINLPLib and GLOBALLib InstancesI

scalar GAMS format
Variables x1,x2,b3,b4,b5,objvar;
Positive Variables x1,x2;
Binary Variables b3,b4,b5;
Equations e1,e2,e3,e4,e5,e6;
e1.. - 2*x1 - 3*x2 - 1.5*b3 - 2*b4 + 0.5*b5 + objvar =E= 0;e2.. sqr(x1) + b3 =E= 1.25;
e3.. x2**1.5 + 1.5*b4 =E= 3;e4..
x1 + b3 =L= 1.6;
e5..
1.333*x2 + b4 =L= 3;
e6.. - b3 - b4 + b5 =L= 0;

Ivarying from small scale (great for debugging!) to large scale real world
instances (agricultural economics, chemical-, civil-, and electrical engineering,finance, management, OR)I

intentionally including instances from badly formulated models or different
formulations of the same problem

4 / 64MINLPLib and GLOBALLib InstancesI

scalar GAMS format
Variables x1,x2,b3,b4,b5,objvar;
Positive Variables x1,x2;
Binary Variables b3,b4,b5;
Equations e1,e2,e3,e4,e5,e6;
e1.. - 2*x1 - 3*x2 - 1.5*b3 - 2*b4 + 0.5*b5 + objvar =E= 0;e2.. sqr(x1) + b3 =E= 1.25;
e3.. x2**1.5 + 1.5*b4 =E= 3;e4..
x1 + b3 =L= 1.6;
e5..
1.333*x2 + b4 =L= 3;
e6.. - b3 - b4 + b5 =L= 0;

Ivarying from small scale (great for debugging!) to large scale real world
instances (agricultural economics, chemical-, civil-, and electrical engineering,finance, management, OR)I

intentionally including instances from badly formulated models or different
formulations of the same problem

I

including solution points for many instances

4 / 64MINLPLib and GLOBALLib InstancesI

scalar GAMS format
Variables x1,x2,b3,b4,b5,objvar;
Positive Variables x1,x2;
Binary Variables b3,b4,b5;
Equations e1,e2,e3,e4,e5,e6;
e1.. - 2*x1 - 3*x2 - 1.5*b3 - 2*b4 + 0.5*b5 + objvar =E= 0;e2.. sqr(x1) + b3 =E= 1.25;
e3.. x2**1.5 + 1.5*b4 =E= 3;e4..
x1 + b3 =L= 1.6;
e5..
1.333*x2 + b4 =L= 3;
e6.. - b3 - b4 + b5 =L= 0;

Ivarying from small scale (great for debugging!) to large scale real world
instances (agricultural economics, chemical-, civil-, and electrical engineering,finance, management, OR)I

intentionally including instances from badly formulated models or different
formulations of the same problem

I

including solution points for many instancesI

solely an instance collection, i.e., consisting of instantiations of models by
specific data sets
4 / 64MINLPLib and GLOBALLib History
Iinstances were harvested from existing collections, initially:
GAMS Model Library
MacMINLP (Leyffer)
I MINOPT library (Floudas)
I Handbook of Test Problems in Local and Global Optimization (Floudas et.al.)
I
I

I

2001 – 2009: maintained by Michael BussieckI

new instances were addedI

new incumbent solutions were addedIin 2009: Michael “volunteered” me as maintainer
700

GLOBALLib + MINLPLib - Number of Instances

600
500
400
300
200
100
0

2003

2005

2007

2009

2011

2013

5 / 64MINLPLib 2

Tasks:
I Adding new problem instances:both convex and nonconvex problems(MI)QPs, (MI)QCQPs, and (MI)NLPs
I easy solvable, solvable, difficult to solve, but not trivial
I
I

I

Categorizing instances
convexity
problem type (quadratic, polynomial, general nonlinear)
I function types (powers, exp/log, trigonometric, ...)
I solved to global optimality?
I
I

I

Providing feasible best known solutions

Work in progress, current version publicly available:
http://www.gamsworld.org/minlp/minlplib2/html/index.html.
6 / 64

New NLP and MINLP Instances
1400

MINLPLib 2 - Number of Instances

1200
1000
800
600
400
200
0

GLOBALLib + MINLPLib 1
MINLPLib 2
2003 2005 2007 2009 2011 2013 2015
7 / 64

Sources of newly added instances
Harvesting mainly fromI CMU-IBM open source MINLP project (convex MINLPs)
I minlp.org
I POLIP (polynomial MINLPs)

MINLPLib 2 instance sources (1357 in total)
minlp.org
Bonmin test library

BARON book
ANTIGONE test library
MacMINLP
Westerlund
GAMS clients
MINOPT
Library
VecchiettiModellibrary

Floudas e.a. handbook

otherGAMS Model Library
POLIP

8 / 64

Instance Formats

Format
GAMS
AIMMS
AMPL
AMPL
OSIL
CPLEX LP
PIP

.gms
.ams
.mod
.nl
XML
.lp
.pip

#instances
1363
1352
(no Gamma, latest additions missing)1337(no errorf/signpower/Gamma/...)
1331(no errorf/signpower/Gamma/..., latest missing)1342(no signpower/Gamma/...)
667(limited to quadratics)
770(limited to polynomial)

9 / 64

Problem types

1000
800
600
400
200

Operands in general nonlinear functions
GLOBALLib + MINLPLib 1
new in MINLPLib 2

300
200
100
0

quadratic polynomial signomial

general

mu
divl
sq
logr
e
vcp xp
ow
sqer
r
st
po in
we
cvp cosr
olowg er
err 10oa rf
rpo bswemo rd
sig mincenpowngatroper
mmy
a

0

400
Number of instances

Number of instances

1200

Equation Types
GLOBALLib + MINLPLib 1
new in MINLPLib 2

10 / 64

Sparsity Pattern – Examples
dosemin2d eg_all_s feedtray2
Radiation Therapy

Feed Tray Location

johnall
Asset Management

mbtd

qapw
Quadratic Assigment(top: Objective Gradient and Jacobian; bottom: Lagrangian Hessian)
11 / 64

Sparsity Pattern – Examples (cont.)
Jacobian densitymod (Density modification based on single-crystal X-ray diffraction data; 23529 vars, 550 cons.)
Jacobian lop97ic (Rail Line Optimization, MIQCQP)

milinfract (Solving Mixed-Integer Linear Fractional Programming Problems with Dinkelbach’s Algorithm)
Objective Gradient + Jacobian
Lagrangian Hessian

12 / 64(Non)Convexity Detection for Functions
Analyze the Hessian:I
I

Given twice differentiable function h(x) and variable bounds [x, x̄].
Compute the spectrum of the Hessian in one random point and conclude
I
I

convexity/concavity/indefiniteness if h(x) is quadratic
nonconvexity/nonconcavity if h(x) is general nonlinear

13 / 64(Non)Convexity Detection for Functions
Analyze the Hessian:I
I

Given twice differentiable function h(x) and variable bounds [x, x̄].
Compute the spectrum of the Hessian in one random point and conclude
I
I

convexity/concavity/indefiniteness if h(x) is quadratic
nonconvexity/nonconcavity if h(x) is general nonlinear

Analyze the Algebraic Expression:(
convex, a ≥ 0
f (x) convex ⇒ a · f (x)concave, a ≤ 0
f (x), g (x) convex ⇒ f (x) + g (x) convex
f (x) concave ⇒ log(f (x)) concaveconvex, ei ≤ 0 ∀iYPeif (x) =
xi , xi ≥ 0 ⇒ f (x) convex, ∃j : ei ≤ 0 ∀i 6= j; i ei ≥ 1P
iconcave, ei ≥ 0 ∀i; i ei ≤ 1

13 / 64(Non)Convexity Detection for Functions
Analyze the Hessian:I
I

Given twice differentiable function h(x) and variable bounds [x, x̄].
Compute the spectrum of the Hessian in one random point and conclude
I
I

convexity/concavity/indefiniteness if h(x) is quadratic
nonconvexity/nonconcavity if h(x) is general nonlinear

Analyze the Algebraic Expression:(
convex, a ≥ 0
f (x) convex ⇒ a · f (x)concave, a ≤ 0
f (x), g (x) convex ⇒ f (x) + g (x) convex
f (x) concave ⇒ log(f (x)) concaveconvex, ei ≤ 0 ∀iYPeif (x) =
xi , xi ≥ 0 ⇒ f (x) convex, ∃j : ei ≤ 0 ∀i 6= j; i ei ≥ 1P
iconcave, ei ≥ 0 ∀i; i ei ≤ 1
Analyze manually.
13 / 64(Non)Convexity in MINLPLib
Numerical analysis of (pointwise) Hessians by LAPACK.
Symbolic analysis of expressions by SCIP.
I Mark additional 71 instances (5%) as convex.
I
I

MINLPLib instances convexity
convex26%

undecided3%

nonconvex
71%

14 / 64

Solution Points
MINLPLib instances traditionally come with known feasible solution points.

15 / 64

Solution Points
MINLPLib instances traditionally come with known feasible solution points.
For MINLPLib 2, we added
Feasibility checking:I

compute maximal (unscaled) violation of constraints, variable bounds, anddiscreteness restrictions

I

uses GAMS/EXAMINER2

15 / 64

Solution Points
MINLPLib instances traditionally come with known feasible solution points.
For MINLPLib 2, we added
Feasibility checking:I

compute maximal (unscaled) violation of constraints, variable bounds, anddiscreteness restrictions

I

uses GAMS/EXAMINER2

Solution polishing: For a given point,
1. project onto variable bounds
2. round values for discrete variables to exact integers
3. ensure that semicontinuity/semiintegrality and special-ordered-set constraints
are exactly satisfied
4. run CONOPT on MINLP with all binary/integer/semi*/SOS variables fixed,
start from updated point, scaling disabled, feasibility tolerance 10−9

15 / 64

Polished Solution Points
160

1200 MINLPLib 2 solution points w.r.t. infeasibility

MINLPLib 1 solution points w.r.t. infeasibility

140

1000

100
80
60
40
20
0
10-15 10-13 10-11 10-9 10-7 10-5 10-3 10-1 101 103 105 107 109
Infeasibility

Number of points

Number of points

120

800
600
400
200
0
10-1510-1310-11 10-9 10-7 10-5 10-3 10-1 101 103 105 107 109
Infeasibility

16 / 64

Polished Solution Points
160

1200 MINLPLib 2 solution points w.r.t. infeasibility

MINLPLib 1 solution points w.r.t. infeasibility

140

1000

100
80
60
40
20
0
10-15 10-13 10-11 10-9 10-7 10-5 10-3 10-1 101 103 105 107 109
Infeasibility

Number of points

Number of points

120

800
600
400
200
0
10-1510-1310-11 10-9 10-7 10-5 10-3 10-1 101 103 105 107 109
Infeasibility

Available in two formats:
GAMS Data Exchange (GDX)

ASCII (.sol)
x11.11803398874989001754
x2
1.31037069710444997739
b4
1.00000000000000000000
b5
1.00000000000000000000
objvar 7.66718006881313041134
16 / 64Dual Bounds(
lower bound on optimal value,
dual bound =upper bound on optimal value,

if minimization
if maximization

17 / 64Dual Bounds(
lower bound on optimal value,
dual bound =upper bound on optimal value,

if minimization
if maximization

Collected dual bounds fromI

solvers for general (MI)NLP(ANTIGONE, BARON, Couenne, Lindo, SCIP)

I

solvers for convex MINLP on proven convex MINLPs(AlphaECP, Bonmin BB, Bonmin Hyb)

17 / 64Dual Bounds(
lower bound on optimal value,
dual bound =upper bound on optimal value,

if minimization
if maximization

Collected dual bounds fromI

solvers for general (MI)NLP(ANTIGONE, BARON, Couenne, Lindo, SCIP)

I

solvers for convex MINLP on proven convex MINLPs(AlphaECP, Bonmin BB, Bonmin Hyb)

But: No way to verify correctness of bound!

17 / 64Dual Bounds(
lower bound on optimal value,
dual bound =upper bound on optimal value,

if minimization
if maximization

Collected dual bounds fromI

solvers for general (MI)NLP(ANTIGONE, BARON, Couenne, Lindo, SCIP)

I

solvers for convex MINLP on proven convex MINLPs(AlphaECP, Bonmin BB, Bonmin Hyb)

But: No way to verify correctness of bound!
Conservative approach: Only trust a solvers dual bound claim if it has beenverified by at least 2 other solvers.

17 / 64

“Open” instances
Feasible solution points ⊕ trusted dual bounds ⇒ trusted gap

1000

Number of instances

800

MINLPLib histogram w.r.t. trusted gap
GLOBALLib + MINLPLib 1
new in MINLPLib 2

600
400
200
0 0.0

0.2

0.4
0.6
Trusted Gap

0.0 , ≤ 10−9

0.8

1.0

1.0 , ≥ 1.0
18 / 64

Query the MINLPLib
Simple script to select instances by specific criteria, e.g.:
I all large convex instances, show # var. and # cons.:$ ./query.py "(nvars > 4242) & (convex == True)" -c nvars -c ncons
nvars
ncons
jbearing100
5304
0squfl030-150
4530
4650
watercontamination0202 106711 107209
watercontamination0303 107222 108217
I

all quadratic instances:./query.py "npolynomfunc == 0 & nsignomfunc == 0 & ngennlfunc == 0"I

all instances with trigonometric functions:./query.py "(opsin == True) or (opcos == True)"I

all separable instances, sorted by problem type:./query.py "nlaghessiannz == nlaghessiandiagnz" -s probtype -c probtype
I

all unsolved instances (w.r.t. “trusted” dual bounds), zipped up:./query.py "gap > 0.1" -c gap -z open.zip

19 / 64

What to do with all these instances?

20 / 64

What to do with all these instances?

General Purpose Global Solvers Benchmark?
date
GAMS ANTIGONE BARON COUENNE

LINDO

SCIP

07/15

9.0.1983.157

3.2.0

24.5α

1.1

15.6.50.5...

20 / 64Today: Go Columnwise
date

GAMS

ANTIGONEBARONCOUENNE

LINDO

SCIP

08/11
04/12
11/12
02/13
07/13
05/14
09/14
06/15
07/15

23.7.323.8.223.9.524.0.224.1.324.2.324.3.324.4.6
24.5α

–
–
–
–
1.1
1.1
1.1
1.1
1.1

9.3.1
10.2.011.5.2
11.9.112.3.312.7.714.0.314.4.015.6.5

0.3
0.4
0.4
0.4
0.4
0.4
0.4
0.4
0.5

6.1.1.588
7.0.1.421
7.0.1.497
7.0.1.497
8.0.1283.385
8.0.1694.498
8.0.1694.550
9.0.1983.157
9.0.1983.157

–
2.1.1
2.1.2
3.0
3.0
3.0
3.1
3.1
3.2.0

I

ANTIGONE by R. Misener (Imperial College) and Ch. Floudas (Texas A&M)

I

BARON by N. Sahinidis (CMU), M. Tawarmalani (Purdue), et.al.

I

Couenne by P. Belotti (now FICO), et.al.; open-source (COIN-OR)

I

Lindo API by Lindo Systems Inc.

ISCIP by Zuse Institute Berlin, TU Darmstadt, FAU Erlangen; free for academic use

Quantify Improvements of global MINLP solvers over the last 4 years!
21 / 64

Which instances to run?
Does the current MINLPLib 2 with its 1363 instances qualify as a good test set?

22 / 64

Which instances to run?
Does the current MINLPLib 2 with its 1363 instances qualify as a good test set?
X large number of instances⇒ 40 solver versions × 1363 instances = 54520 runs (!)

22 / 64

Which instances to run?
Does the current MINLPLib 2 with its 1363 instances qualify as a good test set?
X large number of instances⇒ 40 solver versions × 1363 instances = 54520 runs (!)
X wide variety of applications

22 / 64

Which instances to run?
Does the current MINLPLib 2 with its 1363 instances qualify as a good test set?
X large number of instances⇒ 40 solver versions × 1363 instances = 54520 runs (!)
X wide variety of applications
× dominance of certain models, e.g.,
32 block layout design problems
60 small investor portfolio optimization instances
I ...
I
I

22 / 64

Which instances to run?
Does the current MINLPLib 2 with its 1363 instances qualify as a good test set?
X large number of instances⇒ 40 solver versions × 1363 instances = 54520 runs (!)
X wide variety of applications
× dominance of certain models, e.g.,
32 block layout design problems
60 small investor portfolio optimization instances
I ...
I
I

× many trivial, some hopeless, some numerically dubious instances

22 / 64

Which instances to run?
Does the current MINLPLib 2 with its 1363 instances qualify as a good test set?
X large number of instances⇒ 40 solver versions × 1363 instances = 54520 runs (!)
X wide variety of applications
× dominance of certain models, e.g.,
32 block layout design problems
60 small investor portfolio optimization instances
I ...
I
I

× many trivial, some hopeless, some numerically dubious instancesThus, need to select a reasonable subset of (e.g., 87) instances.

22 / 64

Which instances to run?
Does the current MINLPLib 2 with its 1363 instances qualify as a good test set?
X large number of instances⇒ 40 solver versions × 1363 instances = 54520 runs (!)
X wide variety of applications
× dominance of certain models, e.g.,
32 block layout design problems
60 small investor portfolio optimization instances
I ...
I
I

× many trivial, some hopeless, some numerically dubious instancesThus, need to select a reasonable subset of (e.g., 87) instances.
With 15 co-authors and 8 months of time, this would be no problem.

But with 3 weeks until ISMP: Apply the P.I.T.T. heuristic.
22 / 64

Prune Instances by Tractability and Triviality Heuristic
1. Remove intractable instancesconsider only the 881 instances that aremarked as solved in MINLPLib 2

104

800
Number of instances

Number of Nonlinear Nonzeros in Jacobian

I

1000

MINLPLib histogram w.r.t. trusted gap
GLOBALLib + MINLPLib 1
new in MINLPLib 2

600
400
200
0 0.0

103

0.2

0.4
0.6
Trusted Gap

0.8

1.0

102
101
100
100

101
102
103
Number of Discrete Variables (+1)

23 / 64

Prune Instances by Tractability and Triviality Heuristic
2. For each solver separately:
Remove instances that are solved within 60 seconds by the oldest solver
version (e.g., as in GAMS 23.7).

I

Remove instances that the solver cannot handle (due to trigonometricfunctions, SOS, . . .).
Number of Nonlinear Nonzeros in JacobianI

In case of SCIP:

104
103
102
101
100
100

101
102
103
Number of Discrete Variables (+1)
24 / 64

Prune Instances by Tractability and Triviality Heuristic
For SCIP, this leaves 312 instances:

alkylation
fo9_ar5_1
clay0304hex6_1_1
house
arki0003
gasnetclay0305hex6_1_3
jbearing25
arki0005
genpooling_meyer04
crudeoil_lee2_10 ex6_2_12
jbearing75
arki0006
ghg_1veh
crudeoil_lee3_07 ex6_2_14
johnallarki0019
ghg_3veh
crudeoil_lee3_08 ex6_2_8
kall_circles_c6a
arki0024
glider100
crudeoil_lee3_09 ex6_2_9
kall_circles_c6b
autocorr_bern20-10 crudeoil_lee3_10 ex7_2_4
graphpart_2g-0066-0066 kall_circles_c7a
autocorr_bern20-15 crudeoil_li06
graphpart_2g-0077-0077
ex8_1_7
kall_circles_c8a
autocorr_bern25-06 csched1agraphpart_2g-0088-0088 kall_circlespolygons_c1pex8_2_1b
autocorr_bern25-13 edgecross10-060 ex8_2_4bgraphpart_2g-0099-9211 kall_circlespolygons_c1p
autocorr_bern30-04 edgecross10-070 ex8_4_1
graphpart_2pm-0066-0066 kall_circlesrectangles_c
autocorr_bern35-04 edgecross10-080 ex8_4_3
graphpart_2pm-0077-0777 kall_circlesrectangles_cbatch0812_nc
graphpart_2pm-0088-0888 kall_congruentcircles_c7
edgecross14-039 ex8_4_4
batchs201210m
graphpart_2pm-0099-0999 kall_diffcircles_10
edgecross14-058 ex8_4_5
bayes2_50
edgecross14-078 ex8_4_8_bnd graphpart_3g-0334-0334 kall_diffcircles_5b
blend480
graphpart_3g-0344-0344 kall_diffcircles_7
edgecross14-176 filter
blend531
graphpart_3g-0444-0444 kall_diffcircles_8
edgecross20-040 fin2bb
blend718
graphpart_3pm-0244-0244 kall_diffcircles_9
edgecross22-048 flay05h
blend852
graphpart_3pm-0333-0333 launch
emfl050_5_5
fo7
carton7graphpart_3pm-0334-0334 lop97icx
emfl100_5_5
fo8
casctanks
ethanolh
fo8_ar25_1 graphpart_3pm-0344-0344 mathopt5_7
cecil_13
graphpart_3pm-0444-0444 mathopt5_8
ethanolm
fo8_ar2_1
chemgraphpart_clique-30
ex1252afo9
mhw4d
clay0203hex14_1_1
fo9_ar25_1 graphpart_clique-40
milinfract
clay0204h
gsg_0001
ex14_1_7
fo9_ar2_1
minlphix
clay0205hhda
ex4_1_5
fo9_ar3_1
minsurf100
clay0303hheatexch_trigen
ex4_1_6
fo9_ar4_1...
25 / 64

Prune Instances by Tractability and Triviality Heuristic
For SCIP, this leaves 312 instances – obvious dominance by some models:alkylation
fo9_ar5_1
clay0304hex6_1_1
house
arki0003
gasnetclay0305hex6_1_3
jbearing25
arki0005
genpooling_meyer04
crudeoil_lee2_10 ex6_2_12
jbearing75
arki0006
ghg_1veh
crudeoil_lee3_07 ex6_2_14
johnallarki0019
ghg_3veh
crudeoil_lee3_08 ex6_2_8
kall_circles_c6a
arki0024
glider100
crudeoil_lee3_09 ex6_2_9
kall_circles_c6b
autocorr_bern20-10 crudeoil_lee3_10 ex7_2_4
graphpart_2g-0066-0066 kall_circles_c7a
autocorr_bern20-15 crudeoil_li06
graphpart_2g-0077-0077
ex8_1_7
kall_circles_c8a
autocorr_bern25-06 csched1agraphpart_2g-0088-0088 kall_circlespolygons_c1pex8_2_1b
autocorr_bern25-13 edgecross10-060 ex8_2_4bgraphpart_2g-0099-9211 kall_circlespolygons_c1p
autocorr_bern30-04 edgecross10-070 ex8_4_1
graphpart_2pm-0066-0066 kall_circlesrectangles_c
autocorr_bern35-04 edgecross10-080 ex8_4_3
graphpart_2pm-0077-0777 kall_circlesrectangles_cbatch0812_nc
graphpart_2pm-0088-0888 kall_congruentcircles_c7
edgecross14-039 ex8_4_4
batchs201210m
graphpart_2pm-0099-0999 kall_diffcircles_10
edgecross14-058 ex8_4_5
bayes2_50
edgecross14-078 ex8_4_8_bnd graphpart_3g-0334-0334 kall_diffcircles_5b
blend480
graphpart_3g-0344-0344 kall_diffcircles_7
edgecross14-176 filter
blend531
graphpart_3g-0444-0444 kall_diffcircles_8
edgecross20-040 fin2bb
blend718
graphpart_3pm-0244-0244 kall_diffcircles_9
edgecross22-048 flay05h
blend852
graphpart_3pm-0333-0333 launch
emfl050_5_5
fo7
carton7graphpart_3pm-0334-0334 lop97icx
emfl100_5_5
fo8
casctanks
ethanolh
fo8_ar25_1 graphpart_3pm-0344-0344 mathopt5_7
cecil_13
graphpart_3pm-0444-0444 mathopt5_8
ethanolm
fo8_ar2_1
chemgraphpart_clique-30
ex1252afo9
mhw4d
clay0203hex14_1_1
fo9_ar25_1 graphpart_clique-40
milinfract
clay0204h
gsg_0001
ex14_1_7
fo9_ar2_1
minlphix
clay0205hhda
ex4_1_5
fo9_ar3_1
minsurf100
clay0303hheatexch_trigen
ex4_1_6
fo9_ar4_1...
25 / 64

P.I.T.T.E.D. Heuristic: P.I.T.T. with Eased Dominance
3. Ensure uniqueness of 6-characters-prefix of instances names.

26 / 64

P.I.T.T.E.D. Heuristic: P.I.T.T. with Eased Dominance
3. Ensure uniqueness of 6-characters-prefix of instances names.

alkylation
fo9_ar5_1
clay0304hex6_1_1
house
arki0003
gasnetclay0305hex6_1_3
jbearing25
genpooling_meyer04
arki0005
crudeoil_lee2_10 ex6_2_12
jbearing75
ghg_1veharki0006
crudeoil_lee3_07 ex6_2_14
johnall
ghg_3veharki0019
crudeoil_lee3_08 ex6_2_8
kall_circles_c6aglider100
arki0024
crudeoil_lee3_09 ex6_2_9
kall_circles_c6bgraphpart_2g-0066-0066 kall_circles_c7a
autocorr_bern20-10 crudeoil_lee3_10 ex7_2_4
autocorr_bern20-15 crudeoil_li06
graphpart_2g-0077-0077
ex8_1_7
kall_circles_c8a
autocorr_bern25-06 csched1agraphpart_2g-0088-0088 kall_circlespolygons_c1pex8_2_1b
autocorr_bern25-13 edgecross10-060 ex8_2_4bgraphpart_2g-0099-9211 kall_circlespolygons_c1p
autocorr_bern30-04 edgecross10-070 ex8_4_1
graphpart_2pm-0066-0066 kall_circlesrectangles_c
autocorr_bern35-04 edgecross10-080 ex8_4_3
graphpart_2pm-0077-0777 kall_circlesrectangles_cbatch0812_nc
graphpart_2pm-0088-0888 kall_congruentcircles_c7
edgecross14-039 ex8_4_4
batchs201210m
graphpart_2pm-0099-0999 kall_diffcircles_10
edgecross14-058 ex8_4_5
bayes2_50
edgecross14-078 ex8_4_8_bnd graphpart_3g-0334-0334 kall_diffcircles_5b
blend480
graphpart_3g-0344-0344 kall_diffcircles_7
edgecross14-176 filter
blend531
graphpart_3g-0444-0444 kall_diffcircles_8
edgecross20-040 fin2bb
blend718
graphpart_3pm-0244-0244 kall_diffcircles_9
edgecross22-048 flay05h
blend852
graphpart_3pm-0333-0333 launch
fo7
emfl050_5_5
carton7graphpart_3pm-0334-0334 lop97icx
fo8
emfl100_5_5
casctanks
ethanolh
fo8_ar25_1 graphpart_3pm-0344-0344 mathopt5_7
cecil_13
graphpart_3pm-0444-0444 mathopt5_8
ethanolm
fo8_ar2_1
chemgraphpart_clique-30
ex1252afo9
mhw4d
clay0203hex14_1_1
fo9_ar25_1 graphpart_clique-40
milinfract
clay0204h
gsg_0001
ex14_1_7
fo9_ar2_1
minlphix
clay0205hhda
ex4_1_5
fo9_ar3_1
minsurf100
clay0303hheatexch_trigen
ex4_1_6
fo9_ar4_1...
26 / 64

P.I.T.T.E.D. SCIP testset
In summary:
1. Keep only instances that are marked as solved in MINLPLib 2.
2. Keep only instances that take ≥ 60s with oldest version of solver and that canbe handled by solver.
3. Reduce instances with similar names.
For SCIP, this reduces from 1363 to 881 to 123 instances:
alkylation
emfl050_5_5
arki0003
emfl100_5_5
autocorr_bern20-10 ethanolh
batch0812_nc
ex1252abatchs201210m
ex14_1_1
bayes2_50
ex4_1_5
blend480ex6_1_1
blend531
ex6_2_12
blend718
ex7_2_4
blend852
ex8_1_7
carton7ex8_2_1b
casctanks
ex8_4_1
cecil_13
filterchem
fin2bbclay0203h
flay05h
clay0303hfo7
crudeoil_lee2_10 fo8
csched1afo8_ar25_1
edgecross10-060
fo9

milinfract
fo9_ar25_1
minlphix
gasnetgenpooling_meyer04 minsurf100
multiplants_mtg1aghg_1veh
no7_ar3_1
ghg_3veh
nous1
glider100
nvs09
graphpart_2g-0066-0066
nvs22
gsg_0001
o7
hda
o7_2
heatexch_trigen
o7_ar25_1
house
o7_ar3_1
jbearing25
o7_ar4_1
johnallkall_circles_c6a o7_ar5_1
kall_diffcircles_10o8_ar4_1
o9_ar4_1
launchoillop97icx
oil2
mathopt5_7
parallelmhw4d

pinene50
pointpack08
sssd12-05
pooling_epa1
sssd15-04
prob07sssd16-07
process
sssd18-06
procsynsssd20-04
prologsssd25-04
qp3
st_e35routingdelay_bigm stockcycle
rsyn0805m02h
supplychaisepasequ_convent syn10m03h
sfacloc2_2_80
syn15m02h
slay07hsyn20m02h
slay09hsyn30hslay10hsyn30m02h
smallinvDAXr1b020-022
syn40hsporttournament14 syn40m02h
squfl010-025
sssd08-04
27 / 64

PITT test set for each solver

103
102
1

10

100
100

101
102
103
Number of Discrete Variables (+1)
LINDO

4

10

103
102
1

10

100
100

101
102
103
Number of Discrete Variables (+1)

BARON
10

4

103
102
10

1

100
100

101
102
103
Number of Discrete Variables (+1)

Number of Nonlinear Nonzeros in Jacobian

10

Number of Nonlinear Nonzeros in Jacobian

ANTIGONE4

Number of Nonlinear Nonzeros in Jacobian

Number of Nonlinear Nonzeros in Jacobian

Number of Nonlinear Nonzeros in Jacobian

Removed easy and unsolvable instances:
COUENNE
4

10

103
102
101
100
100

101
102
103
Number of Discrete Variables (+1)

SCIP
10

4

103
102
101
100
100

101
102
103
Number of Discrete Variables (+1)

28 / 64

PITTED test set for each solver

103
102
1

10

100
100

101
102
103
Number of Discrete Variables (+1)
LINDO

4

10

103
102
1

10

100
100

101
102
103
Number of Discrete Variables (+1)

BARON
10

4

103
102
10

1

100
100

101
102
103
Number of Discrete Variables (+1)

Number of Nonlinear Nonzeros in Jacobian

10

Number of Nonlinear Nonzeros in Jacobian

ANTIGONE4

Number of Nonlinear Nonzeros in Jacobian

Number of Nonlinear Nonzeros in Jacobian

Number of Nonlinear Nonzeros in Jacobian

Removed easy and unsolvable instances, then filter by name:
COUENNE
4

10

103
102
101
100
100

101
102
103
Number of Discrete Variables (+1)

SCIP
10

4

103
102
101
100
100

101
102
103
Number of Discrete Variables (+1)

29 / 64

Run jobs
date

GAMS

ANTIGONEBARONCOUENNE

LINDO

SCIP

08/11
04/12
11/12
02/13
07/13
05/14
09/14
06/15
07/15

23.7.323.8.223.9.524.0.224.1.324.2.324.3.324.4.6
24.5α

–
–
–
–
1.1
1.1
1.1
1.1
1.1

9.3.1
10.2.011.5.2
11.9.112.3.312.7.714.0.314.4.015.6.5

0.3
0.4
0.4
0.4
0.4
0.4
0.4
0.4
0.5

6.1.1.588
7.0.1.421
7.0.1.497
7.0.1.497
8.0.1283.385
8.0.1694.498
8.0.1694.550
9.0.1983.157
9.0.1983.157

–
2.1.1
2.1.2
3.0
3.0
3.0
3.1
3.1
3.2.0

for GAMS in $GAMSS ; dofor SOLVER in $SOLVERS($GAMS) ; dofor INSTANCE in $TESTSET($SOLVER) ; dosbatch --exclusive --time=0:1800 $GAMS $INSTANCE SOLVER=$SOLVER
done
done
done
Hardware: Dell PowerEdge M1000e, 48GB RAM, Intel Xeon X5672@3.2GHz
30 / 64

BARON: Fails% failed instances (out of 188)

A solver failed, if it
I crashed, orI reported an infeasible point as feasible (tolerance: 10−4 ), orI reported a suboptimal solution as optimal (tolerance: 10−4 )100same as previous version
additional to previous version
80
60
40
20
4.3 4.8 3.2 4.3 5.3
01

1.6 1.1

1

9.3 10.2 11.5 11.9 12.3 12.7 14.0 14.4 15.6
BARON version

31 / 64

BARON: Solved
Solved: solver did not fail and reports a relative optimality gap ≤ 10−4
same as previous version
additional to previous version
only in previous version% solved instances (out of 188)

100
79.2

80
60.1 62.8

60

83.5 84.6

67.1 68.6

45.2 47.940
200

9.3 10.2 11.5 11.9 12.3 12.7 14.0 14.4 15.6
BARON version
32 / 64

BARON: Solved – What happened?
% solved instances (out of 188)

100
79.2

80
60.1 62.8

60

83.5 84.6

67.1 68.6

45.2 47.940
200

9.3 10.2 11.5 11.9 12.3 12.7 14.0 14.4 15.6
BARON version

From the release notes:
11.0: “This version comes with a wealth of new branching, relaxation, convexity
exploitation, local search, and range reduction techniques.”
11.5: “Improvements in local search” (dive-and-round heuristic for MINLPs,
automatically select and switch back and forth between NLP solvers)
12.3: “New relaxations for certain types of quadratic problems”, “Improved integer
presolve”, “Incorporation of convex envelopes for certain low-dimensional
functions”
14.0: “Significant advances in the handling of integer programs.” (integer cutting
planes, calls to MIP solvers, hybrid LP/MIP/NLP relaxations)
33 / 64

BARON: Found optimal solution% found optimal solution (out of 188)

same as previous version
additional to previous version
only in previous version
100
87.8
80
66 63.8

68.6 67 71.3

93.1 93.1

76

60
40
20
0

9.3 10.2 11.5 11.9 12.3 12.7 14.0 14.4 15.6
BARON version
34 / 64

BARON: Solving time on instances that never failed (163)
wilcoxon p

nanmean time, speedup
25
mean solving time (s)

100
20
80

1.56
15

60

1.43
10
5
0

1.48

0.74
40

1.18
2.02
2.12 0.77

9.3 10.2 11.5 11.9 12.3 12.7 14.0 14.4 15.6

200

BARON version

Overall speedup: 9.00
12.7: “Automatic setting of many options based on problem characteristics and learningalgorithms.“
35 / 64

BARON: Solving time on instances solved by all vers. (69)wilcoxon p
100

0.8
1.46

75
1.06 1.00
1.13

0.6

1.20

50
1.25

0.4

1.58 0.88
0.2

20

0

10
5
0

9.3 10.2 11.5 11.9 12.3 12.7 14.0 14.4 15.6

100 × Wilcoxon p-valuemean solving time (s)nanmean time, speedup

BARON version
Overall speedup: 3.67
36 / 64ANTIGONE: Fails% failed instances (out of 189)100
same as previous version
additional to previous version

80
60
40
20
0

2.6

2.6

2.6

1.1

1.1

1.1

6.3

6.3

1.1

1.1

ANTIGONE version

37 / 64ANTIGONE: Solved% solved instances (out of 189)100
same as previous version
additional to previous version
only in previous version

80
61.9

62.4

62.4

1.1

1.1

1.1

60

59.3

59.3

1.1

1.1

40
20
0

ANTIGONE version

38 / 64ANTIGONE: Found optimal solution% found optimal solution (out of 189)

same as previous version
additional to previous version
only in previous version
100
83.6

85.2

84.6

1.1

1.1

1.1

80

80.4

80.4

1.1

1.1

60
40
20
0

ANTIGONE version
39 / 64ANTIGONE: Solving time on instances that never failed (177)
wilcoxon p

nanmean time, speedup

mean solving time (s)

12

1.18

1.00

1.05

1.00

100

10
80
8
60
6
40

4

20

2
0

1.1

1.1

1.1

1.1

1.1

0

ANTIGONE version
40 / 64ANTIGONE: Solving time on instances solved by all (110)wilcoxon p

nanmean time, speedup

mean solving time (s)

1.03

0.99

1.00
75

1.5
50
1
0.5

20

0

10
5
0

1.1

1.1

1.1

1.1

1.1

100 × Wilcoxon p-value

1.19
2

100

ANTIGONE version
41 / 64COUENNE: Fails% failed instances (out of 124)

100
same as previous version
additional to previous version
only in previous version

80
60
37.1

40
20
0

16.9

20.9 22.6 20.1 20.1 19.3

17.7

14.6

0.3 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.5
COUENNE version

42 / 64COUENNE: Solved% solved instances (out of 124)

100
same as previous version
additional to previous version
only in previous version

80
60
40

34.7
29
21.8

27.5 25

25 25.8 26.6 26.6

200

0.3 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.5
COUENNE version

43 / 64COUENNE: Found optimal solution% found optimal solution (out of 124)

same as previous version
additional to previous version
only in previous version
100
80
60
46
40

45.2 43.5 42.7
35.5

40.3 41.9 41.9

43.6

20
0

0.3 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.5
COUENNE version
44 / 64COUENNE: Solving time on instances that never failed (65)800

1.25 0.98 1.00
mean solving time (s)wilcoxon p

nanmean time, speedup
0.94 1.03

100

1.02 1.00
1.11

600

80
60

400

40
200
20
0

0.3 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.5

0

COUENNE version
Overall speedup: 1.35
45 / 64

Couenne] Couenne stable release 0.4

http://list.coin-or.org/pipermail/couenne/2011-A...

COUENNE: Solving time on instances that never failed (65)
Pietro Belotti pbelott at clemson.edu
Mon Aug 8 05:13:15 EDT 2011
Next message: [Couenne] stable/0.4 does not compile
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]800

wilcoxon p

nanmean time, speedup
1.25 0.98 1.00

mean solving time (s)[Couenne] Couenne stable release
0.4

0.94 1.03

100

1.02 1.00
1.11

600

80
60

400

40
200
20

Dear Couenne users,
this is to announce the 0.4 stable version of Couenne. There are a numberof additions and improvements, including:1) a Feasibility Pump heuristic for non-convex MINLP, developed with Timo
Berthold at the ZIB institute.

0

0.3 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.5

0

COUENNE version

1. Feasibility Pump

2) Orbital Branching for MINLP, developed with Jim Ostrowski and Leo
Liberti.
3) Fixed Point Bound tightening, a bound reduction procedure developedwith Sonia Cafieri, Jon Lee, and Leo Liberti.

2. Orbital Branching4) "semi-auxiliaries", i.e., auxiliary variables defined as y >= f(x) or y<= f(x) instead of just y = f(x). The purpose is to save on the number ofauxiliaries generated and hence on the size of the LP relaxation.
5) "Two-Implied bound tightening", a new bound reduction proceduredescribed in http://www.optimization-online.org/DB_FILE/2011/02/2931.pdf

3. Fixed Point BT6) various bug fixes.
Release 0.4.0 is a snapshot of the new stable version. The new features
will soon be documented in Couenne's user manual, available at
http://www.coin-or.org/Couenne/couenne-user-manual.pdf
Happy MINLPing,
Pietro-Pietro Belotti
Dept. of Mathematical Sciences
Clemson University
email: pbelott at clemson.edu
phone: 864-656-6765
web:myweb.clemson.edu/~pbelott

4. “semi-auxiliaries”
5. Two-Implied BT
6. various bug fixes

46 / 64

Couenne] Couenne stable release 0.4

http://list.coin-or.org/pipermail/couenne/2011-A...

COUENNE: Solving time on instances that never failed (65)
Pietro Belotti pbelott at clemson.edu
Mon Aug 8 05:13:15 EDT 2011
Next message: [Couenne] stable/0.4 does not compile
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]800

wilcoxon p

nanmean time, speedup
1.25 0.98 1.00

mean solving time (s)[Couenne] Couenne stable release
0.4

0.94 1.03

100

1.02 1.00
1.11

600

80
60

400

40
200
20

Dear Couenne users,
this is to announce the 0.4 stable version of Couenne. There are a numberof additions and improvements, including:1) a Feasibility Pump heuristic for non-convex MINLP, developed with Timo
Berthold at the ZIB institute.
2) Orbital Branching for MINLP, developed with Jim Ostrowski and Leo
Liberti.
3) Fixed Point Bound tightening, a bound reduction procedure developedwith Sonia Cafieri, Jon Lee, and Leo Liberti.
4) "semi-auxiliaries", i.e., auxiliary variables defined as y >= f(x) or y<= f(x) instead of just y = f(x). The purpose is to save on the number ofauxiliaries generated and hence on the size of the LP relaxation.
5) "Two-Implied bound tightening", a new bound reduction proceduredescribed in http://www.optimization-online.org/DB_FILE/2011/02/2931.pdf6) various bug fixes.
Release 0.4.0 is a snapshot of the new stable version. The new features
will soon be documented in Couenne's user manual, available at
http://www.coin-or.org/Couenne/couenne-user-manual.pdf
Happy MINLPing,
Pietro-Pietro Belotti
Dept. of Mathematical Sciences
Clemson University
email: pbelott at clemson.edu
phone: 864-656-6765
web:myweb.clemson.edu/~pbelott0

0.3 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.5

0

COUENNE version

1. Feasibility Pump
feasibility_pump no
2. Orbital Branchingorbital_branching no
3. Fixed Point BT
fixpoint_bt 0
4. “semi-auxiliaries”
use_semiaux yes
5. Two-Implied BT
two_implied_bt 0
6. various bug fixes

46 / 64COUENNE: Solving time on instances solved by all (16)nanmean time, speedup

wilcoxon p
100

75

100
0.87
80
2.10

0.91 1.00

1.14

1.09 1.02 1.04

50

60
40
20

20
0

0.3 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.5

100 × Wilcoxon p-valuemean solving time (s)

120

10
5
0

COUENNE version
Overall speedup: 2.16
47 / 64

LINDO: Fails% failed instances (out of 128)

100
same as previous version
additional to previous version
only in previous version

80
60
40

24.2 23.4
20

14.8
5.5

0

6.2 8.6 8.6

12.4 12.5

6.1 7.0 7.0 7.0 8.0 8.0 8.0 9.0 9.0
LINDO version

48 / 64

LINDO: Solved% solved instances (out of 128)

100
same as previous version
additional to previous version
only in previous version

80
60

48.4 48.5 46.9
40

39.1

43

43

35.2 34.4 32.8

200

6.1 7.0 7.0 7.0 8.0 8.0 8.0 9.0 9.0
LINDO version

LINDO 8.0: improvements in primal heuristics for MIP (feas. pump) andnonconvex NLP (multistart)
49 / 64

LINDO: Found optimal solution% found optimal solution (out of 128)

same as previous version
additional to previous version
only in previous version
100
80
60

64.8 63.3 66.4 62.5 62.5

59.4
53.1

47.6 46.1

40
200

6.1 7.0 7.0 7.0 8.0 8.0 8.0 9.0 9.0
LINDO version
50 / 64

LINDO: Solving time on instances that never failed (72)wilcoxon p

nanmean time, speedup

mean solving time (s)

100
600
80
400

2.01

60

1.10 1.04

40
200

0

1.85

1.32 1.06 0.93 1.00

6.1 7.0 7.0 7.0 8.0 8.0 8.0 9.0 9.0

20
0

LINDO version
Overall speedup: 5.48
51 / 64

LINDO: Solving time on instances solved by all vers. (16)wilcoxon p

nanmean time, speedup

mean solving time (s)

75

150
1.57

1.06 0.98
50

100
0.80
2.37

50

0

0.91 1.07 1.00
20

6.1 7.0 7.0 7.0 8.0 8.0 8.0 9.0 9.0

100 × Wilcoxon p-value

100

200

10
5
0

LINDO version
Overall speedup: 3.02
52 / 64SCIP: Fails% failed instances (out of 123)

100
same as previous version
additional to previous version
only in previous version

80
60
40
20
0

13

2.1

9
2.1

10.5 8.9
3.0

3.0

8.1
3.0

1.6

1.6

1.6

3.1

3.1

3.2

SCIP version

53 / 64SCIP: Solved% solved instances (out of 123)

100
same as previous version
additional to previous version
only in previous version

80
60

55.3

52 54.5

3.1

3.1

48 45.5
44.7
40

32.5 34.2

20
0

2.1

2.1

3.0

3.0

3.0

3.2

SCIP version

54 / 64SCIP: Found optimal solution% found optimal solution (out of 123)

same as previous version
additional to previous version
only in previous version
100
80
60

61.8 60.2

65.1

66.6 68.3

3.0

3.0

75.6 73.2 77.2

40
20
0

2.1

2.1

3.0

3.1

3.1

3.2

SCIP version
55 / 64SCIP: Solving time on instances that never failed (96)time, speedup
nanmean
1.03

wilcoxon p

mean solving time (s)

600

100
80

400
60
2.53 0.97 1.01
200

1.67 0.90 1.17

40
20

0

2.1

2.1

3.0

3.0

3.0

3.1

3.1

3.2

0

SCIP version
Overall speedup: 4.49
56 / 64SCIP: Solving time on instances solved by all vers. (31)wilcoxon p
100

75
150

100
2.78

0.87 0.99 1.07 0.92

0.93

50

50

20

0

10
5
0

2.1

2.1

3.0

3.0

3.0

3.1

3.1

3.2

100 × Wilcoxon p-valuemean solving time (s)200nanmean time, speedup
1.10

SCIP version
Overall speedup: 2.42
57 / 64

“Virtual Best” Solver
common subset of instances
I for each instance and GAMS version, pick best results among all solvers

2

10

101
100
100

101
102
103
Number of Discrete Variables (+1)
LINDO

4

10

103
102
1

10

100
100

101
102
103
Number of Discrete Variables (+1)

10

103
10

2

101
100
100

101
102
103
Number of Discrete Variables (+1)
SCIP

10

4

103
102
10

1

100
100

101
102
103
Number of Discrete Variables (+1)

Number of Nonlinear Nonzeros in Jacobian

103

BARON
4

Number of Nonlinear Nonzeros in Jacobian

10

Number of Nonlinear Nonzeros in Jacobian

ANTIGONE4

Number of Nonlinear Nonzeros in Jacobian

Number of Nonlinear Nonzeros in Jacobian

Number of Nonlinear Nonzeros in Jacobian

I

COUENNE
4

10

103
102
101
100
100

101
102
103
Number of Discrete Variables (+1)
Virtual Best

10

4

103
102
101
100
100

101
102
103
Number of Discrete Variables (+1)

58 / 64

Virtual Best: Fails% failed instances (out of 71)

20
same as previous version
additional to previous version
15

10

5
1.4
0

0
0
0
0
0
0
0
0
23.7 23.8 23.9 24.0 24.1 24.2 24.3 24.4 24.5
GAMS version

59 / 64

Virtual Best: Solved
virt.best

ANTIGONEBARONCOUENNE

23.8

24.0 24.1 24.2
GAMS version

LINDO APISCIP

% solved instances (out of 71)

100

80

60

40

20

0
23.7

23.9

24.3

24.4

24.5
60 / 64

Virtual Best: Solving time on instances that never failed (70)mean time, speedup

wilcoxon p

1,200
mean solving time (s)

100
1,000
80
800

nan
60

600
40

400
3.10
200
0

0.87
3.33 1.00 0.73 2.22

20
1.32 0.85

23.7 23.8 23.9 24.0 24.1 24.2 24.3 24.4 24.5

0GAMS version
Overall speedup: 14.84
61 / 64

Virtual Best: Solving time on instances that never failed (70)ANTIGONEvirt.best

mean solving time (s)

1,200

BARON

COUENNEwilcoxon p virt. best

LINDO

SCIP

100

1,000
80
800

nan
60

600
40

400
3.10
200
0

0.87
3.33 1.00 0.73 2.22

20
1.32 0.85

23.7 23.8 23.9 24.0 24.1 24.2 24.3 24.4 24.5

0GAMS version
Overall speedup: 14.84
62 / 64

Virtual Best: Solving time on instances solved by all (17)nanmean time, speedup

wilcoxon p
100

mean solving time (s)

200

80
150
60
100
40

0.76
4.30
50

0

2.33 0.94 1.12

1.56 1.19 0.96

23.7 23.8 23.9 24.0 24.1 24.2 24.3 24.4 24.5

200GAMS version
Overall speedup: 14.30
63 / 64End.
http://www.gamsworld.org/minlp/minlplib2/html/
Future Work:I

add more NLPs (from PrincetonLib, COCONUT, NEOS, ...)

I

semi-automatic identification of duplicatesI

more structure recognition, e.g., second-order conesI

define interesting subsets, especially a benchmark set for
global solvers

Call for contributions:
I

Contribute your own (MI)NLP instances!

I

Ideally from a model for a real life problem.

I

Also infeasible instances are welcomed.

I

Any (well-known) format is good (e.g., AMPL, GAMS, ZIMPL, BARON, CPLEX
LP, MPS, PIP, OSiL).

IMINLPLib instances are anonymized (scalar format using generic names).

I

Your benefit: Solver developers may test and tune their solver for yourproblem.

(Or send your model to minlp.org!)

64 / 64