Package: qpsolvers_benchmark
QP solvers benchmarkBenchmark for quadratic programming (QP) solvers available in Python.The goal of this benchmark is to help users compare and select QP solvers. Its methodology is open todiscussions. The benchmark ships standard and communitytest sets, as well as aqpsolvers_benchmarkcommand-line tool to run test sets directly. The main output of the benchmark are standardized reports evaluating allmetricsacross all QP solvers available on the test machine. This repository also distributesresultsfrom running the benchmark on a reference computer.New test sets are welcome! The benchmark is designed so that each test-set comes in a standalone directory. Feel free to create a new one andcontribute ithere so that we grow the collection over time.Test setsThe benchmark comes with standard and community test sets to represent different use cases for QP solvers:Test setProblemsBrief descriptionMaros-Meszaros138Standard, designed to be difficult.Maros-Meszaros dense62Subset of Maros-Meszaros restricted to smaller dense problems.GitHub free-for-all12Community-built, new problemsare welcome!SolversSolverKeywordAlgorithmMatricesLicenseClarabelclarabelInterior pointSparseApache-2.0CVXOPTcvxoptInterior pointDenseGPL-3.0DAQPdaqpActive setDenseMITECOSecosInterior pointSparseGPL-3.0GurobigurobiInterior pointSparseCommercialHiGHShighsActive setSparseMITHPIPMhpipmInterior pointDenseBSD-2-ClauseMOSEKmosekInterior pointSparseCommercialNPPronpproActive setDenseCommercialOSQPosqpDouglas–RachfordSparseApache-2.0PIQPpiqpProximal Interior PointDense & SparseBSD-2-ClauseProxQPproxqpAugmented LagrangianDense & SparseBSD-2-ClauseqpOASESqpoasesActive setDenseLGPL-2.1qpSWIFTqpswiftInterior pointSparseGPL-3.0quadprogquadprogGoldfarb-IdnaniDenseGPL-2.0SCSscsDouglas–RachfordSparseMITMetricsWe evaluate QP solvers based on the following metrics:Success rate:percentage of problems a solver is able to solve on a given test set.Computation time:time a solver takes to solve a given problem.Optimality conditions:we evaluate all threeoptimality conditions:Primal residual:maximum error on equality and inequality constraints at the returned solution.Dual residual:maximum error on the dual feasibility condition at the returned solution.Duality gap:value of the duality gap at the returned solution.Cost error:difference between the solution cost and the known optimal cost.Shifted geometric meanEach metric (computation time, primal and dual residuals, duality gap) produces a different ranking of solvers for each problem. To aggregate those rankings into a single metric over the whole test set, we use theshifted geometric mean(shm), which is a standard to aggregate computation times inbenchmarks for optimization software. This mean has the advantage of being compromised by neither large outliers (as opposed to the arithmetic mean) nor by small outliers (in contrast to the geometric geometric mean). Check out thereferencesbelow for further details.Here are some intuitive interpretations:A solver with a shifted-geometric-mean runtime of $Y$ is $Y$ times slower than the best solver over the test set.A solver with a shifted-geometric-mean primal residual $R$ is $R$ times less accurate on equality and inequality constraints than the best solver over the test set.ResultsThe outcome from running a test set is a standardized report comparingsolversagainst the differentmetrics. Here are the results obtained on a reference computer:Test setResultsCPU infoGitHub free-for-allFull reportIntel(R) Core(TM) i7-6500U CPU @ 2.50GHzMaros-MeszarosFull reportIntel(R) Core(TM) i7-6500U CPU @ 2.50GHzMaros-Meszaros denseFull reportIntel(R) Core(TM) i7-6500U CPU @ 2.50GHzYou can check out results from a variety of machines, and share the reports produced by running the benchmark on your own machine, in theResults categoryof the discussions forum.LimitationsHere are some known areas of improvement for this benchmark:Cold start only:we don't evaluate warm-start performance for now.CPU thermal throttling:the benchmark currently does not check the status of CPU thermal throttling. Adding this feature is agood way to start contributingto the benchmark.Check out theissue trackerfor ongoing works and future improvements.InstallationYou can install the benchmark and its dependencies in an isolated environment usingconda:conda env create -f environment.yamlconda activate qpsolvers_benchmarkAlternatively, you can install the benchmark on your system usingpip:pip install qpsolvers_benchmarkBy default, the benchmark will run all supported solvers it finds.Running the benchmarkOnce the benchmark is installed, you will be able to run theqpsolvers_benchmarkcommand. Provide it with the script corresponding to thetest setyou want to run, followed by a benchmark command such as "run". For instance, let's run the "dense" subset of the Maros-Meszaros test set:qpsolvers_benchmark maros_meszaros/maros_meszaros_dense.py runYou can also run a specific solver, problem or set of solver settings:qpsolvers_benchmark maros_meszaros/maros_meszaros_dense.py run --solver proxqp --settings defaultCheck outqpsolvers_benchmark --helpfor a list of available commands and arguments.PlotsThe command line ships aplotcommand to compare solver performances over a test set for a specific metric. For instance, run:qpsolvers_benchmark maros_meszaros/maros_meszaros_dense.py plot runtime high_accuracyTo generate the following plot:ContributingContributions to improving this benchmark are welcome. You can for instance propose new problems, or share the runtimes you obtain on your machine. Check out thecontribution guidelinesfor details.CitationTo cite this benchmark in your scientific works, follow theCite this repositorybutton on GitHub (top-right of therepository page).See alsoReferencesHow not to lie with statistics: the correct way to summarize benchmark results: why geometric means should always be used to summarize normalized results.Optimality conditions and numerical tolerances in QP solvers: note written while figuring out thehigh_accuracysettings of this benchmark.Other benchmarksBenchmarks for optimization softwareby Hans Mittelmann, which includes reports on the Maros-Meszaros test set.jrl-qp/benchmarks: benchmark of QP solvers available in C++.osqp_benchmark: benchmark examples for the OSQP solver.proxqp_benchmark: benchmark examples for the ProxQP solver.