Package: dbmsbenchmarker
DBMS-BenchmarkerDBMS-Benchmarker is a Python-based application-level blackbox benchmark tool for Database Management Systems (DBMS).
It aims at reproducible measuring and easy evaluation of the performance the user receives even in complex benchmark situations.
It connects to a given list of DBMS (via JDBC) and runs a given list of (SQL) benchmark queries.
Queries can be parametrized and randomized.
Results and evaluations are available via a Python interface and can be inspected with standard Python tools like pandas DataFrames.
An interactive visual dashboard assists in multi-dimensional analysis of the results.See thehomepageand thedocumentation.If you encounter any issues, please report them to ourGithub issue tracker.Key FeaturesDBMS-Benchmarkeris Python3-basedhelps tobenchmark DBMSconnects to all DBMS having a JDBC interface - including GPU-enhanced DBMSrequiresonlyJDBC - no vendor specific supplements are usedbenchmarks arbitrary SQL queries - in all dialectsallows planning of complex test scenarios - to simulate realistic or revealing use casesallows easy repetition of benchmarks in varying settings - different hardware, DBMS, DBMS configurations, DB settings etcinvestigates a number of timing aspects - connection, execution, data transfer, in total, per session etcinvestigates a number of other aspects - received result sets, precision, number of clientscollects hardware metrics from a Prometheus server - hardware utilization, energy consumption etchelps toevaluate results- by providingmetrics that can be analyzed by aggregation in multi-dimensions, like maximum throughput per DBMS, average CPU utilization per query or geometric mean of run latency per workloadpredefined evaluations like statisticsin standard Python data structuresinJupyter notebooksseerendered examplein an interactive dashboardFor more informations, see abasic exampleor take a look in thedocumentationfor a full list of options.The code uses several Python modules, in particularjaydebeapifor handling DBMS.
This module has been tested with Citus Data (Hyperscale), Clickhouse, CockroachDB, Exasol, IBM DB2, MariaDB, MariaDB Columnstore, MemSQL (SingleStore), MonetDB, MySQL, OmniSci (HEAVY.AI), Oracle DB, PostgreSQL, SQL Server, SAP HANA, TimescaleDB, and Vertica.InstallationRunpip install dbmsbenchmarkerto install the package.You will also need to haveJavainstalled(we tested with Java 8)JAVA_HOMEset correctlya JDBC driver suitable for the DBMS you want to connect to (optionally located in yourCLASSPATH)Basic UsageThe following very simple use case runs the querySELECT COUNT(*) FROM test10 times against one local MySQL installation.
As a result we obtain an interactive dashboard to inspect timing aspects.ConfigurationWe need to provideaDBMS configuration file, e.g. in./config/connections.config[
  {
    'name': "MySQL",
    'active': True,
    'JDBC': {
      'driver': "com.mysql.cj.jdbc.Driver",
      'url': "jdbc:mysql://localhost:3306/database",
      'auth': ["username", "password"],
      'jar': "mysql-connector-java-8.0.13.jar"
    }
  }
]the required JDBC driver, e.g.mysql-connector-java-8.0.13.jaraQueries configuration file, e.g. in./config/queries.config{
  'name': 'Some simple queries',
  'connectionmanagement': {
        'timeout': 5 # in seconds
    },
  'queries':
  [
    {
      'title': "Count all rows in test",
      'query': "SELECT COUNT(*) FROM test",
      'numRun': 10
    }
  ]
}Perform BenchmarkRun the CLI command:dbmsbenchmarker run -e yes -b -f ./config-e yes: This will precompile some evaluations and generate the timer cube.-b: This will suppress some output-f: This points to a folder having the configuration files.This is equivalent topython benchmark.py run -e yes -b -f ./configAfter benchmarking has been finished we will see a message likeExperiment <code> has been finishedThe script has created a result folder in the current directory containing the results.<code>is the name of the folder.Evaluate Results in DashboardRun the command:dbmsdashboardThis will start the evaluation dashboard atlocalhost:8050.
Visit the address in a browser and select the experiment<code>.Alternatively you may use aJupyter notebook, see arendered example.LimitationsLimitations are:strict black box perspective - may not use all tricks available for a DBMSstrict JDBC perspective - depends on a JVM and provided driversstrict user perspective - client system, network connection and other host workloads may affect performancenot officially applicable for well known benchmark standards - partially, but not fully complying with TPC-H and TPC-DShardware metrics are collected from a monitoring system - not as precise as profilingno GUI for configurationstrictly Python - a very good and widely used language, but maybe not your choiceOther comparable products you might likeApache JMeter- Java-based performance measure tool, including a configuration GUI and reporting to HTMLHammerDB- industry accepted benchmark tool based on Tcl, but limited to some DBMSSysbench- a scriptable multi-threaded benchmark tool based on LuaJITOLTPBench- Java-based performance measure tool, using JDBC and including a lot of predefined benchmarksBenchBase- successor of OLTPBenchContributing, Bug ReportsIf you have any question or found a bug, please report them to ourGithub issue tracker.
In any bug report, please let us know:Which operating system and hardware (32 bit or 64 bit) you are usingPython versionDBMSBenchmarker version (or git commit/date)DBMS you are connecting toTraceback that occurs (the full error message)We are always looking for people interested in helping with code development, documentation writing, technical administration, and whatever else comes up.
If you wish to contribute, please first read thecontribution sectionor visit thedocumentation.Benchmarking in a Kubernetes CloudThis module can serve as thequery executor[2] andevaluator[1] for distributed parallel benchmarking experiments in a Kubernetes Cloud, see theorchestratorfor more details.ReferencesIf you use DBMSBenchmarker in work contributing to a scientific publication, we kindly ask that you cite our application note [1] and/or [3]:[1]A Framework for Supporting Repetition and Evaluation in the Process of Cloud-Based DBMS Performance BenchmarkingErdelt P.K. (2021)
A Framework for Supporting Repetition and Evaluation in the Process of Cloud-Based DBMS Performance Benchmarking.
In: Nambiar R., Poess M. (eds) Performance Evaluation and Benchmarking. TPCTC 2020.
Lecture Notes in Computer Science, vol 12752. Springer, Cham.https://doi.org/10.1007/978-3-030-84924-5_6[2]Orchestrating DBMS Benchmarking in the Cloud with KubernetesErdelt P.K. (2022)
Orchestrating DBMS Benchmarking in the Cloud with Kubernetes.
In: Nambiar R., Poess M. (eds) Performance Evaluation and Benchmarking. TPCTC 2021.
Lecture Notes in Computer Science, vol 13169. Springer, Cham.https://doi.org/10.1007/978-3-030-94437-7_6[3]DBMS-Benchmarker: Benchmark and Evaluate DBMS in PythonErdelt P.K., Jestel J. (2022).
DBMS-Benchmarker: Benchmark and Evaluate DBMS in Python.
Journal of Open Source Software, 7(79), 4628https://doi.org/10.21105/joss.04628