Package: benchmark-runner
Benchmark-Runner: Running benchmarksWhat is it?benchmark-runneris a containerized Python lightweight and flexible framework for running benchmark workloads
on Kubernetes/OpenShift runtype kinds Pod, kata and VM.This framework support the following embedded workloads:hammerdb: running hammerdb workload on the following databases: MSSQL, Mariadb, Postgresql in Pod, Kata or VM withConfigurationstressng: running stressng workload in Pod, Kata or VMConfigurationuperf: running uperf workload in Pod, Kata or VM withConfigurationvdbench: running vdbench workload in Pod, Kata or VM withConfigurationbootstorm: calculate VMs boot load timeConfiguration** For hammerdb mssql must run oncepermissionBenchmark-runner grafana dashboard example:Reference:The benchmark-runner package is located inPyPiThe benchmark-runner container image is located inQuay.ioDocumentationDocumentation is available atbenchmark-runner.readthedocs.ioTable of ContentsBenchmark-RunnerDocumentationRun workload using Podman or DockerRun workload in Pod using Kubernetes or OpenShiftGrafana dashboardsInspect Prometheus MetricsHow to develop in benchmark-runnerRun workload using Podman or DockerThe following options may be passed via command line flags or set in the environment:mandatory:KUBEADMIN_PASSWORD=$KUBEADMIN_PASSWORDmandatory:$KUBECONFIG [ kubeconfig file path]mandatory:WORKLOAD=$WORKLOADChoose one from the following list:['stressng_pod', 'stressng_vm', 'stressng_kata', 'uperf_pod', 'uperf_vm', 'uperf_kata', 'hammerdb_pod_mariadb', 'hammerdb_vm_mariadb', 'hammerdb_kata_mariadb', 'hammerdb_pod_mariadb_lso', 'hammerdb_vm_mariadb_lso', 'hammerdb_kata_mariadb_lso', 'hammerdb_pod_postgres', 'hammerdb_vm_postgres', 'hammerdb_kata_postgres', 'hammerdb_pod_postgres_lso', 'hammerdb_vm_postgres_lso', 'hammerdb_kata_postgres_lso', 'hammerdb_pod_mssql', 'hammerdb_vm_mssql', 'hammerdb_kata_mssql', 'hammerdb_pod_mssql_lso', 'hammerdb_vm_mssql_lso', 'hammerdb_kata_mssql_lso', 'vdbench_pod', 'vdbench_kata', 'vdbench_vm', 'clusterbuster', 'bootstorm_vm']** clusterbuster workloads: cpusoaker, files, fio, uperf. for more detailsseeNot mandatory:auto:NAMESPACE=benchmark-operator [ The default namespace is benchmark-operator ]auto:ODF_PVC=True [ True=ODF PVC storage, False=Ephemeral storage, default True ]auto:EXTRACT_PROMETHEUS_SNAPSHOT=True [ True=extract Prometheus snapshot into artifacts, false=don't, default True ]auto:SYSTEM_METRICS=False [ True=collect metric, False=not collect metrics, default False ]auto:RUNNER_PATH=/tmp [ The default work space is /tmp ]optional:PIN_NODE_BENCHMARK_OPERATOR=$PIN_NODE_BENCHMARK_OPERATOR [node selector for benchmark operator pod]optional:PIN_NODE1=$PIN_NODE1 [node1 selector for running the workload]optional:PIN_NODE2=$PIN_NODE2 [node2 selector for running the workload, i.e. uperf server and client, hammerdb database and workload]optional:ELASTICSEARCH=$ELASTICSEARCH [ elasticsearch service name]optional:ELASTICSEARCH_PORT=$ELASTICSEARCH_PORToptional:CLUSTER=$CLUSTER [ set CLUSTER='kubernetes' to run workload on a kubernetes cluster, default 'openshift' ]optional:scaleSCALE=$SCALE [For Vdbench/Bootstorm: Scale in each node]optional:scaleSCALE_NODES=$SCALE_NODES [For Vdbench/Bootstorm: Scale's node]optional:scaleREDIS=$REDIS [For Vdbench only: redis for scale synchronization]optional:LSO_DISK_ID=$LSO_DISK_ID [LSO_DISK_ID='scsi-<replace_this_with_your_actual_disk_id>' For using LSO Operator in hammerdb]optional:WORKER_DISK_IDS=$WORKER_DISK_IDS [WORKER_DISK_IDS For ODF/LSO workloads hammerdb/vdbench]For example:podmanrun--rm--workload=$WORKLOAD--kubeadmin-password=$KUBEADMIN_PASSWORD--pin-node-benchmark-operator=$PIN_NODE_BENCHMARK_OPERATOR--pin-node1=$PIN_NODE1--pin-node2=$PIN_NODE2--elasticsearch=$ELASTICSEARCH--elasticsearch-port=$ELASTICSEARCH_PORT-v$KUBECONFIG:/root/.kube/config--privilegedquay.io/ebattat/benchmark-runner:latestorpodmanrun--rm-eWORKLOAD=$WORKLOAD-eKUBEADMIN_PASSWORD=$KUBEADMIN_PASSWORD-ePIN_NODE_BENCHMARK_OPERATOR=$PIN_NODE_BENCHMARK_OPERATOR-ePIN_NODE1=$PIN_NODE1-ePIN_NODE2=$PIN_NODE2-eELASTICSEARCH=$ELASTICSEARCH-eELASTICSEARCH_PORT=$ELASTICSEARCH_PORT-elog_level=INFO-v$KUBECONFIG:/root/.kube/config--privilegedquay.io/ebattat/benchmark-runner:latestSAVE RUN ARTIFACTS LOCAL:add-e SAVE_ARTIFACTS_LOCAL='True'or--save-artifacts-local=trueadd-v /tmp:/tmpgit clone -b v1.0.2https://github.com/cloud-bulldozer/benchmark-operator/tmp/benchmark-operatorRun vdbench workload in Pod using OpenShiftRun vdbench workload in Pod using KubernetesRun workload in Pod using Kubernetes or OpenShift[TBD]Grafana dashboardsThere are 2 grafana dashboards templates:FuncCi dashboardPerfCi dashboard** PerfCi dashboard is generated automatically inBuild GitHub actionsfrommain.libsonnet** After importing json in grafana, you need to configure elasticsearch data source. (for more details: seeHOW_TO.md)Inspect Prometheus MetricsThe CI jobs store snapshots of the Prometheus database for each run as part of the artifacts.  Within the artifact directory is a Prometheus snapshot directory named:promdb-YYYY_MM_DDTHH_mm_ss+0000_YYYY_MM_DDTHH_mm_ss+0000.tarThe timestamps are for the start and end of the metrics capture; they
are stored in UTC time (+0000).  It is possible to run containerized
Prometheus on it to inspect the metrics.Note that Prometheus
requires write access to its database, so it will actually write to
the snapshot.So for example if you have downloaded artifacts for a
run namedhammerdb-vm-mariadb-2022-01-04-08-21-23and the Prometheus
snapshot within is namedpromdb_2022_01_04T08_21_52+0000_2022_01_04T08_45_47+0000, you could run as follows:$ local_prometheus_snapshot=/hammerdb-vm-mariadb-2022-01-04-08-21-23/promdb_2022_01_04T08_21_52+0000_2022_01_04T08_45_47+0000
$ chmod -R g-s,a+rw "$local_prometheus_snapshot"
$ sudo podman run --rm -p 9090:9090 -uroot -v "$local_prometheus_snapshot:/prometheus" --privileged prom/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=100000d --storage.tsdb.retention.size=1000PBand point your browser at port 9090 on your local system, you can run queries against it, e.g.sum(irate(node_cpu_seconds_total[2m])) by (mode,instance) > 0It is important to use the--storage.tsdb.retention.timeoption to
Prometheus, as otherwise Prometheus may discard the data in the
snapshot.  And note that you must set the time bounds on the
Prometheus query to fit the start and end times as recorded in the
name of the promdb snapshot.How to develop in benchmark-runnerseeHOW_TO.md