Project: Vald (App Definition and Development / Database)
File: continuous-benchmark.md

Continuous Benchmark Tool allows you to continuously benchmark the Vald cluster. Assumed use case is: Verification with workload close to the production environment Verification before service installation when Vald version up Continuous Benchmark Tool has following 2 components: Benchmark Operator: Manages benchmark jobs Benchmark Job: Executes CRUD request to the target Vald cluster Manages benchmark jobs according to applied manifest. Apply method: Scenario method: one manifest with multiple benchmark jobs Job method: one manifest with one benchmark job Executes CRUD request to the target Vald cluster based on defined config. Execute steps are: Load dataset (valid only for HDF5 format) Execute request with load dataset Benchmark workload can be set by applying the Kubernetes Custom Resources(CRDs), `ValdBenchmarkScenarioResource` or `ValdBenchmarkJobResource`. Benchmark Operator manages benchmark job according to the applied manifest. is used for executing single benchmark job. And, Benchmark Operator also applies it to the Kubernetes cluster based on `ValdBenchmarkScenarioResource`. main properties | Name | mandatory | Description | type | sample | | :- | :-- | :-- | :-- | :- | | target | \* | target Vald cluster | object | ref: | | dataset | \* | dataset information | object | ref: | | job_type | \* | execute job type | string enum: [insert, update, upsert, remove, search, getobject, exists] | search | | repetition | | the number of job repetitions<BR>default: `1` | integer | 1 | | replica | | the number of job concurrent job executions<BR>default: `1` | integer | 2 | | rps | | designed request per sec to the target cluster<BR>default: `1000` | integer | 1000 | | concurrency_limit | | goroutine count limit for rps adjustment<BR>default: `200` | integer | 20 | | ttlsecondsafter_finished | | time until deletion of Pod after job end<BR>default: `600` | integer | 120 | | insert_config | | request config for insert job | object | ref: | | update_config | | request config for update job | object | ref: | | upsert_config | | request config for upsert job | object | ref: | | search_config | | request config for search job | object | ref: | | remove_config | | request config for remove job | object | ref: | | object_config | | request config for object job | object | ref: | | client_config | | gRPC client config for running benchmark job<BR>Tune if can not getting the expected performance with default config. | object | ref: | | server_config | | server config for benchmark job pod<BR>Tune if can not getting the expected performance with default

| object | ref: | <a id="target-prop" /> target target Vald cluster information type: object | property | mandatory | description | type | sample | | :- | :-- | :-- | : | :-- | | host | \* | target cluster's host | string | localhost | | port | \* | target cluster's port | integer | 8081 | <a id="dataset-prop" /> dataset dataset which is used for executing job operation type: object | property | mandatory | description | type | sample | | :- | :-- | : | :- | : | | name | \* | dataset name | string enum: [fashion-mnist, original] | fashion-mnist | | group | \* | group name | string enum: [train, test, neighbors] | train | | indexes | \* | amount of index size | integer | 1000000 | | range | \* | range of indexes to be used (if there are many indexes, the range will be corrected on the job side) | object | - | | range.start | \* | start of range | integer | 1 | | range.end | \* | end of range | integer | 1000000 | | url | | the dataset url. It should be set when set `name` as `original` | string | | <a id="insert-cfg-props" /> insert_config rpc config for insert request type: object | property | mandatory | description | type | sample | | :- | :-- | :-- | :-- | : | | skipstrictexist_check | | Check whether the same vector is already inserted or not.<br>The ID should be unique if the value is `true`. | bool | false | | timestamp | | The timestamp of the vector inserted.<br>If it is N/A, the current time will be used. | string | 1707272658 | <a id="update-cfg-props" /> update_config rpc config for update request type: object | property | mandatory | description | type | sample | | :- | :-- | :-- | :-- | : | | skipstrictexist_check | | Check whether the same vector is already inserted or not.<br>The ID should be unique if the value is `true`. | bool | false | | timestamp | | The timestamp of the vector inserted.<br>If it is N/A, the current time will be used. | string | 1707272658 | | disablebalancedupdate | | A flag to disable balanced update (split remove -&gt; insert operation) during update operation. | bool | false | <a id="upsert-cfg-props" /> upsert_config rpc config for upsert request type: object | property | mandatory | description | type | sample | | :- | :-- | :-- | :-- | : | | skipstrictexist_check | | Check whether the same vector is already inserted or not.<br>The ID should be unique if the value is `true`. | bool | false | | timestamp | | The timestamp of the vector inserted.<br>If it is N/A, the current time will be used. | string | 1707272658 | | disablebalancedupdate | | A flag to disable balanced update (split remove -&gt; insert operation) during update operation. | bool | false | <a id="search-cfg-props" /> upsert_config rpc config for search request type: object | property | mandatory | description | type | sample | | :-- | :-- | :- | : | :-- | | radius | | The search radius.<BR>default: `-1` | number | -1 | | epsilon | | The search coefficient.<BR>default: `0.05` | number | 0.05 | | num | \* | The maximum number of results to be returned. | integer | 10 | | min_num | | The minimum number of results to be returned. | integer | 5 | | timeout | | Search timeout in nanoseconds<BR>default: `10s` | string | 3s | | enablelinearsearch | | A flag to enable linear search operation for estimating search recall.<BR>If it is `true`, search operation with linear operation will execute. | bool | false | | aggregation_algorithm | | The search aggregation algorithm option.<BR>default: `Unknown` | string enum: ["Unknown", "ConcurrentQueue", "SortSlice", "SortPoolSlice", "PairingHeap"] | | <a id="remove-cfg-props" /> remove_config rpc config for remove request type: object | property | mandatory | description | type | sample | | :- | :-- | :-- | :-- | : | | skipstrictexist_check | | Check whether the same vector is already inserted or

ID should be unique if the value is `true`. | bool | false | | timestamp | | The timestamp of the vector inserted.<br>If it is N/A, the current time will be used. | string | 1707272658 | <a id="object-cfg-props" /> object_config rpc config for get object request type: object | property | mandatory | description | type | sample | | :-- | :-- | :- | :- | :-- | | filter_config.targets | | filter target host and port for bypassing filter component. | []object | | is used for executing single or multiple benchmark job. Benchmark Operator decomposes manifest and creates benchmark resources one by one. The `target` and `dataset` property are the global config for scenario, they can be overwritten when each job has own config. main properties | property | mandatory | description | type | sample | | :- | :-- | :- | :-- | :-- | | target | \* | target Vald cluster information<BR>It will be overwritten when each job has own config | object | ref: | | dataset | \* | dataset information<BR>It will be overwritten when each job has own config | object | ref: | | jobs | \* | benchmark job config<BR>The jobs written above will be executed in order. | object | ref: | Continuous benchmark operator can be applied with `Helm` same as Vald cluster. It requires `ValdBenchmarkOperatorRelease` for deploying `vald-benchmark-operator`. It is not must to apply, so please edit and apply as necessary. <div class="notice"> `job.client_config` is used as the default gRPC client configuration for all benchmark jobs.<BR> Although it is possible to override using `ValdBenchmarkScenarioRelease` or `ValdBenchmarkJobRelease`, we recommend configuring common setting items here. </div> <details><summary>Sample ValdBenchmarkOperatorRelease YAML</summary> ```yaml name: vald-benchmark-operator time_zone: "" image: repository: vdaas/vald-benchmark-operator tag: v1.7.5 pullPolicy: Always job: image: repository: vdaas/vald-benchmark-job tag: v1.7.12 pullPolicy: Always client_config: addrs: [] healthcheckduration: "1s" connection_pool: enablednsresolver: true enable_rebalance: true rebalance_duration: 30m size: 3 oldconnclose_duration: "2m" backoff: initial_duration: 5ms backofftimelimit: 5s maximum_duration: 5s jitter_limit: 100ms backoff_factor: 1.1 retry_count: 100 enableerrorlog: true circuit_breaker: closederrorrate: 0.7 halfopenerror_rate: 0.5 min_samples: 1000 open_timeout: "1s" closedrefreshtimeout: "10s" call_option: waitforready: true maxretryrpcbuffersize: 0 maxrecvmsg_size: 0 maxsendmsg_size: 0 dial_option: writebuffersize: 0 readbuffersize: 0 initialwindowsize: 0 initialconnectionwindow_size: 0 maxmsgsize: 0 backoffmaxdelay: "120s" backoffbasedelay: "1s" backoff_multiplier: 1.6 backoff_jitter: 0.2 minconnectiontimeout: "20s" enable_backoff: false insecure: true timeout: "" interceptors: [] net: dns: cache_enabled: true refresh_duration: 30m cache_expiration: 1h dialer: timeout: "" keepalive: "" dualstackenabled: true socket_option: reuse_port: true reuse_addr: true tcpfastopen: true tcpnodelay: true tcp_cork: false tcpquickack: true tcpdeferaccept: true ip_transparent: false iprecoverdestination_addr: false keepalive: time: "120s" timeout: "30s" permitwithoutstream: true resources: limits: cpu: 300m memory: 300Mi requests: cpu: 200m memory: 200Mi logging: logger: glg level: debug format: raw ``` </details> For more details of the configuration of `vald-benchmark-operator-release`, please refer to Add Vald repo into the helm repo ```bash helm repo add vald https://vdaas.vald.org ``` Deploy `vald-benchmark-operator-release` ```bash helm install vald-benchmark-operator-release vald/vald-benchmark-operator ``` Apply `vbor.yaml` (optional) ```bash kubectl apply -f vbor.yaml ``` After deploy the benchmark operator, you can execute continuous benchmark by applying `ValdBenchmarkScenarioRelease` or `ValdBenchmarkJobRelease`. Please configure designed benchmark and apply by `kubectl` command. The sample manifests are . Metrics monitoring can be set in the same way as Vald cluster. For information on building a monitoring environment, please refer to . To monitor metrics about continuous benchmarking, please edit `ValdBenchmarkOperatorRelease` as follows: ```yaml observability: enabled: true otlp: collector_endpoint: "opentelemetry-collector-collector.default.svc.cluster.local:4317" trace: enabled: true ``` After apply it, the metrics can be shown on the Grafana dashboard like as below. <img src="../../assets/docs/performance/benchmark-grafana.png" alt="Grafana dashboard showing benchmark metrics" />