Package: annb
ANNB: Approximate Nearest Neighbor BenchmarkNote: This is a work in progress. The API/CLI is not stable yet.Installationpipinstallannb# install vector search index/client you may need for benchmark# e.g install faiss for run faiss index benchmarkUsageCLI UsageRun Benchmarkstart first benchmark with a randome dataset.Just runannb-testto start your first benchmark with a random dataset.annb-testIt will produce a result like this:❯ annb-test
... some logs ...

BenchmarkResult:
  attributes:
    query_args: [{'nprobe': 1}]
    topk: 10
    jobs: 1
    loop: 5
    step: 10
    name: Test
    dataset: .annb_random_d256_l2_1000.hdf5
    index: Test
    dim: 256
    metric_type: MetricType.L2
    index_args: {'index': 'ivfflat', 'nlist': 128}
    started: 2023-08-14 13:03:40

  durations:
    training: 1 items, 1000 total, 1490.03266ms
    insert: 1 items, 1000 total, 132.439627ms
    query:
      nprobe=1,recall=0.2173 -> 1000 items, 18.615083ms, 53719.878659686874qps, latency=0.18615083ms, p95=0.31939ms, p99=0.41488msThis is a simple benchmark test with default index(faiss) with random l2 dataset.
If you wants to generate more data or with some different specifications for the dataset, you could see below options:--index-dim         The dimension of the index, default is 256--index-metric-type   Index metric type, l2 or ip, default is l2--topk TOPK           topk used for query, default is 10--step STEP           the query step, default annb will query 10 items per query, you could set it to 0 for query all items in one query (similar like batch for ann-benchmarks)--batch               batch mode, alias --step 0--count COUNT         the total number of items in the dataset, default is 1000run benchmark with a specific datasetYou could also use ann-benchmarks'sdatasetto run benchmark. download them locally and run benchmark with--datasetoption.annb-test--datasetsift-128-euclidean.hdf5run benchmark with query argsYou mary benchmark with different query args, e.g. different nprobe for faiss ivfflat index. you could try--query-argsoption.annb-test--query-argsnprobe=10--query-argsnprobe=20will output below result:durations:
    training: 1 items, 1000 total, 1548.84968ms
    insert: 1 items, 1000 total, 143.402532ms
    query:
      nprobe=1,recall=0.2173 -> 1000 items, 20.074236ms, 49815.09632545916qps, latency=0.20074235999999998ms, p95=0.332276ms, p99=0.455525ms
      nprobe=10,recall=0.5221 -> 1000 items, 49.141931ms, 20349.2207092961qps, latency=0.49141931ms, p95=0.722628ms, p99=0.818012ms
      nprobe=20,recall=0.6861 -> 1000 items, 69.284072ms, 14433.331805324606qps, latency=0.69284072ms, p95=1.126946ms, p99=1.350359msrun multiple benchmarks with config fileYou may run multiple benchmarks with different index and dataset. you could use--run-filerun benchmarks from a config file.Below is a example config file:config.yamldefault:index_factory:annb.anns.faiss.indexes.index_under_test_factoryindex_factory_args:{}index_name:Testdataset:gist-960-euclidean.hdf5topk:10step:10jobs:1loop:2result:output.pthruns:-name:faiss-gist960-gpu-ivfflatindex_args:gpu:yesindex:ivfflatnlist:1024query_args:-nprobe:1-nprobe:16-nprobe:256-name:faiss-gist960-gpu-ivfpq8index_args:gpu:yesindex:ivfpqnlist:1024query_args:-nprobe:1-nprobe:16-nprobe:256Explanation for above config file:The default section is the default config for all benchmarks.The config keys are generally same as the options forannb-testcommand. e.g.index_factoryis same as--index-factory.You could define multiple benchmarks inrunssection. and each run config will override the default config. In this example, we define use gist-960-euclidean.hdf5 as dataset, so it will use this dataset for all benchmarks. and we use different index and query args for each benchmark. for index_args, we use ivfflat(nlist=1024) and ivfpq(nlist=1024) as two benchmark series. and for query_args, we use nprobe=1,16,256 for each benchmark. That means we will run 6 benchmarks in total, each series will run 3 benchmarks with different nprobe.The result will be saved to output.pth file by default setting. Actually, each benchmark series will save to a separate file. so in this example, we will get two files:output-1.pthandoutput-2.pth. you could useannb-reportto view them.more optionsYou could useannb-test --helpto see more options.❯annb-test--helpCheck Benchmark ResultsTheannb-reportis use to view benchmark results as plain/csv text, or export them to Chart graphic.annb-report--helpexamples for view/export benchmark resultsview benchmark results as plain textannb-reportoutput.pthview benchmark results as csv textannb-reportoutput.pth--formatcsvexport benchmark results to chart graphic(multiple series)annb-reportoutput.pth--formatpng--outputoutput.pngoutput-1.pthoutput-2.pth