Project: Qdrant (App Definition and Development / Database)
File: DEVELOPMENT.md

Build your own from source ```bash docker build . --tag=qdrant/qdrant ``` Or use latest pre-built image from ```bash docker pull qdrant/qdrant ``` To run the container, use the command: ```bash docker run -p 6333:6333 qdrant/qdrant ``` And once you need a fine-grained setup, you can also define a storage path and custom configuration: ```bash docker run -p 6333:6333 \ -v $(pwd)/path/to/data:/qdrant/storage \ -v $(pwd)/path/to/snapshots:/qdrant/snapshots \ -v $(pwd)/path/to/custom_config.yaml:/qdrant/config/production.yaml \ qdrant/qdrant ``` `/qdrant/storage` - is the place where Qdrant persists all your data. Make sure to mount it as a volume, otherwise docker will drop it with the container. `/qdrant/snapshots` - is the place where Qdrant stores `/qdrant/config/production.yaml` - is the file with engine configuration. You can override any value from the Now Qdrant should be accessible at . To run Qdrant on local development environment you need to install below: Install Rust, follow: Install `rustfmt` toolchain for Rust ```shell rustup component add rustfmt ``` Install dependencies: ```shell sudo apt-get update -y sudo apt-get upgrade -y sudo apt-get install -y curl unzip gcc-multilib \ clang cmake jq \ g++-9-aarch64-linux-gnu \ gcc-9-aarch64-linux-gnu ``` Install `protoc` from source ```shell PROTOC_VERSION=22.2 PKGNAME=$(uname -s | awk '{print ($1 == "Darwin") ? "osx-universalbinary" : (($1 == "Linux") ? "linux-x86_64" : "")}') curl -LO https://github.com/protocolbuffers/protobuf/releases//download/v$PROTOCVERSION/protoc-$PROTOCVERSION-$PKG_NAME.zip unzip protoc-$PROTOCVERSION-$PKGNAME.zip -d $HOME/.local export PATH="$PATH:$HOME/.local/bin" rm protoc-$PROTOCVERSION-$PKGNAME.zip protoc --version ``` Build and run the app ```shell cargo build --release --bin qdrant ./target/release/qdrant ``` Use the web UI Web UI repo is , but there's a utility script to sync it to the `static` folder: ```shell ./tools/sync-web-ui.sh ``` There are several benchmarks implemented in Qdrant. Benchmarks are not included in CI/CD and might take some time to execute. So the expected approach to benchmarking is to run only ones which might be affected by your changes. To run benchmark, use the following command inside a related sub-crate: ```bash cargo bench --bench nameofbenchmark ``` In this case you will see the execution timings and, if you launched this bench earlier, the difference in execution time. Example output: ``` scoring-vector/basic-score-point time: [111.81 us 112.07 us 112.31 us] change: [+19.567% +20.454% +21.404%] (p = 0.00 < 0.05) Performance has regressed. Found 9 outliers among 100 measurements (9.00%) 3 (3.00%) low severe 3 (3.00%) low mild 2 (2.00%) high mild 1 (1.00%) high severe scoring-vector/basic-score-point-10x time: [111.86 us 112.44 us 113.04 us] change: [-1.6120% -0.5554% +0.5103%] (p = 0.32 > 0.05) No change in performance detected. Found 1 outliers among 100 measurements (1.00%) 1

high mild ``` To run benchmarks with profiler to generate FlameGraph - use the following command: ```bash cargo bench --bench nameofbenchmark -- --profile-time=60 ``` This command will run each benchmark iterator for `60` seconds and generate FlameGraph svg along with profiling records files. These records could later be used to generate visualisation of the call-graph. Use and the following command to generate `svg` with a call graph: ```bash ~/go/bin/pprof -output=profile.svg -svg ${qdrantroot}/target/criterion/${benchmarkname}/${function_name}/profile/profile.pb ``` Qdrant have basic [`tracing`] support with [`Tracy`] profiler and [`tokio-console`] integrations that can be enabled with optional features. [`tracing`] is an optional dependency that can be enabled with `tracing` feature `tracy` feature enables [`Tracy`] profiler integration `console` feature enables [`tokio-console`] integration note, that you'll also have to to enable this feature by default [`tokio-console`] binds to `127.0.0.1:6669` if you want to connect [`tokio-console`] to Qdrant instance running inside a Docker container or on remote server, you can define `TOKIOCONSOLEBIND` when running Qdrant to override it (e.g., `TOKIOCONSOLEBIND=0.0.0.0:6669` to listen on all interfaces) `tokio-tracing` feature explicitly enables note, that you'll also have to to enable this feature this is required (and enabled automatically) by the `console` feature but you can enable it explicitly with the `tracy` feature, to see Tokio traces in [`Tracy`] profiler Qdrant code is not instrumented by default, so you'll have to manually add `#[tracing::instrument]` attributes on functions and methods that you want to profile. Qdrant uses [`tracing-log`] as the [`log`] backend, so `log` and `log-always` features of the [`tracing`] crate ! ```rust // `tracing` crate is an optional dependency in `lib/*` crates, so if you want the code to compile // when `tracing` feature is disabled, you have to use `#[cfg_attr(...)]`... // // See https://doc.rust-lang.org/reference/conditional-compilation.html#the-cfg_attr-attribute fn myfunction(someparameter: String) { // ... } // ...or if you just want to do some quick-and-dirty profiling, you can use `#[tracing::instrument]` // directly, just don't forget to add `--features tracing` when running `cargo` (or add `tracing` // to default features in `Cargo.toml`) fn someotherfunction() { // ... } ``` Qdrant uses the specification to document its API. This means changes to the API must be followed by changes to the specification. This is enforced by CI. Here is a quick step-by-step guide: code endpoints and model in Rust change specs in `/openapi/*ytt.yaml` add new schema definitions to `src/schema_generator.rs` run `/tools/generateopenapimodels.sh` to generate specs update integration tests `tests/openapi/openapiintegration` and run them with `./tests/openapiintegration_test.sh` expose file by starting an HTTP server, for instance `python -m http.server`, in `/docs/redoc` validate specs by browsing redoc on `http://localhost:8000/?v=master` validate `openapi-merged.yaml` using Qdrant uses to serve gRPC traffic. Our protocol buffers are defined in `lib/api/src/grpc/proto/*.proto` define request and response types using protocol buffers (use for enums payloads) specify RPC methods inside the service definition using protocol buffers `cargo build` will generate the struct definitions and a service trait implement the service trait in Rust start server `cargo run --bin qdrant` run integration test `./tests/basicgrpctest.sh` generate docs `./tools/generategrpcdocs.sh` Here is a good for reference.