Project: Vald (App Definition and Development / Database)
File: vald-agent-standalone-on-docker.md

Vald is designed and implemented based on Cloud-Native architecture. However, there may be cases that want to use only Vald Agent without Kubernetes. This article will show you how to deploy and run the Vald Agent on Docker. Fashion-MNIST is used as an example dataset, same as . Docker: v19.0 ~ Go: v1.15 ~ libhdf5 (only required for this tutorial.) HDF5 is required for this tutorial. If hdf5 is not installed, please install . <details><summary>[Optional] Install HDF5</summary><br> ```bash yum install -y hdf5-devel apt-get install libhdf5-serial-dev brew install hdf5 ``` </details> This chapter will show you how to deploy Vald Agent on docker.<br> This chapter will use NGT for the core engine of Vald Agent. Clone the vdaas/vald repository ```bash git clone https://github.com/vdaas/vald.git ``` Create a directory for setting deploy vald-agent-ngt ```bash cd vald && mkdir -p tutorial && cd tutorial ``` Create the `config.yaml` The configuration of Vald agent for docker is set using `config.yaml`<br> You can also check . ```bash cat << EOF > config.yaml version: v0.0.0 time_zone: JST logging: logger: glg level: debug format: raw server_config: servers: name: agent-grpc host: 0.0.0.0 port: 8081 mode: GRPC probewaittime: "3s" http: shutdown_duration: "5s" handler_timeout: "" idle_timeout: "" readheadertimeout: "" read_timeout: "" write_timeout: "" startup_strategy: agent-grpc shutdown_strategy: agent-grpc fullshutdownduration: 600s tls: enabled: false ngt: index_path: "/etc/server/backup" dimension: 784 bulkinsertchunk_size: 10 distance_type: l2 object_type: float creationedgesize: 20 searchedgesize: 10 autoindexduration_limit: 1m autoindexcheck_duration: 10s autoindexlength: 100 autosaveindex_duration: 90s initialdelaymax_duration: 60s defaultpoolsize: 500 EOF ``` Create a backup directory To avoid removing the indexing data due to any trouble after finishing indexing, we should prepare the path for auto backup. ```bash mkdir -p backup ``` Deploy Vald Agent on Docker To deploy Vald agent on docker with `config.yaml`, you can run the below command. Note: Please check whether there is `config.yaml` file and `backup` directory in your current

```bash docker run -v $(pwd):/etc/server -u "$(id -u $USER):$(id -g $USER)" -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -p 8081:8081 --rm -it vdaas/vald-agent-ngt ``` Verify If the deployment success, you can confirm the output will be similar to below. ```bash 2020-07-01 03:02:41 [INFO]: maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined 2020-07-01 03:02:41 [INFO]: service agent ngt v0.0.0 starting... 2020-07-01 03:02:41 [INFO]: executing daemon pre-start function 2020-07-01 03:02:41 [INFO]: executing daemon start function 2020-07-01 03:02:41 [INFO]: server agent-grpc executing preStartFunc 2020-07-01 12:02:41 [INFO]: gRPC server agent-grpc starting on 0.0.0.0:8081 ``` Download dataset This tutorial uses as a dataset for indexing and search query. ```bash cd example/client/agent ``` ```bash wget https://ann-benchmarks.com/fashion-mnist-784-euclidean.hdf5 ``` Running example Vald provides multiple language client libraries such as Go, Java, Node.js, Python, etc.<br> This example will insert and index 400 vectors into the Vald from the Fashion-MNIST dataset via gRPC. And then, after waiting for indexing, it will request to search the nearest vector 10 times. You will get the 10 nearest neighbor vectors for each search query.<br> We use to run the example. The example code is the same as running an example-only Vald agent on Kubernetes. If you want to learn the detail of running an example, please refer to the tutorial of . ```bash go run main.go ``` <div class="caution"> It would be best to run CreateIndex() after Insert() without waiting for auto-indexing in your client code, even if you can wait for the finishing auto createIndex function, which sometimes takes a long time. The backup files (e.g., ngt-meta.kvsdb) will be in your mount directory when vald-agent-ngt finishes indexing. </div> <div class="warning"> If you use Go(v1.16~) and catch the error like `missing go.sum entry to add it` when running `go run main.go`, please run `go mod tidy` and retry. This error comes from Go Command Changes of Go 1.16 Release Notes.(Please refer to https://golang.org/doc/go1.16#go-command for more details). </div> Clean Up Stop the Vald Agent docker container via `Ctrl+C`. Congratulation! You achieved this tutorial! If you want, you can try other tutorials such as: For more information, we recommend you to check: