Repository: jina-ai/serve
Pull request: feat: add beta support to scale Stateful Executors with consensus using RAFT
**Goals:**
Provide a PoC of how StatefulExecutors could handle with consensus

Implementation:

- [x] Handle raftadmin add voters from jina orchestrate
- [x] Handle terminate signals on the raft node
- [x] SIGTERM Shutdown handling
- [x] Logger in raft naming editing (if they merge and release)
- [x] Check the replica trying for indexing #5601
- [x] Make it work with shards (workspace handling)
- [x] Expose RAFT options in parser
- [ ] Panic if FSM Apply fails https://github.com/hashicorp/raft/issues/307 (We have some failure reasons like grpc connectivity that should not be there)
- [x] Implement proper Snapshot and Restore
- [x] properly create workspace folders and log/data files, without the need for explicitly setting `raft_bootstrap` parameter
- [x] Improve AddVoter calls (we do 10 for now until it works)
- [x] Improve readiness check, (node ready before restore is done), RAFT node should expose a health check service which is a proxy to executor health check service. It should be used by Pod.wait_for_ready. 
- [x] Implement bulding wheels for different platforms (Look at ANNLite for wheels building)
- [x] HANDLE LICENSE (since we do not change the code, no need to do anything)
- [x] Handle ContainerPod
- [x] Clean logs
- [x] RPC should implement streaming endpoint
- [x] currently, ports are being assigned in an underterministic way. The snapshots store information about each node in the cluster. When a cluster needs to be restored, we have to make sure it is spawned with the same ports attributed to replicas.
- [x] Fix raft library compilation for emulated python on mac (will build to `arm` architecture and therefore can't be linked to `x86_64` emulated python)
- [x] Handle better reduction in Shards (Head)
- [ ] (OPTIONAL) Implement ApplyBatch in the FSM that will help on reusing client and connection
- [ ] (OPTIONAL) Make jina.pb.go independent of docarray proto. (Maybe assume DocArray is just binary?)
- [ ] (OPTIONAL) Study if we can call WorkerRequestHandler directly from Golang


Testing:

- [ ] Unit test golang module
- [x] Integration test
- [ ] Try killing nodes, see leadership changes


Further understanding:
- [ ] How to handle timeout leader in snapshotting (Right now FAIL, can we give up leadership?, run snapshot in new process (copy-on-write)?
- [x] Understand restore behavior
- [x] Understand vote, bootstrap, etc ...


Problem statement:
In Local, same request is not guaranteed to try every replica when retries= -1
When the retry happens in the ConnectionPool, the ReplicaList asynchornously gets next connections, but this may imply that for same request, the same replica is tried more than once even in case of failure.

Hints from issue discussion:
@jina-ai/product This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days

Commit message: docs: add numbers of QPS in example

Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>

Patch to review:
diff --git a/.github/workflows/cd.yml b/.github/workflows/cd.yml
index 21b2cd51ccf67..21e3ee7ccae5f 100644
--- a/.github/workflows/cd.yml
+++ b/.github/workflows/cd.yml
@@ -129,14 +129,15 @@ jobs:
           python -m pip install wheel
           pip install -U protobuf==3.20.3
           pip install ".[common,devel,test]" --no-cache-dir
-          pip install -U "git+https://github.com/docarray/docarray@feat-rewrite-v2#egg=docarray[common]"
+          pip install -U "docarray[common]"
           jina
           export JINA_LOG_LEVEL="ERROR"
       - name: Test
         id: test
         run: |
           pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/hub_usage/dummyhub*' tests/integration/docarray_v2
-
+          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/hub_usage/dummyhub*' tests/integration/stateful
+          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/hub_usage/dummyhub*' tests/integration/deployment_http_composite
           echo "flag it as jina for codeoverage"
           echo "codecov_flag=jina" >> $GITHUB_OUTPUT
         timeout-minutes: 30
@@ -185,7 +186,7 @@ jobs:
       - name: Test
         id: test
         run: |
-          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/docarray_v2/*' --ignore-glob='tests/integration/hub_usage/dummyhub*' ${{ matrix.test-path }}
+          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/docarray_v2/*' --ignore-glob='tests/integration/stateful/*' --ignore-glob='tests/integration/hub_usage/dummyhub*' ${{ matrix.test-path }}
           echo "flag it as jina for codeoverage"
           echo "codecov_flag=jina" >> $GITHUB_OUTPUT
         timeout-minutes: 30
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 1738f500ff8c9..a9cc492e1aef6 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -405,7 +405,9 @@ jobs:
       - name: Test
         id: test
         run: |
-          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/hub_usage/dummyhub*' tests/integration/docarray_v2  tests/integration/deployment_http_composite
+          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/hub_usage/dummyhub*' tests/integration/docarray_v2
+          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/hub_usage/dummyhub*' tests/integration/stateful
+          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/hub_usage/dummyhub*' tests/integration/deployment_http_composite
 
           echo "flag it as jina for codeoverage"
           echo "codecov_flag=jina" >> $GITHUB_OUTPUT
@@ -426,6 +428,123 @@ jobs:
           flags: ${{ steps.test.outputs.codecov_flag }}
           fail_ci_if_error: false
 
+  # Build the wheels for Linux and macOS for Python 3.8 and newer
+  build-wheels:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      # Ensure that a wheel builder finishes even if another fails
+      fail-fast: false
+      matrix:
+        include:
+          # linux
+          - os: ubuntu-latest
+            python: '3.8'
+            python-manylinux-tag: "cp38-cp38"
+          - os: ubuntu-latest
+            python: '3.9'
+            python-manylinux-tag: "cp39-cp39"
+          - os: ubuntu-latest
+            python: '3.10'
+            python-manylinux-tag: "cp310-cp310"
+          - os: ubuntu-latest
+            python: '3.11'
+            python-manylinux-tag: "cp311-cp311"
+
+          # MacOS emulated
+          - os: macos-latest
+            python: '3.7'
+            python-cibuildwheels: '37'
+            platform_id: macosx_x86_64
+          - os: macos-latest
+            python: '3.8'
+            python-cibuildwheels: '38'
+            platform_id: macosx_x86_64
+          - os: macos-latest
+            python: '3.9'
+            python-cibuildwheels: '39'
+            platform_id: macosx_x86_64
+          - os: macos-latest
+            python: '3.10'
+            python-cibuildwheels: '310'
+            platform_id: macosx_x86_64
+          - os: macos-latest
+            python: '3.11'
+            python-cibuildwheels: '311'
+            platform_id: macosx_x86_64
+
+          # MacOS native
+          - os: macos-latest
+            python: '3.8'
+            python-cibuildwheels: '38'
+            platform_id: macosx_arm64
+          - os: macos-latest
+            python: '3.9'
+            python-cibuildwheels: '39'
+            platform_id: macosx_arm64
+          - os: macos-latest
+            python: '3.10'
+            python-cibuildwheels: '310'
+            platform_id: macosx_arm64
+          - os: macos-latest
+            python: '3.11'
+            python-cibuildwheels: '311'
+            platform_id: macosx_arm64
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v2
+
+      - name: Set up Python ${{ matrix.python }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Set up Golang
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.19.5
+
+      - name: Update version
+        shell: bash
+        run: |
+          git fetch --depth=1 origin +refs/tags/*:refs/tags/*
+          ./scripts/update-version.sh stateful
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install setuptools wheel
+          pip install git+https://github.com/alaeddine-13/setuptools-golang.git
+          pip install cibuildwheel
+
+      - name: Build wheels with setuptools-golang-build-manylinux-wheel
+        if: ${{ matrix.os == 'ubuntu-latest' }}
+        run: |
+          setuptools-golang-build-manylinux-wheels --pythons ${{ matrix.python-manylinux-tag }}
+
+      - name: Build wheels with cibuildwheels on macos
+        env:
+          CIBW_BUILD: cp${{ matrix.python-cibuildwheels }}-${{ matrix.platform_id }}
+          CIBW_ARCHS: all
+          CIBW_TEST_COMMAND: python -c "import jina; import jraft"
+          CIBW_BUILD_VERBOSITY: 1
+        if: ${{ matrix.os == 'macos-latest' }}
+        run: |
+          python -m cibuildwheel --output-dir dist
+
+      - name: Test wheels
+        run: |
+          WHEEL_FILE=$(ls dist/*.whl)
+          python -m pip install $WHEEL_FILE
+          python -c "import jraft"
+        if: ${{ matrix.os != 'macos-latest' || matrix.platform_id != 'macosx_arm64' }} # runners do not necessarily have macos ARM, so cannot run this test for it
+
+      - name: Upload wheels as artifacts
+        uses: actions/upload-artifact@v2
+        with:
+          path: dist/*.whl
+
+
   core-test:
     needs: prep-testbed
     runs-on: ubuntu-latest
@@ -442,6 +561,9 @@ jobs:
         uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
+      - uses: actions/setup-go@v4
+        with:
+          go-version: '^1.19.5' # The Go version to download (if necessary) and use.
       - name: Prepare environment
         run: |
           docker build -f Dockerfiles/test-pip.Dockerfile -t jinaai/jina:test-pip .
@@ -453,14 +575,12 @@ jobs:
       - name: Test
         id: test
         run: |
-          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/docarray_v2/*' --ignore-glob='tests/integration/hub_usage/dummyhub*' ${{ matrix.test-path }}
+          pytest --suppress-no-test-exit-code --force-flaky --min-passes 1 --max-runs 5 --cov=jina --cov-report=xml --timeout=600 -v -s --ignore-glob='tests/integration/docarray_v2/*' --ignore-glob='tests/integration/stateful/*' --ignore-glob='tests/integration/hub_usage/dummyhub*' ${{ matrix.test-path }}
           if [[ "${{ matrix.test-path }}" == *"tests/integration/instrumentation/" ]]
           then
             echo "sleeping to let open telemetry export daemon thread expire by itself"
             sleep 2
-          fi 
-
-
+          fi
           echo "flag it as jina for codeoverage"
           echo "codecov_flag=jina" >> $GITHUB_OUTPUT
         timeout-minutes: 30
diff --git a/.github/workflows/stateful-build-wheels.yml b/.github/workflows/stateful-build-wheels.yml
new file mode 100644
index 0000000000000..c09cf719d1256
--- /dev/null
+++ b/.github/workflows/stateful-build-wheels.yml
@@ -0,0 +1,153 @@
+name: Jina StatefulExecutor build and release wheels Workflow
+
+on:
+  push:
+    branches:
+      - jina-v-raft # change later with jina-v-raft
+
+jobs:
+  # Build the wheels for Linux, Windows and macOS for Python 3.8 and newer
+  build-wheels:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      # Ensure that a wheel builder finishes even if another fails
+      fail-fast: false
+      matrix:
+        include:
+          # linux
+          - os: ubuntu-latest
+            python: '3.8'
+            python-manylinux-tag: "cp38-cp38"
+          - os: ubuntu-latest
+            python: '3.9'
+            python-manylinux-tag: "cp39-cp39"
+          - os: ubuntu-latest
+            python: '3.10'
+            python-manylinux-tag: "cp310-cp310"
+          - os: ubuntu-latest
+            python: '3.11'
+            python-manylinux-tag: "cp311-cp311"
+
+          # MacOS emulated
+          - os: macos-latest
+            python: '3.7'
+            python-cibuildwheels: '37'
+            platform_id: macosx_x86_64
+          - os: macos-latest
+            python: '3.8'
+            python-cibuildwheels: '38'
+            platform_id: macosx_x86_64
+          - os: macos-latest
+            python: '3.9'
+            python-cibuildwheels: '39'
+            platform_id: macosx_x86_64
+          - os: macos-latest
+            python: '3.10'
+            python-cibuildwheels: '310'
+            platform_id: macosx_x86_64
+          - os: macos-latest
+            python: '3.11'
+            python-cibuildwheels: '311'
+            platform_id: macosx_x86_64
+
+          # MacOS native
+          - os: macos-latest
+            python: '3.8'
+            python-cibuildwheels: '38'
+            platform_id: macosx_arm64
+          - os: macos-latest
+            python: '3.9'
+            python-cibuildwheels: '39'
+            platform_id: macosx_arm64
+          - os: macos-latest
+            python: '3.10'
+            python-cibuildwheels: '310'
+            platform_id: macosx_arm64
+          - os: macos-latest
+            python: '3.11'
+            python-cibuildwheels: '311'
+            platform_id: macosx_arm64
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 200
+
+      - name: Set up Python ${{ matrix.python }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Set up Golang
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.19.5
+
+      - name: Update version
+        shell: bash
+        run: |
+          git fetch --depth=200 origin +refs/tags/*:refs/tags/*
+          echo "commits since v3.14.1"
+          git rev-list v3.14.1..HEAD --count
+          ./scripts/update-version.sh stateful
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install setuptools wheel
+          pip install git+https://github.com/alaeddine-13/setuptools-golang.git
+          pip install cibuildwheel
+
+      - name: Build wheels with setuptools-golang-build-manylinux-wheel
+        if: ${{ matrix.os == 'ubuntu-latest' }}
+        run: |
+          setuptools-golang-build-manylinux-wheels --pythons ${{ matrix.python-manylinux-tag }}
+
+      - name: Build wheels with cibuildwheels on macos
+        env:
+          CIBW_BUILD: cp${{ matrix.python-cibuildwheels }}-${{ matrix.platform_id }}
+          CIBW_ARCHS: all
+          CIBW_TEST_COMMAND: python -c "import jina; import jraft"
+          CIBW_BUILD_VERBOSITY: 1
+        if: ${{ matrix.os == 'macos-latest' }}
+        run: |
+          python -m cibuildwheel --output-dir dist
+
+      - name: Test wheels
+        run: |
+          WHEEL_FILE=$(ls dist/*.whl)
+          python -m pip install $WHEEL_FILE
+          python -c "import jraft"
+        if: ${{ matrix.os != 'macos-latest' || matrix.platform_id != 'macosx_arm64' }} # runners do not necessarily have macos ARM, so cannot run this test for it
+
+      - name: Upload wheels as artifacts
+        uses: actions/upload-artifact@v2
+        with:
+          path: dist/*.whl
+
+  stateful-release:
+    needs: build-wheels
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          token: ${{ secrets.JINA_DEV_BOT }}
+          fetch-depth: 100  # means max contribute history is limited to 100 lines
+      #          submodules: true
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.7
+      - uses: actions/download-artifact@v3
+        with:
+          name: artifact
+          path: dist
+
+
+      - name: Pre-release (.devN)
+        run: |
+          pip install twine wheel
+          twine upload dist/*
+        env:
+          TWINE_USERNAME: ${{ secrets.TWINE_USERNAME }}
+          TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }}
diff --git a/Dockerfiles/pip-perf.Dockerfile b/Dockerfiles/pip-perf.Dockerfile
index 2668bdb6a1d78..52f28f2c0af1c 100644
--- a/Dockerfiles/pip-perf.Dockerfile
+++ b/Dockerfiles/pip-perf.Dockerfile
@@ -3,10 +3,20 @@ ARG PIP_TAG
 
 FROM python:${PY_VERSION}-slim
 
-RUN apt-get update && apt-get install --no-install-recommends -y gcc libc6-dev net-tools procps htop lsof dnsutils
+RUN apt-get update && apt-get install --no-install-recommends -y gcc libc6-dev net-tools procps htop lsof dnsutils pkg-config wget
 
 RUN if [[ $PY_VERSION==3.11 ]]; then apt-get install --no-install-recommends -y build-essential ; fi
 
+# Download and extract Go 1.19
+RUN wget -q https://golang.org/dl/go1.19.linux-amd64.tar.gz \
+    && tar -C /usr/local -xzf go1.19.linux-amd64.tar.gz \
+    && rm go1.19.linux-amd64.tar.gz
+
+# Set up environment variables for Go
+ENV PATH="/usr/local/go/bin:${PATH}"
+ENV GOPATH="/go"
+ENV GOBIN="/go/bin"
+
 COPY . /jina/
 
 RUN cd /jina && pip install ."$PIP_TAG"
diff --git a/Dockerfiles/pip.Dockerfile b/Dockerfiles/pip.Dockerfile
index c4c023f24f2f2..46a97c9ffe889 100644
--- a/Dockerfiles/pip.Dockerfile
+++ b/Dockerfiles/pip.Dockerfile
@@ -3,10 +3,20 @@ ARG PIP_TAG
 
 FROM python:${PY_VERSION}-slim
 
-RUN apt-get update && apt-get install --no-install-recommends -y gcc libc6-dev
+RUN apt-get update && apt-get install --no-install-recommends -y gcc libc6-dev pkg-config wget
 
 RUN if [[ $PY_VERSION==3.11 ]]; then apt-get install --no-install-recommends -y build-essential ; fi
 
+# Download and extract Go 1.19
+RUN wget -q https://golang.org/dl/go1.19.linux-amd64.tar.gz \
+    && tar -C /usr/local -xzf go1.19.linux-amd64.tar.gz \
+    && rm go1.19.linux-amd64.tar.gz
+
+# Set up environment variables for Go
+ENV PATH="/usr/local/go/bin:${PATH}"
+ENV GOPATH="/go"
+ENV GOBIN="/go/bin"
+
 COPY . /jina/
 
 RUN cd /jina && pip install ."$PIP_TAG"
diff --git a/Dockerfiles/protogen.Dockerfile b/Dockerfiles/protogen.Dockerfile
index 74205ec2c8ed8..5c9fb387eb0f8 100644
--- a/Dockerfiles/protogen.Dockerfile
+++ b/Dockerfiles/protogen.Dockerfile
@@ -18,7 +18,7 @@ ADD jina ./jina/
 
 WORKDIR /jina/proto
 
-ENTRYPOINT ["bash", "./build-proto.sh", "/builder/grpc/cmake/build/grpc_python_plugin"]
+ENTRYPOINT ["bash", "./build-proto.sh", "/builder/grpc/cmake/build/grpc_python_plugin", "pb2"]
 
 
 
diff --git a/Dockerfiles/test-pip.Dockerfile b/Dockerfiles/test-pip.Dockerfile
index a8e21252bf293..097e50201c39a 100644
--- a/Dockerfiles/test-pip.Dockerfile
+++ b/Dockerfiles/test-pip.Dockerfile
@@ -1,9 +1,20 @@
 ARG PY_VERSION=3.7
 ARG PIP_TAG
 
+
 FROM python:${PY_VERSION}-slim
 
-RUN apt-get update && apt-get install --no-install-recommends -y gcc libc6-dev net-tools procps htop lsof dnsutils
+RUN apt-get update && apt-get install --no-install-recommends -y gcc libc6-dev net-tools procps htop lsof dnsutils pkg-config wget
+
+# Download and extract Go 1.19
+RUN wget -q https://golang.org/dl/go1.19.linux-amd64.tar.gz \
+    && tar -C /usr/local -xzf go1.19.linux-amd64.tar.gz \
+    && rm go1.19.linux-amd64.tar.gz
+
+# Set up environment variables for Go
+ENV PATH="/usr/local/go/bin:${PATH}"
+ENV GOPATH="/go"
+ENV GOBIN="/go/bin"
 
 COPY . /jina/
 
diff --git a/MANIFEST.in b/MANIFEST.in
index 73be01e35d661..19bb3e55e48a9 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,5 +1,6 @@
 recursive-include docarray/resources *
 recursive-include jina/resources *
+recursive-include jina/serve/consensus *
 include extra-requirements.txt
 include fastentrypoints.py
 include LICENSE
diff --git a/docs/concepts/orchestration/gateway-args.md b/docs/concepts/orchestration/gateway-args.md
index fbf4c07213158..4bd04cc7a8f36 100644
--- a/docs/concepts/orchestration/gateway-args.md
+++ b/docs/concepts/orchestration/gateway-args.md
@@ -49,4 +49,6 @@
 | `traces_exporter_port` | If tracing is enabled, this port will be used to configure the trace exporter agent. | `number` | `None` |
 | `metrics` | If set, the sdk implementation of the OpenTelemetry metrics will be available for default monitoring and custom measurements. Otherwise a no-op implementation will be provided. | `boolean` | `False` |
 | `metrics_exporter_host` | If tracing is enabled, this hostname will be used to configure the metrics exporter agent. | `string` | `None` |
-| `metrics_exporter_port` | If tracing is enabled, this port will be used to configure the metrics exporter agent. | `number` | `None` |
\ No newline at end of file
+| `metrics_exporter_port` | If tracing is enabled, this port will be used to configure the metrics exporter agent. | `number` | `None` |
+| `stateful` | If set, start consensus module to make sure write operations are properly replicated between all the replicas | `boolean` | `False` |
+| `pod_ports` | When using StatefulExecutors, if they want to restart it is important to keep the RAFT cluster configuration | `number` | `None` |
\ No newline at end of file
diff --git a/docs/concepts/orchestration/scale-out.md b/docs/concepts/orchestration/scale-out.md
index ffbfb90284ea4..bcfc144bb4e5a 100644
--- a/docs/concepts/orchestration/scale-out.md
+++ b/docs/concepts/orchestration/scale-out.md
@@ -4,7 +4,7 @@
 By default, all Executors in an Orchestration run with a single instance. If an Executor is particularly slow, then it will reduce the overall throughput. To solve this, you can specify the number of `replicas` to scale out an Executor.
 
 (replicate-executors)=
-## Replicate Executors
+## Replicate stateless Executors
 
 Replication creates multiple copies of the same {class}`~jina.Executor`. Each request in the Orchestration is then passed to only one replica (instance) of that Executor. **All replicas compete for a request. The idle replica gets the request first.**
 
@@ -55,6 +55,213 @@ executors:
 Flow with three replicas of `slow_encoder` and one replica of `fast_indexer`
 ```
 
+(scale-consensus)=
+## Replicate stateful Executors with consensus using RAFT (Beta)
+
+````{admonition} docarray 0.30
+:class: note
+
+Since docarray version > 0.30, docarray changed its interface and implementation drastically. We intend to support these new docarray versions
+in the near future, but not every feature is yet available {ref}`Check here <docarray-v2>`. This feature has been added with these new docarray versions support.
+
+This feature is only available when using `grpc` as the protocol for the Deployment or when the Deployment is part of a Flow
+````
+
+````{admonition} gRPC protocol
+:class: note
+
+This feature is only available when using gRPC as the protocol for the Deployment or when the Deployment is part of a Flow
+````
+
+Replication is used to scale-out Executors by creating copies of them that can handle requests in parallel providing better RPS.
+However, when an Executor keeps some sort of state, then it is not simple to guarantee that each copy of the Executor keeps the same state,
+which can lead to undesired behavior, since each replica can provide different results, depending on the specific state they hold.
+
+In Jina, you can also have replication while guaranteeing the consensus between Executors. For this, we rely on (RAFT)[https://raft.github.io/] which is
+an algorithm that guarantees eventual consistency between replicas. 
+
+Consensus-based replication using RAFT is a distributed algorithm designed to provide fault tolerance and consistency in a distributed system. In a distributed system, the nodes may fail, and messages may be lost or delayed, which can lead to inconsistencies in the system.
+The problem with traditional replication methods is that they can't guarantee consistency in a distributed system in the presence of failures. This is where consensus-based replication using RAFT comes in.
+In this approach, each Executor can be considered as a Finite State Machine, which means that it has a set of states that it can be in, and a set of transitions that it can make between those states. Each request that is sent to the Executor can be considered as a log entry that needs to be replicated across the cluster.
+
+In order to enable this kind of replication, we need to consider:
+
+- Specify which methods of the Executor {ref}` could be updating its internal state <stateful-executor>`.
+- Tell the deployment to use the RAFT consensus algorithm by setting the `--stateful` argument.
+- Set values of replicas compatible with RAFT. RAFT requires a minimum of 3 replicas to guarantee consistency. 
+- Pass the `--peer-ports` argument so that the RAFT cluster can recover from a previous configuration of replicas if existed.
+- Optionally you can pass `--raft-configuration` parameter to tweak the behavior of the consensus module. You can understand the values to pass from
+[Hashicorp's RAFT library](https://github.com/ongardie/hashicorp-raft/blob/master/config.go).
+
+```python
+from jina import Deployment, Executor, requests
+from jina.serve.executors.decorators import write
+from docarray import DocList
+from docarray.documents import TextDoc
+
+
+class MyStateStatefulExecutor(Executor):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._docs_dict = {}
+
+    @requests(on=['/index'])
+    @write
+    def index(self, docs: DocList[TextDoc], **kwargs) -> DocList[TextDoc]:
+        for doc in docs:
+            self._docs_dict[doc.id] = doc
+
+    @requests(on=['/search'])
+    def search(self,  docs: DocList[TextDoc], **kwargs) -> DocList[TextDoc]:
+        for doc in docs:
+            self.logger.debug(f'Searching against {len(self._docs_dict)} documents')
+            doc.text = self._docs_dict[doc.id].text
+
+
+
+d = Deployment(name='stateful_executor', 
+               uses=MyStateStatefulExecutor,
+               replicas=3, 
+               stateful=True, 
+               peer_ports=[12345, 12346, 12347])
+with d:
+    d.block()
+```
+
+This capacity allows us not only to have replicas work with robustness and availability, it also can help us to achieve higher throughput in some cases.
+
+Let's imagine we write an Executor that is used to index and query documents from a vector index.
+
+For this, we are going to use an in memory solution from [DocArray](https://docs.docarray.org/user_guide/storing/index_in_memory/) that performs exact vector search.
+
+```python
+from jina import Deployment, Executor, requests
+from jina.serve.executors.decorators import write
+from docarray import DocList
+from docarray.documents import TextDoc
+from docarray.index.backends.in_memory import InMemoryExactNNIndex
+
+
+class QueryDoc(TextDoc):
+    matches: DocList[TextDoc] = DocList[TextDoc]()
+
+
+class ExactNNSearch(Executor):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._index = InMemoryExactNNIndex[TextDoc]()
+
+    @requests(on=['/index'])
+    @write # I add write decorator to indicate that calling this endpoint updates the inner state
+    def index(self, docs: DocList[TextDoc], **kwargs) -> DocList[TextDoc]:
+        self.logger.info(f'Indexing Document in index with {len(self._index)} documents indexed')
+        self._index.index(docs)
+
+    @requests(on=['/search'])
+    def search(self,  docs: DocList[QueryDoc], **kwargs) -> DocList[QueryDoc]:
+        self.logger.info(f'Searching Document in index with {len(self._index)} documents indexed')
+        for query in docs:
+            docs, scores = self._index.find(query, search_field='embedding', limit=100)
+            query.matches = docs
+
+d = Deployment(name='indexer',
+               port=5555,
+               uses=ExactNNSearch,
+               workspace='./raft',
+               replicas=3,
+               stateful=True,
+               peer_ports=[12345, 12346, 12347])
+with d:
+    d.block()
+```
+
+Then in another terminal, we are going to send index and search requests:
+
+```python
+from jina import Client
+from docarray import DocList
+from docarray.documents import TextDoc
+import time
+import numpy as np
+
+
+class QueryDoc(TextDoc):
+    matches: DocList[TextDoc] = DocList[TextDoc]()
+
+
+NUM_DOCS_TO_INDEX = 100000
+NUM_QUERIES = 1000
+
+c = Client(port=5555)
+
+index_docs = DocList[TextDoc](
+    [TextDoc(text=f'I am document {i}', embedding=np.random.rand(128)) for i in range(NUM_DOCS_TO_INDEX)])
+start_indexing_time = time.time()
+c.post(on='/index', inputs=index_docs, request_size=100)
+print(f'Indexing {NUM_DOCS_TO_INDEX} Documents took {time.time() - start_indexing_time}s')
+time.sleep(2)  # let some time for the data to be replicated
+
+search_da = DocList[QueryDoc](
+    [QueryDoc(text=f'I am document {i}', embedding=np.random.rand(128)) for i in range(NUM_QUERIES)])
+start_querying_time = time.time()
+responses = c.post(on='/search', inputs=search_da, request_size=1)
+print(f'Searching {NUM_QUERIES} Queries took {time.time() - start_querying_time}s')
+for res in responses:
+    print(f'{res.matches}')
+```
+
+In the logs of the `server` you can see how `index` requests reach every replica while `search` requests only reach one replica in a 
+`round robin` fashion.
+
+Eventually every Indexer replica ends up with the same Documents indexed. 
+
+```text
+INFO   indexer/rep-2@923 Indexing Document in index with 99900 documents indexed                                                                                                                                   
+INFO   indexer/rep-0@902 Indexing Document in index with 99200 documents indexed                                                                                                                                   
+INFO   indexer/rep-1@910 Indexing Document in index with 99700 documents indexed                                                                                                                                   
+INFO   indexer/rep-1@910 Indexing Document in index with 99800 documents indexed                                                                                                                [04/28/23 16:51:06]
+INFO   indexer/rep-0@902 Indexing Document in index with 99300 documents indexed                                                                                                                [04/28/23 16:51:06]
+INFO   indexer/rep-1@910 Indexing Document in index with 99900 documents indexed                                                                                                                                   
+INFO   indexer/rep-0@902 Indexing Document in index with 99400 documents indexed                                                                                                                                   
+INFO   indexer/rep-0@902 Indexing Document in index with 99500 documents indexed                                                                                                                                   
+INFO   indexer/rep-0@902 Indexing Document in index with 99600 documents indexed                                                                                                                                   
+INFO   indexer/rep-0@902 Indexing Document in index with 99700 documents indexed                                                                                                                                   
+INFO   indexer/rep-0@902 Indexing Document in index with 99800 documents indexed                                                                                                                                   
+INFO   indexer/rep-0@902 Indexing Document in index with 99900 documents indexed 
+```
+
+But at search time, the consensus module does not affect, and only one replica serves the queries.
+```text
+INFO   indexer/rep-0@902 Searching Document in index with 100000 documents indexed                                                                                                              [04/28/23 16:59:21]
+INFO   indexer/rep-1@910 Searching Document in index with 100000 documents indexed                                                                                                              [04/28/23 16:59:21]
+INFO   indexer/rep-2@923 Searching Document in index with 100000 documents indexed 
+```
+
+If you run the same example by setting replicas to 1 without the consensus module, you can see the benefits it has in the QPS at search time,
+while there is a little cost on the time used for indexing.
+
+```python
+d = Deployment(name='indexer',
+               port=5555,
+               uses=ExactNNSearch,
+               replicas=1)
+```
+
+With 1 replica:
+
+```text
+Indexing 100000 Documents took 18.93274688720703s
+Searching 1000 Queries took 385.96641397476196s
+```
+
+With 3 replicas and consensus:
+```text
+Indexing 100000 Documents took 35.066415548324585s
+Searching 1000 Queries took 202.07950615882874s
+```
+
+This allows to go from 2.5 to 5 QPS.
+
 ## Replicate on multiple GPUs
 
 To replicate your {class}`~jina.Executor`s so that each replica uses a different GPU on your machine, you can tell the Orchestration to use multiple GPUs by passing `CUDA_VISIBLE_DEVICES=RR` as an environment variable.
diff --git a/docs/concepts/serving/executor/create.md b/docs/concepts/serving/executor/create.md
index efd19a0272b9b..638a297be336e 100644
--- a/docs/concepts/serving/executor/create.md
+++ b/docs/concepts/serving/executor/create.md
@@ -11,9 +11,9 @@ An {class}`~jina.Executor` is a self-contained microservice exposed using a gRPC
 It contains functions (decorated with `@requests`) that process `DocumentArray`s. Executors follow three principles:
 
 1. An Executor should subclass directly from the `jina.Executor` class.
-2. An Executor class is a bag of functions with shared state or configuration (via `self`); it can contain an arbitrary number of
-functions with arbitrary names.
+2. An Executor is a Python class; it can contain any number of functions.
 3. Functions decorated by {class}`~jina.requests` are exposed as services according to their `on=` endpoint. These functions can be coroutines (`async def`) or regular functions. This will be explained later in {ref}`Add Endpoints Section<exec-endpoint>`
+4. (Beta) Functions decorated by {class}`~jina.serve.executors.decorators.write` on top of their {class}`~jina.requests` decoration, are considered to update the internal state of the Executor. `__init__` and `close` method are exceptions. Why this can be useful will be explained in {ref}`Stateful-executor<stateful-executor>` 
 
 ## Create an Executor
 
@@ -183,3 +183,33 @@ structure.
 ```{tip}
 In the `jina hub new` wizard you can choose from four Dockerfile templates: `cpu`, `tf-gpu`, `torch-gpu`, and `jax-gpu`.
 ```
+
+## Stateful-Executor (Beta)
+
+Executors sometimes may contain some internal state which changes when some of their methods are called. For instance, an Executor could contain an index of Documents
+to perform vector search.
+
+In these cases, orchestrating these Executors can be tougher than Executors that never change their inner state (Imagine a Machine Learning model served via an Executor that never updates its weights during its lifetime).
+The challenge comes at guaranteeing the consistency between `replicas` of the same Executor inside the same Deployment.
+
+In order to provide this consistency, Executors can mark some of their exposed methods as `write`. This indicates that calls to these endpoints must be consistently replicated between all the replicas
+such that other endpoints can serve independently of the replica that is hit.
+
+````{admonition} Deterministic state update
+:class: note
+
+Another consideration that needs to be there, is that the Executor inner state must evolve in a deterministic manner if we want `replicas` to behave consistently.
+````
+
+By considering this, {ref}`Executors can be scaled in a consistent manner<scale-consensus>`.
+
+### Snapshot and restoring
+
+When having an Stateful Executor, Jina uses RAFT consensus algorithm to guarantee that every replica eventually holds the same inner state. 
+RAFT writes the incoming requests as logs to local storage in every replica to make sure this is achieved. 
+
+This could become problematic if the Executor runs for long time as the logs file could grow indifinitely. However, one can avoid this problem
+by describing the methods `def snapshot(self, snapshot_dir)` and `def restore(self, snapshot_dir)` that are triggered via the RAFT protocol allowing the Executor
+to store its current state or to recover its state from a snapshot. With this mechanism, RAFT can keep cleaning old logs by assuming that the state of the Executor
+at a given time is determined by its latest snapshot and the application of all the requests that arrived since the last snapshot. RAFT algorithm keeps track
+of all these details.
\ No newline at end of file
diff --git a/docs/proto/docs.md b/docs/proto/docs.md
new file mode 100644
index 0000000000000..6e7bcc90c18fc
--- /dev/null
+++ b/docs/proto/docs.md
@@ -0,0 +1,398 @@
+# Protocol Documentation
+<a name="top"></a>
+
+## Table of Contents
+
+- [docarray.proto](#docarray-proto)
+    - [DocumentArrayProto](#docarray-DocumentArrayProto)
+  
+- [jina.proto](#jina-proto)
+    - [DataRequestListProto](#jina-DataRequestListProto)
+    - [DataRequestProto](#jina-DataRequestProto)
+    - [DataRequestProto.DataContentProto](#jina-DataRequestProto-DataContentProto)
+    - [DataRequestProtoWoData](#jina-DataRequestProtoWoData)
+    - [EndpointsProto](#jina-EndpointsProto)
+    - [HeaderProto](#jina-HeaderProto)
+    - [JinaInfoProto](#jina-JinaInfoProto)
+    - [JinaInfoProto.EnvsEntry](#jina-JinaInfoProto-EnvsEntry)
+    - [JinaInfoProto.JinaEntry](#jina-JinaInfoProto-JinaEntry)
+    - [RelatedEntity](#jina-RelatedEntity)
+    - [RouteProto](#jina-RouteProto)
+    - [StatusProto](#jina-StatusProto)
+    - [StatusProto.ExceptionProto](#jina-StatusProto-ExceptionProto)
+  
+    - [StatusProto.StatusCode](#jina-StatusProto-StatusCode)
+  
+    - [JinaDataRequestRPC](#jina-JinaDataRequestRPC)
+    - [JinaDiscoverEndpointsRPC](#jina-JinaDiscoverEndpointsRPC)
+    - [JinaGatewayDryRunRPC](#jina-JinaGatewayDryRunRPC)
+    - [JinaInfoRPC](#jina-JinaInfoRPC)
+    - [JinaRPC](#jina-JinaRPC)
+    - [JinaSingleDataRequestRPC](#jina-JinaSingleDataRequestRPC)
+  
+- [Scalar Value Types](#scalar-value-types)
+
+
+
+<a name="docarray-proto"></a>
+<p align="right"><a href="#top">Top</a></p>
+
+## docarray.proto
+
+
+
+<a name="docarray-DocumentArrayProto"></a>
+
+### DocumentArrayProto
+this file is just a placeholder for the DA coming from jina._docarray dependency
+
+
+
+
+
+ 
+
+ 
+
+ 
+
+ 
+
+
+
+<a name="jina-proto"></a>
+<p align="right"><a href="#top">Top</a></p>
+
+## jina.proto
+
+
+
+<a name="jina-DataRequestListProto"></a>
+
+### DataRequestListProto
+Represents a list of data requests
+This should be replaced by streaming
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| requests | [DataRequestProto](#jina-DataRequestProto) | repeated | requests in this list |
+
+
+
+
+
+
+<a name="jina-DataRequestProto"></a>
+
+### DataRequestProto
+Represents a DataRequest
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| header | [HeaderProto](#jina-HeaderProto) |  | header contains meta info defined by the user |
+| parameters | [google.protobuf.Struct](#google-protobuf-Struct) |  | extra kwargs that will be used in executor |
+| routes | [RouteProto](#jina-RouteProto) | repeated | status info on every routes |
+| data | [DataRequestProto.DataContentProto](#jina-DataRequestProto-DataContentProto) |  | container for docs and groundtruths |
+
+
+
+
+
+
+<a name="jina-DataRequestProto-DataContentProto"></a>
+
+### DataRequestProto.DataContentProto
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| docs | [docarray.DocumentArrayProto](#docarray-DocumentArrayProto) |  | the docs in this request |
+| docs_bytes | [bytes](#bytes) |  | the docs in this request as bytes |
+
+
+
+
+
+
+<a name="jina-DataRequestProtoWoData"></a>
+
+### DataRequestProtoWoData
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| header | [HeaderProto](#jina-HeaderProto) |  | header contains meta info defined by the user |
+| parameters | [google.protobuf.Struct](#google-protobuf-Struct) |  | extra kwargs that will be used in executor |
+| routes | [RouteProto](#jina-RouteProto) | repeated | status info on every routes |
+
+
+
+
+
+
+<a name="jina-EndpointsProto"></a>
+
+### EndpointsProto
+Represents the set of Endpoints exposed by an Executor
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| endpoints | [string](#string) | repeated | list of endpoints exposed by an Executor |
+
+
+
+
+
+
+<a name="jina-HeaderProto"></a>
+
+### HeaderProto
+Represents a Header.
+- The header&#39;s content will be defined by the user request.
+- It will be copied to the envelope.header
+- In-flow operations will modify the envelope.header
+- While returning, copy envelope.header back to request.header
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| request_id | [string](#string) |  | the unique ID of this request. Multiple requests with the same ID will be gathered |
+| status | [StatusProto](#jina-StatusProto) |  | status info |
+| exec_endpoint | [string](#string) | optional | the endpoint specified by `@requests(on=&#39;/abc&#39;)` |
+| target_executor | [string](#string) | optional | if set, the request is targeted to certain executor, regex strings |
+| timeout | [uint32](#uint32) | optional | epoch time in seconds after which the request should be dropped |
+
+
+
+
+
+
+<a name="jina-JinaInfoProto"></a>
+
+### JinaInfoProto
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| jina | [JinaInfoProto.JinaEntry](#jina-JinaInfoProto-JinaEntry) | repeated | information about the system running and package version information including jina |
+| envs | [JinaInfoProto.EnvsEntry](#jina-JinaInfoProto-EnvsEntry) | repeated | the environment variable setting |
+
+
+
+
+
+
+<a name="jina-JinaInfoProto-EnvsEntry"></a>
+
+### JinaInfoProto.EnvsEntry
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| key | [string](#string) |  |  |
+| value | [string](#string) |  |  |
+
+
+
+
+
+
+<a name="jina-JinaInfoProto-JinaEntry"></a>
+
+### JinaInfoProto.JinaEntry
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| key | [string](#string) |  |  |
+| value | [string](#string) |  |  |
+
+
+
+
+
+
+<a name="jina-RelatedEntity"></a>
+
+### RelatedEntity
+Represents an entity (like an ExecutorRuntime)
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| id | [string](#string) |  | unique id of the entity, like the name of a pod |
+| address | [string](#string) |  | address of the entity, could be an IP address, domain name etc, does not include port |
+| port | [uint32](#uint32) |  | port this entity is listening on |
+| shard_id | [uint32](#uint32) | optional | the id of the shard it belongs to, if it is a shard |
+
+
+
+
+
+
+<a name="jina-RouteProto"></a>
+
+### RouteProto
+Represents a the route paths of this message as perceived by the Gateway
+start_time is set when the Gateway sends a message to a Pod
+end_time is set when the Gateway receives a message from a Pod
+thus end_time - start_time includes Executor computation, runtime overhead, serialization and network
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| executor | [string](#string) |  | the name of the BasePod |
+| start_time | [google.protobuf.Timestamp](#google-protobuf-Timestamp) |  | time when the Gateway starts sending to the Pod |
+| end_time | [google.protobuf.Timestamp](#google-protobuf-Timestamp) |  | time when the Gateway received it from the Pod |
+| status | [StatusProto](#jina-StatusProto) |  | the status of the execution |
+
+
+
+
+
+
+<a name="jina-StatusProto"></a>
+
+### StatusProto
+Represents a Status
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| code | [StatusProto.StatusCode](#jina-StatusProto-StatusCode) |  | status code |
+| description | [string](#string) |  | error description of the very first exception |
+| exception | [StatusProto.ExceptionProto](#jina-StatusProto-ExceptionProto) |  | the details of the error |
+
+
+
+
+
+
+<a name="jina-StatusProto-ExceptionProto"></a>
+
+### StatusProto.ExceptionProto
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| name | [string](#string) |  | the class name of the exception |
+| args | [string](#string) | repeated | the list of arguments given to the exception constructor. |
+| stacks | [string](#string) | repeated | the exception traceback stacks |
+| executor | [string](#string) |  | the name of the executor bind to that Executor (if applicable) |
+
+
+
+
+
+ 
+
+
+<a name="jina-StatusProto-StatusCode"></a>
+
+### StatusProto.StatusCode
+
+
+| Name | Number | Description |
+| ---- | ------ | ----------- |
+| SUCCESS | 0 | success |
+| ERROR | 1 | error |
+
+
+ 
+
+ 
+
+
+<a name="jina-JinaDataRequestRPC"></a>
+
+### JinaDataRequestRPC
+jina gRPC service for DataRequests.
+
+| Method Name | Request Type | Response Type | Description |
+| ----------- | ------------ | ------------- | ------------|
+| process_data | [DataRequestListProto](#jina-DataRequestListProto) | [DataRequestProto](#jina-DataRequestProto) | Used for passing DataRequests to the Executors |
+
+
+<a name="jina-JinaDiscoverEndpointsRPC"></a>
+
+### JinaDiscoverEndpointsRPC
+jina gRPC service to expose Endpoints from Executors.
+
+| Method Name | Request Type | Response Type | Description |
+| ----------- | ------------ | ------------- | ------------|
+| endpoint_discovery | [.google.protobuf.Empty](#google-protobuf-Empty) | [EndpointsProto](#jina-EndpointsProto) |  |
+
+
+<a name="jina-JinaGatewayDryRunRPC"></a>
+
+### JinaGatewayDryRunRPC
+jina gRPC service to expose Endpoints from Executors.
+
+| Method Name | Request Type | Response Type | Description |
+| ----------- | ------------ | ------------- | ------------|
+| dry_run | [.google.protobuf.Empty](#google-protobuf-Empty) | [StatusProto](#jina-StatusProto) |  |
+
+
+<a name="jina-JinaInfoRPC"></a>
+
+### JinaInfoRPC
+jina gRPC service to expose information about running jina version and environment.
+
+| Method Name | Request Type | Response Type | Description |
+| ----------- | ------------ | ------------- | ------------|
+| _status | [.google.protobuf.Empty](#google-protobuf-Empty) | [JinaInfoProto](#jina-JinaInfoProto) |  |
+
+
+<a name="jina-JinaRPC"></a>
+
+### JinaRPC
+jina streaming gRPC service.
+
+| Method Name | Request Type | Response Type | Description |
+| ----------- | ------------ | ------------- | ------------|
+| Call | [DataRequestProto](#jina-DataRequestProto) stream | [DataRequestProto](#jina-DataRequestProto) stream | Pass in a Request and a filled Request with matches will be returned. |
+
+
+<a name="jina-JinaSingleDataRequestRPC"></a>
+
+### JinaSingleDataRequestRPC
+jina gRPC service for DataRequests.
+This is used to send requests to Executors when a list of requests is not needed
+
+| Method Name | Request Type | Response Type | Description |
+| ----------- | ------------ | ------------- | ------------|
+| process_single_data | [DataRequestProto](#jina-DataRequestProto) | [DataRequestProto](#jina-DataRequestProto) | Used for passing DataRequests to the Executors |
+
+ 
+
+
+
+## Scalar Value Types
+
+| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby |
+| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- |
+| <a name="double" /> double |  | double | double | float | float64 | double | float | Float |
+| <a name="float" /> float |  | float | float | float | float32 | float | float | Float |
+| <a name="int32" /> int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
+| <a name="int64" /> int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum |
+| <a name="uint32" /> uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) |
+| <a name="uint64" /> uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) |
+| <a name="sint32" /> sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
+| <a name="sint64" /> sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum |
+| <a name="fixed32" /> fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) |
+| <a name="fixed64" /> fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum |
+| <a name="sfixed32" /> sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
+| <a name="sfixed64" /> sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum |
+| <a name="bool" /> bool |  | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass |
+| <a name="string" /> string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) |
+| <a name="bytes" /> bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) |
+
diff --git a/jina/__init__.py b/jina/__init__.py
index 847c8f80628de..8cdd00578a421 100644
--- a/jina/__init__.py
+++ b/jina/__init__.py
@@ -65,7 +65,7 @@ def _warning_on_one_line(message, category, filename, lineno, *args, **kwargs):
 
 # do not change this line manually
 # this is managed by proto/build-proto.sh and updated on every execution
-__proto_version__ = '0.1.17'
+__proto_version__ = '0.1.27'
 
 try:
     __docarray_version__ = _docarray.__version__
diff --git a/jina/checker.py b/jina/checker.py
index d878c1209d09b..e0e2c9d691c44 100644
--- a/jina/checker.py
+++ b/jina/checker.py
@@ -1,6 +1,5 @@
 import argparse
 
-from jina.enums import ProtocolType
 from jina.helper import parse_host_scheme
 from jina.logging.predefined import default_logger
 
diff --git a/jina/constants.py b/jina/constants.py
index 6bdf6c28ede75..98593461bed5a 100644
--- a/jina/constants.py
+++ b/jina/constants.py
@@ -78,3 +78,5 @@
 
 __all__ = [_s for _s in dir() if not _s.startswith('_')]
 __all__.extend(_names_with_underscore)
+
+RAFT_TO_EXECUTOR_PORT = 100
diff --git a/jina/orchestrate/deployments/__init__.py b/jina/orchestrate/deployments/__init__.py
index 337ce9eb3142a..1c1f6bf5a8c05 100644
--- a/jina/orchestrate/deployments/__init__.py
+++ b/jina/orchestrate/deployments/__init__.py
@@ -5,7 +5,9 @@
 import re
 import subprocess
 import threading
+import multiprocessing
 import time
+import warnings
 from argparse import Namespace
 from collections import defaultdict
 from contextlib import ExitStack
@@ -50,8 +52,6 @@
 WRAPPED_SLICE_BASE = r'\[[-\d:]+\]'
 
 if TYPE_CHECKING:
-    import multiprocessing
-
     from jina.clients.base import BaseClient
     from jina.serve.executors import BaseExecutor
 
@@ -62,6 +62,28 @@ class DeploymentType(type(ExitStack), type(JAMLCompatible)):
     pass
 
 
+def _call_add_voters(leader, voters, replica_ids):
+    # this method needs to be run in multiprocess, importing jraft in main process
+    # makes it impossible to do tests sequentially
+
+    import jraft
+
+    for voter_address, replica_id in zip(voters, replica_ids):
+        success = False
+        for _ in range(3):
+            try:
+                jraft.add_voter(leader, str(replica_id), voter_address)
+                success = True
+                break
+            except ValueError:
+                time.sleep(2.0)
+
+        if not success:
+            warnings.warn(
+                f'Failed to add {str(replica_id)} as voter with address {voter_address} to leader at {leader}'
+            )
+
+
 class Deployment(JAMLCompatible, PostMixin, BaseOrchestrator, metaclass=DeploymentType):
     """A Deployment is an immutable set of pods, which run in replicas. They share the same input and output socket.
     Internally, the pods can run with the process/thread backend. They can also be run in their own containers
@@ -82,6 +104,36 @@ def __init__(
             self._pods = []
             self.head_pod = head_pod
 
+        def _add_voter_to_leader(self):
+            leader_address = f'{self._pods[0].runtime_ctrl_address}'
+            voter_addresses = [pod.runtime_ctrl_address for pod in self._pods[1:]]
+            replica_ids = [pod.args.replica_id for pod in self._pods[1:]]
+            process = multiprocessing.Process(
+                target=_call_add_voters,
+                kwargs={
+                    'leader': leader_address,
+                    'voters': voter_addresses,
+                    'replica_ids': replica_ids,
+                },
+            )
+            process.start()
+            process.join()
+
+        async def _async_add_voter_to_leader(self):
+            from concurrent.futures import ProcessPoolExecutor
+
+            leader_address = f'{self._pods[0].runtime_ctrl_address}'
+            voter_addresses = [pod.runtime_ctrl_address for pod in self._pods[1:]]
+            replica_ids = [pod.args.replica_id for pod in self._pods[1:]]
+            loop = asyncio.get_running_loop()
+            await loop.run_in_executor(
+                ProcessPoolExecutor(max_workers=1),
+                _call_add_voters,
+                leader_address,
+                voter_addresses,
+                replica_ids,
+            )
+
         @property
         def is_ready(self):
             return all(p.is_ready.is_set() for p in self._pods)
@@ -100,12 +152,21 @@ def join(self):
         def wait_start_success(self):
             for pod in self._pods:
                 pod.wait_start_success()
+            # TODO: is this the right behavior ?
+            # should this be done only when the cluster is started ?
+            if self._pods[0].args.stateful:
+                self._add_voter_to_leader()
 
         async def async_wait_start_success(self):
             await asyncio.gather(
                 *[pod.async_wait_start_success() for pod in self._pods]
             )
 
+            # TODO: is this the right behavior ?
+            # should this be done only when the cluster is started ?
+            if self._pods[0].args.stateful:
+                await self._async_add_voter_to_leader()
+
         def __enter__(self):
             for _args in self.args:
                 _args.noblock_on_start = True
@@ -166,6 +227,7 @@ def __init__(
         py_modules: Optional[List[str]] = None,
         quiet: Optional[bool] = False,
         quiet_error: Optional[bool] = False,
+        raft_configuration: Optional[dict] = None,
         reload: Optional[bool] = False,
         replicas: Optional[int] = 1,
         retries: Optional[int] = -1,
@@ -173,6 +235,7 @@ def __init__(
         shards: Optional[int] = 1,
         ssl_certfile: Optional[str] = None,
         ssl_keyfile: Optional[str] = None,
+        stateful: Optional[bool] = False,
         timeout_ctrl: Optional[int] = 60,
         timeout_ready: Optional[int] = 600000,
         timeout_send: Optional[int] = None,
@@ -269,6 +332,7 @@ def __init__(
           `Executor cookbook <https://docs.jina.ai/concepts/executor/executor-files/>`__
         :param quiet: If set, then no log will be emitted from this object.
         :param quiet_error: If set, then exception stack information will not be added to the log
+        :param raft_configuration: Dictionary of kwargs arguments that will be passed to the RAFT node as configuration options when starting the RAFT node.
         :param reload: If set, the Executor will restart while serving if YAML configuration source or Executor modules are changed. If YAML configuration is changed, the whole deployment is reloaded and new processes will be restarted. If only Python modules of the Executor have changed, they will be reloaded to the interpreter without restarting process.
         :param replicas: The number of replicas in the deployment
         :param retries: Number of retries per gRPC call. If <0 it defaults to max(3, num_replicas)
@@ -276,6 +340,7 @@ def __init__(
         :param shards: The number of shards in the deployment running at the same time. For more details check https://docs.jina.ai/concepts/flow/create-flow/#complex-flow-topologies
         :param ssl_certfile: the path to the certificate file
         :param ssl_keyfile: the path to the key file
+        :param stateful: If set, start consensus module to make sure write operations are properly replicated between all the replicas
         :param timeout_ctrl: The timeout in milliseconds of the control request, -1 for waiting forever
         :param timeout_ready: The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever
         :param timeout_send: The timeout in milliseconds used when sending data requests to Executors, -1 means no timeout, disabled by default
@@ -378,6 +443,15 @@ def __init__(
                 f'It is not supported to have {ProtocolType.WEBSOCKET.to_string()} deployment for '
                 f'Deployments with more than one shard'
             )
+
+        if self.args.stateful and (
+            ProtocolType.WEBSOCKET in self.args.protocol
+            or ProtocolType.HTTP in self.args.protocol
+            or len(self.args.protocol) > 1
+        ):
+            raise RuntimeError(
+                f'Stateful feature is only available for Deployments using a single {ProtocolType.GRPC.to_string()} protocol. {self.args.protocol} were requested'
+            )
         self.needs = (
             needs or set()
         )  #: used in the :class:`jina.flow.Flow` to build the graph
@@ -1208,13 +1282,39 @@ def _set_pod_args(self) -> Dict[int, List[Namespace]]:
                 cuda_visible_devices, replicas
             )
 
+        all_shard_pod_ports = self.args.peer_ports or {'0': []}
+        if isinstance(all_shard_pod_ports, str):
+            all_shard_pod_ports = json.loads(all_shard_pod_ports)
+        if isinstance(all_shard_pod_ports, list):
+            # it is a single shard
+            all_shard_pod_ports = {'0': all_shard_pod_ports}
+
+        peer_ports_all_shards = {}
+        for k, v in all_shard_pod_ports.items():
+            peer_ports_all_shards[str(k)] = v
+
+        if self.args.stateful and len(peer_ports_all_shards.keys()) < shards:
+            raise ValueError(
+                'The configuration of `peer_ports` does not match the number of shards requested'
+            )
+
         for shard_id in range(shards):
+            peer_ports = peer_ports_all_shards.get(str(shard_id), [])
+            if len(peer_ports) > 0 and len(peer_ports) != replicas:
+                raise ValueError(
+                    f'peer-ports argument does not match number of replicas, it will be ignored'
+                )
+            elif len(peer_ports) == 0:
+                peer_ports = [random_port() for _ in range(replicas)]
+
             replica_args = []
-            for replica_id in range(replicas):
+
+            for replica_id, peer_port in zip(range(replicas), peer_ports):
                 _args = copy.deepcopy(self.args)
                 if self.args.deployment_role == DeploymentRoleType.GATEWAY:
                     _args.replicas = replicas
                 _args.shard_id = shard_id
+                _args.replica_id = replica_id
                 # for gateway pods, the pod role shouldn't be changed
                 if _args.pod_role != PodRoleType.GATEWAY:
                     _args.pod_role = PodRoleType.WORKER
@@ -1263,6 +1363,7 @@ def _set_pod_args(self) -> Dict[int, List[Namespace]]:
                         _args.port = [
                             random_port() for _ in range(len(self.args.protocol))
                         ]
+                        _args.port[0] = peer_port
                     elif shards > 1:
                         port_monitoring_index = (
                             replica_id + replicas * shard_id + 1
@@ -1275,9 +1376,9 @@ def _set_pod_args(self) -> Dict[int, List[Namespace]]:
                                 port_monitoring_index
                             ]  # we skip the head port here
                         )
-                        _args.port = [random_port()]
+                        _args.port = [peer_port]
                     else:
-                        _args.port = [random_port()]
+                        _args.port = [peer_port]
                         _args.port_monitoring = random_port()
 
                 else:
@@ -1332,6 +1433,16 @@ def _parse_base_deployment_args(self, args):
             'pods': {},
         }
 
+        if self.args.stateful and self.args.replicas in [1, 2]:
+            self.logger.debug(
+                f'Stateful Executor is not recommended to be used less than 3 replicas'
+            )
+
+        if self.args.stateful and self.args.workspace is None:
+            raise ValueError(
+                f'Stateful Executors need to be provided `workspace` when used in a Deployment'
+            )
+
         # a gateway has no heads and uses
         # also there a no heads created, if there are no shards
         if self.role != DeploymentRoleType.GATEWAY and getattr(args, 'shards', 1) > 1:
diff --git a/jina/orchestrate/flow/base.py b/jina/orchestrate/flow/base.py
index 265a1b6ce7139..bc97460d4a008 100644
--- a/jina/orchestrate/flow/base.py
+++ b/jina/orchestrate/flow/base.py
@@ -205,6 +205,7 @@ def __init__(
         runtime_cls: Optional[str] = 'GatewayRuntime',
         ssl_certfile: Optional[str] = None,
         ssl_keyfile: Optional[str] = None,
+        stateful: Optional[bool] = False,
         timeout_ctrl: Optional[int] = 60,
         timeout_ready: Optional[int] = 600000,
         timeout_send: Optional[int] = None,
@@ -279,6 +280,7 @@ def __init__(
         :param runtime_cls: The runtime class to run inside the Pod
         :param ssl_certfile: the path to the certificate file
         :param ssl_keyfile: the path to the key file
+        :param stateful: If set, start consensus module to make sure write operations are properly replicated between all the replicas
         :param timeout_ctrl: The timeout in milliseconds of the control request, -1 for waiting forever
         :param timeout_ready: The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever
         :param timeout_send: The timeout in milliseconds used when sending data requests to Executors, -1 means no timeout, disabled by default
@@ -469,6 +471,7 @@ def __init__(
         :param runtime_cls: The runtime class to run inside the Pod
         :param ssl_certfile: the path to the certificate file
         :param ssl_keyfile: the path to the key file
+        :param stateful: If set, start consensus module to make sure write operations are properly replicated between all the replicas
         :param timeout_ctrl: The timeout in milliseconds of the control request, -1 for waiting forever
         :param timeout_ready: The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever
         :param timeout_send: The timeout in milliseconds used when sending data requests to Executors, -1 means no timeout, disabled by default
@@ -862,6 +865,7 @@ def add(
         py_modules: Optional[List[str]] = None,
         quiet: Optional[bool] = False,
         quiet_error: Optional[bool] = False,
+        raft_configuration: Optional[dict] = None,
         reload: Optional[bool] = False,
         replicas: Optional[int] = 1,
         retries: Optional[int] = -1,
@@ -869,6 +873,7 @@ def add(
         shards: Optional[int] = 1,
         ssl_certfile: Optional[str] = None,
         ssl_keyfile: Optional[str] = None,
+        stateful: Optional[bool] = False,
         timeout_ctrl: Optional[int] = 60,
         timeout_ready: Optional[int] = 600000,
         timeout_send: Optional[int] = None,
@@ -965,6 +970,7 @@ def add(
           `Executor cookbook <https://docs.jina.ai/concepts/executor/executor-files/>`__
         :param quiet: If set, then no log will be emitted from this object.
         :param quiet_error: If set, then exception stack information will not be added to the log
+        :param raft_configuration: Dictionary of kwargs arguments that will be passed to the RAFT node as configuration options when starting the RAFT node.
         :param reload: If set, the Executor will restart while serving if YAML configuration source or Executor modules are changed. If YAML configuration is changed, the whole deployment is reloaded and new processes will be restarted. If only Python modules of the Executor have changed, they will be reloaded to the interpreter without restarting process.
         :param replicas: The number of replicas in the deployment
         :param retries: Number of retries per gRPC call. If <0 it defaults to max(3, num_replicas)
@@ -972,6 +978,7 @@ def add(
         :param shards: The number of shards in the deployment running at the same time. For more details check https://docs.jina.ai/concepts/flow/create-flow/#complex-flow-topologies
         :param ssl_certfile: the path to the certificate file
         :param ssl_keyfile: the path to the key file
+        :param stateful: If set, start consensus module to make sure write operations are properly replicated between all the replicas
         :param timeout_ctrl: The timeout in milliseconds of the control request, -1 for waiting forever
         :param timeout_ready: The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever
         :param timeout_send: The timeout in milliseconds used when sending data requests to Executors, -1 means no timeout, disabled by default
@@ -1126,6 +1133,7 @@ def add(
           `Executor cookbook <https://docs.jina.ai/concepts/executor/executor-files/>`__
         :param quiet: If set, then no log will be emitted from this object.
         :param quiet_error: If set, then exception stack information will not be added to the log
+        :param raft_configuration: Dictionary of kwargs arguments that will be passed to the RAFT node as configuration options when starting the RAFT node.
         :param reload: If set, the Executor will restart while serving if YAML configuration source or Executor modules are changed. If YAML configuration is changed, the whole deployment is reloaded and new processes will be restarted. If only Python modules of the Executor have changed, they will be reloaded to the interpreter without restarting process.
         :param replicas: The number of replicas in the deployment
         :param retries: Number of retries per gRPC call. If <0 it defaults to max(3, num_replicas)
@@ -1133,6 +1141,7 @@ def add(
         :param shards: The number of shards in the deployment running at the same time. For more details check https://docs.jina.ai/concepts/flow/create-flow/#complex-flow-topologies
         :param ssl_certfile: the path to the certificate file
         :param ssl_keyfile: the path to the key file
+        :param stateful: If set, start consensus module to make sure write operations are properly replicated between all the replicas
         :param timeout_ctrl: The timeout in milliseconds of the control request, -1 for waiting forever
         :param timeout_ready: The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever
         :param timeout_send: The timeout in milliseconds used when sending data requests to Executors, -1 means no timeout, disabled by default
@@ -1319,6 +1328,7 @@ def config_gateway(
         runtime_cls: Optional[str] = 'GatewayRuntime',
         ssl_certfile: Optional[str] = None,
         ssl_keyfile: Optional[str] = None,
+        stateful: Optional[bool] = False,
         timeout_ctrl: Optional[int] = 60,
         timeout_ready: Optional[int] = 600000,
         timeout_send: Optional[int] = None,
@@ -1393,6 +1403,7 @@ def config_gateway(
         :param runtime_cls: The runtime class to run inside the Pod
         :param ssl_certfile: the path to the certificate file
         :param ssl_keyfile: the path to the key file
+        :param stateful: If set, start consensus module to make sure write operations are properly replicated between all the replicas
         :param timeout_ctrl: The timeout in milliseconds of the control request, -1 for waiting forever
         :param timeout_ready: The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever
         :param timeout_send: The timeout in milliseconds used when sending data requests to Executors, -1 means no timeout, disabled by default
@@ -1492,6 +1503,7 @@ def config_gateway(
         :param runtime_cls: The runtime class to run inside the Pod
         :param ssl_certfile: the path to the certificate file
         :param ssl_keyfile: the path to the key file
+        :param stateful: If set, start consensus module to make sure write operations are properly replicated between all the replicas
         :param timeout_ctrl: The timeout in milliseconds of the control request, -1 for waiting forever
         :param timeout_ready: The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever
         :param timeout_send: The timeout in milliseconds used when sending data requests to Executors, -1 means no timeout, disabled by default
diff --git a/jina/orchestrate/pods/__init__.py b/jina/orchestrate/pods/__init__.py
index 54ae3d77ceef6..44a598baea414 100644
--- a/jina/orchestrate/pods/__init__.py
+++ b/jina/orchestrate/pods/__init__.py
@@ -1,9 +1,9 @@
 import argparse
+import copy
 import multiprocessing
-import os
 import time
 from abc import ABC, abstractmethod
-from typing import TYPE_CHECKING, Dict, Optional, Type, Union
+from typing import Optional
 
 from jina.constants import __ready_msg__, __stop_msg__, __windows__
 from jina.enums import PodRoleType
@@ -13,104 +13,12 @@
 from jina.logging.logger import JinaLogger
 from jina.orchestrate.pods.helper import ConditionalEvent, _get_event
 from jina.parsers.helper import _update_gateway_args
-from jina.serve.runtimes.asyncio import AsyncNewLoopRuntime
-
-if TYPE_CHECKING:
-    import threading
+from jina.serve.executors.run import run, run_raft
+from jina.constants import RAFT_TO_EXECUTOR_PORT
 
 __all__ = ['BasePod', 'Pod']
 
 
-def run(
-        args: 'argparse.Namespace',
-        name: str,
-        runtime_cls: Type[AsyncNewLoopRuntime],
-        envs: Dict[str, str],
-        is_started: Union['multiprocessing.Event', 'threading.Event'],
-        is_shutdown: Union['multiprocessing.Event', 'threading.Event'],
-        is_ready: Union['multiprocessing.Event', 'threading.Event'],
-        jaml_classes: Optional[Dict] = None,
-):
-    """Method representing the :class:`BaseRuntime` activity.
-
-    This method is the target for the Pod's `thread` or `process`
-
-    .. note::
-        :meth:`run` is running in subprocess/thread, the exception can not be propagated to the main process.
-        Hence, please do not raise any exception here.
-
-    .. note::
-        Please note that env variables are process-specific. Subprocess inherits envs from
-        the main process. But Subprocess's envs do NOT affect the main process. It does NOT
-        mess up user local system envs.
-
-    .. warning::
-        If you are using ``thread`` as backend, envs setting will likely be overidden by others
-
-    .. note::
-        `jaml_classes` contains all the :class:`JAMLCompatible` classes registered in the main process.
-        When using `spawn` as the multiprocessing start method, passing this argument to `run` method re-imports
-        & re-registers all `JAMLCompatible` classes.
-
-    :param args: namespace args from the Pod
-    :param name: name of the Pod to have proper logging
-    :param runtime_cls: the runtime class to instantiate
-    :param envs: a dictionary of environment variables to be set in the new Process
-    :param is_started: concurrency event to communicate runtime is properly started. Used for better logging
-    :param is_shutdown: concurrency event to communicate runtime is terminated
-    :param is_ready: concurrency event to communicate runtime is ready to receive messages
-    :param jaml_classes: all the `JAMLCompatible` classes imported in main process
-    """
-    req_handler_cls = None
-    if runtime_cls == 'GatewayRuntime':
-        from jina.serve.runtimes.gateway.request_handling import GatewayRequestHandler
-        req_handler_cls = GatewayRequestHandler
-    elif runtime_cls == 'WorkerRuntime':
-        from jina.serve.runtimes.worker.request_handling import WorkerRequestHandler
-        req_handler_cls = WorkerRequestHandler
-    elif runtime_cls == 'HeadRuntime':
-        from jina.serve.runtimes.head.request_handling import HeaderRequestHandler
-        req_handler_cls = HeaderRequestHandler
-
-    logger = JinaLogger(name, **vars(args))
-
-    def _unset_envs():
-        if envs:
-            for k in envs.keys():
-                os.environ.pop(k, None)
-
-    def _set_envs():
-        if args.env:
-            os.environ.update({k: str(v) for k, v in envs.items()})
-
-    try:
-        _set_envs()
-
-        runtime = AsyncNewLoopRuntime(
-            args=args,
-            req_handler_cls=req_handler_cls,
-            gateway_load_balancer=getattr(args, 'gateway_load_balancer', False)
-        )
-    except Exception as ex:
-        logger.error(
-            f'{ex!r} during {runtime_cls!r} initialization'
-            + f'\n add "--quiet-error" to suppress the exception details'
-            if not args.quiet_error
-            else '',
-            exc_info=not args.quiet_error,
-        )
-    else:
-        if not is_shutdown.is_set():
-            is_started.set()
-            with runtime:
-                is_ready.set()
-                runtime.run_forever()
-    finally:
-        _unset_envs()
-        is_shutdown.set()
-        logger.debug(f'process terminated')
-
-
 class BasePod(ABC):
     """
     :class:`BasePod` is an interface from which all the classes managing the lifetime of a Runtime inside a local process,
@@ -334,10 +242,30 @@ class Pod(BasePod):
     def __init__(self, args: 'argparse.Namespace'):
         super().__init__(args)
         self.runtime_cls = self._get_runtime_cls()
+        cargs = None
+        self.raft_worker = None
+        if self.args.stateful and self.args.pod_role == PodRoleType.WORKER:
+            cargs_stateful = copy.deepcopy(args)
+            self.raft_worker = multiprocessing.Process(
+                target=run_raft,
+                kwargs={
+                    'args': cargs_stateful,
+                    'is_ready': self.is_ready,
+                },
+                name=self.name,
+                daemon=True,
+            )
+            cargs = copy.deepcopy(cargs_stateful)
+
+            if isinstance(cargs.port, int):
+                cargs.port += RAFT_TO_EXECUTOR_PORT
+            elif isinstance(cargs.port, list):
+                cargs.port = [port + RAFT_TO_EXECUTOR_PORT for port in cargs.port]
+        # if stateful, have a raft_worker
         self.worker = multiprocessing.Process(
             target=run,
             kwargs={
-                'args': args,
+                'args': cargs or args,
                 'name': self.name,
                 'envs': self._envs,
                 'is_started': self.is_started,
@@ -356,7 +284,8 @@ def start(self):
         .. #noqa: DAR201
         """
         self.worker.start()
-
+        if self.raft_worker is not None:
+            self.raft_worker.start()
         if not self.args.noblock_on_start:
             self.wait_start_success()
         return self
@@ -370,6 +299,8 @@ def join(self, *args, **kwargs):
         """
         self.logger.debug(f'joining the process')
         self.worker.join(*args, **kwargs)
+        if self.raft_worker is not None:
+            self.raft_worker.join(*args, **kwargs)
         self.logger.debug(f'successfully joined the process')
 
     def _terminate(self):
@@ -378,6 +309,8 @@ def _terminate(self):
         """
         self.logger.debug(f'terminating the runtime process')
         self.worker.terminate()
+        if self.raft_worker is not None:
+            self.raft_worker.terminate()
         self.logger.debug(f'runtime process properly terminated')
 
     def _get_runtime_cls(self):
diff --git a/jina/orchestrate/pods/helper.py b/jina/orchestrate/pods/helper.py
index 198e9ed0faa49..527c8afd4c846 100644
--- a/jina/orchestrate/pods/helper.py
+++ b/jina/orchestrate/pods/helper.py
@@ -1,5 +1,4 @@
 import multiprocessing
-import re
 from copy import deepcopy
 from functools import partial
 from typing import TYPE_CHECKING
@@ -16,7 +15,7 @@
 
 def _get_event(obj) -> multiprocessing.Event:
     if isinstance(obj, multiprocessing.Process) or isinstance(
-            obj, multiprocessing.context.ForkProcess
+        obj, multiprocessing.context.ForkProcess
     ):
         return multiprocessing.Event()
     elif isinstance(obj, multiprocessing.context.SpawnProcess):
diff --git a/jina/parsers/helper.py b/jina/parsers/helper.py
index cdd50729787c0..bbc19029fb8e0 100644
--- a/jina/parsers/helper.py
+++ b/jina/parsers/helper.py
@@ -326,6 +326,37 @@ def __call__(self, parser, args, values, option_string=None):
         setattr(args, self.dest, d)
 
 
+class CastPeerPorts(argparse.Action):
+    """argparse action to cast potential inputs to `peer-ports` argument"""
+
+    def __call__(self, parser, args, values, option_string=None):
+        """
+        call the CastPeerPorts
+
+
+        .. # noqa: DAR401
+        :param parser: the parser
+        :param args: args to initialize the values
+        :param values: the values to add to the parser
+        :param option_string: inherited, not used
+        """
+        import json
+        d = {0: []}
+        for value in values:
+            if isinstance(value, str):
+                value = json.loads(value)
+            if isinstance(value, dict):
+                for k, vlist in value.items():
+                    d[k] = []
+                    for v in vlist:
+                        d[k].append(_port_to_int(v))
+            elif isinstance(value, int):
+                d[0].append(value)
+            else:
+                d[0] = [_port_to_int(port) for port in value]
+        setattr(args, self.dest, d)
+
+
 def _port_to_int(port):
     try:
         return int(port)
diff --git a/jina/parsers/orchestrate/pod.py b/jina/parsers/orchestrate/pod.py
index 687ca2ab19845..62e886247352c 100644
--- a/jina/parsers/orchestrate/pod.py
+++ b/jina/parsers/orchestrate/pod.py
@@ -9,6 +9,7 @@
 from jina.parsers.helper import (
     _SHOW_ALL_ARGS,
     CastToIntAction,
+    CastPeerPorts,
     KVAppendAction,
     add_arg_group,
 )
@@ -108,6 +109,16 @@ def mixin_pod_parser(parser, pod_type: str = 'worker'):
         help='If set, the current Pod/Deployment can not be further chained, '
              'and the next `.add()` will chain after the last Pod/Deployment not this current one.',
     )
+
+    gp.add_argument(
+        '--replica-id',
+        type=int,
+        default=0,
+        help='defines the replica identifier for the executor. It is used when `stateful` is set to true'
+        if _SHOW_ALL_ARGS
+        else argparse.SUPPRESS,
+    )
+
     if pod_type != 'gateway':
         gp.add_argument(
             '--reload',
@@ -132,6 +143,7 @@ def mixin_pod_parser(parser, pod_type: str = 'worker'):
             help='If set, the Gateway will restart while serving if YAML configuration source is changed.',
         )
     mixin_pod_runtime_args_parser(gp, pod_type=pod_type)
+    mixin_stateful_parser(gp)
 
 
 def mixin_pod_runtime_args_parser(arg_group, pod_type='worker'):
@@ -242,3 +254,29 @@ def mixin_pod_runtime_args_parser(arg_group, pod_type='worker'):
         default=None,
         help='If tracing is enabled, this port will be used to configure the metrics exporter agent.',
     )
+
+
+def mixin_stateful_parser(parser):
+    """Mixing in arguments required to work with Stateful Executors into the given parser.
+    :param parser: the parser instance to which we add arguments
+    """
+
+    gp = add_arg_group(parser, title='Stateful Executor')
+
+    gp.add_argument(
+        '--stateful',
+        action='store_true',
+        default=False,
+        help='If set, start consensus module to make sure write operations are properly replicated between all the replicas',
+    )
+    gp.add_argument(
+        '--peer-ports',
+        type=str,
+        default=None,
+        help='When using --stateful option, it is required to tell the cluster what are the cluster configuration. This is important'
+             'when the Deployment is restarted. It indicates the ports to which each replica of the cluster binds.'
+             ' It is expected to be a single list if shards == 1 or a dictionary if shards > 1.',
+        action=CastPeerPorts,
+        nargs='+',
+    )
+
diff --git a/jina/parsers/orchestrate/runtimes/runtime.py b/jina/parsers/orchestrate/runtimes/runtime.py
index 4922a7cfb0295..70054b57bb597 100644
--- a/jina/parsers/orchestrate/runtimes/runtime.py
+++ b/jina/parsers/orchestrate/runtimes/runtime.py
@@ -13,6 +13,23 @@ def mixin_base_runtime_parser(arg_group):
         action=KVAppendAction,
         metavar='KEY: VALUE',
         nargs='*',
-        help="Dictionary of kwargs arguments that will be passed to the grpc server as options when starting the server, example : {'grpc.max_send_message_length': -1}",
+        help="Dictionary of kwargs arguments that will be passed to the grpc server as options when starting the "
+        "server, example : {'grpc.max_send_message_length': -1}",
+        default=None,
+    )
+
+
+def mixin_raft_parser(arg_group):
+    """Mixing in arguments required by the RAFT Node. All these args are used to configure the RAFT nodes
+    :param arg_group: the parser instance to which we add arguments
+    """
+
+    arg_group.add_argument(
+        '--raft-configuration',
+        action=KVAppendAction,
+        metavar='KEY: VALUE',
+        nargs='*',
+        help="Dictionary of kwargs arguments that will be passed to the RAFT node as configuration options when "
+        "starting the RAFT node.",
         default=None,
     )
diff --git a/jina/parsers/orchestrate/runtimes/worker.py b/jina/parsers/orchestrate/runtimes/worker.py
index b26f103ff6dcc..07645fd9959e0 100644
--- a/jina/parsers/orchestrate/runtimes/worker.py
+++ b/jina/parsers/orchestrate/runtimes/worker.py
@@ -4,7 +4,10 @@
 from jina.parsers.orchestrate.runtimes.grpc_channel import (
     mixin_grpc_channel_options_parser,
 )
-from jina.parsers.orchestrate.runtimes.runtime import mixin_base_runtime_parser
+from jina.parsers.orchestrate.runtimes.runtime import (
+    mixin_base_runtime_parser,
+    mixin_raft_parser,
+)
 
 
 def mixin_worker_runtime_parser(parser):
@@ -110,7 +113,8 @@ def mixin_worker_runtime_parser(parser):
         '--disable-reduce',
         action='store_true',
         default=False,
-        help='Disable the built-in reduction mechanism. Set this if the reduction is to be handled by the Executor itself by operating on a `docs_matrix` or `docs_map`',
+        help='Disable the built-in reduction mechanism. Set this if the reduction is to be handled by the Executor '
+        'itself by operating on a `docs_matrix` or `docs_map`',
     )
 
     gp.add_argument(
@@ -121,4 +125,5 @@ def mixin_worker_runtime_parser(parser):
     )
 
     mixin_base_runtime_parser(gp)
+    mixin_raft_parser(gp)
     mixin_grpc_channel_options_parser(gp)
diff --git a/jina/proto/docarray_v1/build-proto.sh b/jina/proto/docarray_v1/build-proto.sh
index 9e286ecca9b9e..bdee4a25fb771 100755
--- a/jina/proto/docarray_v1/build-proto.sh
+++ b/jina/proto/docarray_v1/build-proto.sh
@@ -66,4 +66,4 @@ sed -i 's/^__proto_version__ = '\''\(.*\)'\''/__proto_version__ = '\'"$NEWVER"\'
 
 
 printf "\e[1;32mAll done!\e[0m\n"
-printf "if you are running this inside Docker container, you may manually bump proto version to:\t\e[1;32m$NEWVER\e[0m\n"
\ No newline at end of file
+printf "if you are running this inside Docker container, you may manually bump proto version to:\t\e[1;32m$NEWVER\e[0m\n"
diff --git a/jina/proto/docarray_v1/docarray.proto b/jina/proto/docarray_v1/docarray.proto
index 52c71c97d1a9e..f2eb878cd2160 100644
--- a/jina/proto/docarray_v1/docarray.proto
+++ b/jina/proto/docarray_v1/docarray.proto
@@ -3,7 +3,7 @@ syntax = "proto3";
 package docarray;
 
 /*
- this file is just a placeholder for the DA coming from jina._docarray dependency
+ this file is just a placeholder for the DA coming from docarray dependency
  */
 message DocumentArrayProto {
 
diff --git a/jina/proto/docarray_v1/jina.proto b/jina/proto/docarray_v1/jina.proto
index a021b39d6f1e1..c46797123402b 100644
--- a/jina/proto/docarray_v1/jina.proto
+++ b/jina/proto/docarray_v1/jina.proto
@@ -52,6 +52,7 @@ message EndpointsProto {
 
     // list of endpoints exposed by an Executor
     repeated string endpoints = 1;
+    repeated string write_endpoints = 2;
 }
 
 /**
@@ -193,3 +194,110 @@ service JinaInfoRPC {
     }
 }
 
+/**
+* An ID representation for the Snapshot Job
+*/
+message SnapshotId {
+    string value = 1;
+}
+
+/**
+* An ID representation for the Restore Job
+*/
+message RestoreId {
+    string value = 1;
+}
+
+
+/**
+* Represents the status of a Snapshot.
+*/
+message SnapshotStatusProto {
+
+    SnapshotId id = 1; // An ID for the Snapshot Job.
+
+    enum Status {
+        STATE_UNSPECIFIED = 0;
+        
+        QUEUED = 1; // Snapshot Job is admitted (validated and persisted) and waiting for resources.
+        
+        SCHEDULED = 2; // Snapshot Job is scheduled to run as soon as resource allocation is ready.
+        
+        RUNNING = 3; // Resource allocation has been successful. Snapshot Job is RUNNING.
+        
+        SUCCEEDED = 4; // Snapshot Job has finished successfully.
+        
+        FAILED = 5; // Snapshot Job has failed.
+
+        NOT_FOUND = 6; // If the specific snapshot id is unknown or not found
+    }
+
+    // status code
+    Status status = 2;
+    
+    // snapshot file
+    string snapshot_file = 3;
+}
+
+/**
+* jina gRPC service to trigger a snapshot at the Executor Runtime.
+*/
+service JinaExecutorSnapshot {
+    rpc snapshot (google.protobuf.Empty) returns (SnapshotStatusProto) {
+    }
+}
+
+/**
+* jina gRPC service to trigger a snapshot at the Executor Runtime.
+*/
+service JinaExecutorSnapshotProgress {
+    rpc snapshot_status (SnapshotId) returns (SnapshotStatusProto) {
+    }
+}
+
+/**
+* Represents the status of a Restore.
+*/
+message RestoreSnapshotStatusProto {
+
+    RestoreId id = 1; // An ID for the Restore Job.
+
+    enum Status {
+        STATE_UNSPECIFIED = 0;
+
+        RUNNING = 1; // Resource allocation has been successful. Restore Job is RUNNING.
+
+        SUCCEEDED = 2; // Resource Job has finished successfully.
+
+        FAILED = 3; // Resource Job has failed.
+
+        NOT_FOUND = 6; // If the specific restore id is unknown or not found
+    }
+
+    // status code
+    Status status = 2;
+}
+
+/**
+* Commands to restore an Executor from a snapshot file
+*/
+message RestoreSnapshotCommand {
+    // snapshot file
+    string snapshot_file = 1;
+}
+
+/**
+* jina gRPC service to trigger a restore at the Executor Runtime.
+*/
+service JinaExecutorRestore {
+    rpc restore (RestoreSnapshotCommand) returns (RestoreSnapshotStatusProto) {
+    }
+}
+
+/**
+* jina gRPC service to trigger a snapshot at the Executor Runtime.
+*/
+service JinaExecutorRestoreProgress {
+    rpc restore_status (RestoreId) returns (RestoreSnapshotStatusProto) {
+    }
+}
diff --git a/jina/proto/docarray_v1/pb/jina_pb2.py b/jina/proto/docarray_v1/pb/jina_pb2.py
index f13b6bd31e3bd..4187395a01696 100644
--- a/jina/proto/docarray_v1/pb/jina_pb2.py
+++ b/jina/proto/docarray_v1/pb/jina_pb2.py
@@ -17,13 +17,14 @@
 import docarray.proto.pb.docarray_pb2 as docarray__pb2
 
 
-DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\njina.proto\x12\x04jina\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x0e\x64ocarray.proto\"\x9f\x01\n\nRouteProto\x12\x10\n\x08\x65xecutor\x18\x01 \x01(\t\x12.\n\nstart_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12!\n\x06status\x18\x04 \x01(\x0b\x32\x11.jina.StatusProto\"\xc3\x01\n\rJinaInfoProto\x12+\n\x04jina\x18\x01 \x03(\x0b\x32\x1d.jina.JinaInfoProto.JinaEntry\x12+\n\x04\x65nvs\x18\x02 \x03(\x0b\x32\x1d.jina.JinaInfoProto.EnvsEntry\x1a+\n\tJinaEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x1a+\n\tEnvsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xc6\x01\n\x0bHeaderProto\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12!\n\x06status\x18\x02 \x01(\x0b\x32\x11.jina.StatusProto\x12\x1a\n\rexec_endpoint\x18\x03 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0ftarget_executor\x18\x04 \x01(\tH\x01\x88\x01\x01\x12\x14\n\x07timeout\x18\x05 \x01(\rH\x02\x88\x01\x01\x42\x10\n\x0e_exec_endpointB\x12\n\x10_target_executorB\n\n\x08_timeout\"#\n\x0e\x45ndpointsProto\x12\x11\n\tendpoints\x18\x01 \x03(\t\"\xf9\x01\n\x0bStatusProto\x12*\n\x04\x63ode\x18\x01 \x01(\x0e\x32\x1c.jina.StatusProto.StatusCode\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x33\n\texception\x18\x03 \x01(\x0b\x32 .jina.StatusProto.ExceptionProto\x1aN\n\x0e\x45xceptionProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04\x61rgs\x18\x02 \x03(\t\x12\x0e\n\x06stacks\x18\x03 \x03(\t\x12\x10\n\x08\x65xecutor\x18\x04 \x01(\t\"$\n\nStatusCode\x12\x0b\n\x07SUCCESS\x10\x00\x12\t\n\x05\x45RROR\x10\x01\"^\n\rRelatedEntity\x12\n\n\x02id\x18\x01 \x01(\t\x12\x0f\n\x07\x61\x64\x64ress\x18\x02 \x01(\t\x12\x0c\n\x04port\x18\x03 \x01(\r\x12\x15\n\x08shard_id\x18\x04 \x01(\rH\x00\x88\x01\x01\x42\x0b\n\t_shard_id\"\xa0\x02\n\x10\x44\x61taRequestProto\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\x12\x35\n\x04\x64\x61ta\x18\x04 \x01(\x0b\x32\'.jina.DataRequestProto.DataContentProto\x1a\x63\n\x10\x44\x61taContentProto\x12,\n\x04\x64ocs\x18\x01 \x01(\x0b\x32\x1c.docarray.DocumentArrayProtoH\x00\x12\x14\n\ndocs_bytes\x18\x02 \x01(\x0cH\x00\x42\x0b\n\tdocuments\"\x8a\x01\n\x16\x44\x61taRequestProtoWoData\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\"@\n\x14\x44\x61taRequestListProto\x12(\n\x08requests\x18\x01 \x03(\x0b\x32\x16.jina.DataRequestProto2Z\n\x12JinaDataRequestRPC\x12\x44\n\x0cprocess_data\x12\x1a.jina.DataRequestListProto\x1a\x16.jina.DataRequestProto\"\x00\x32\x63\n\x18JinaSingleDataRequestRPC\x12G\n\x13process_single_data\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00\x32G\n\x07JinaRPC\x12<\n\x04\x43\x61ll\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00(\x01\x30\x01\x32`\n\x18JinaDiscoverEndpointsRPC\x12\x44\n\x12\x65ndpoint_discovery\x12\x16.google.protobuf.Empty\x1a\x14.jina.EndpointsProto\"\x00\x32N\n\x14JinaGatewayDryRunRPC\x12\x36\n\x07\x64ry_run\x12\x16.google.protobuf.Empty\x1a\x11.jina.StatusProto\"\x00\x32G\n\x0bJinaInfoRPC\x12\x38\n\x07_status\x12\x16.google.protobuf.Empty\x1a\x13.jina.JinaInfoProto\"\x00\x62\x06proto3')
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\njina.proto\x12\x04jina\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x0e\x64ocarray.proto\"\x9f\x01\n\nRouteProto\x12\x10\n\x08\x65xecutor\x18\x01 \x01(\t\x12.\n\nstart_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12!\n\x06status\x18\x04 \x01(\x0b\x32\x11.jina.StatusProto\"\xc3\x01\n\rJinaInfoProto\x12+\n\x04jina\x18\x01 \x03(\x0b\x32\x1d.jina.JinaInfoProto.JinaEntry\x12+\n\x04\x65nvs\x18\x02 \x03(\x0b\x32\x1d.jina.JinaInfoProto.EnvsEntry\x1a+\n\tJinaEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x1a+\n\tEnvsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xc6\x01\n\x0bHeaderProto\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12!\n\x06status\x18\x02 \x01(\x0b\x32\x11.jina.StatusProto\x12\x1a\n\rexec_endpoint\x18\x03 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0ftarget_executor\x18\x04 \x01(\tH\x01\x88\x01\x01\x12\x14\n\x07timeout\x18\x05 \x01(\rH\x02\x88\x01\x01\x42\x10\n\x0e_exec_endpointB\x12\n\x10_target_executorB\n\n\x08_timeout\"<\n\x0e\x45ndpointsProto\x12\x11\n\tendpoints\x18\x01 \x03(\t\x12\x17\n\x0fwrite_endpoints\x18\x02 \x03(\t\"\xf9\x01\n\x0bStatusProto\x12*\n\x04\x63ode\x18\x01 \x01(\x0e\x32\x1c.jina.StatusProto.StatusCode\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x33\n\texception\x18\x03 \x01(\x0b\x32 .jina.StatusProto.ExceptionProto\x1aN\n\x0e\x45xceptionProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04\x61rgs\x18\x02 \x03(\t\x12\x0e\n\x06stacks\x18\x03 \x03(\t\x12\x10\n\x08\x65xecutor\x18\x04 \x01(\t\"$\n\nStatusCode\x12\x0b\n\x07SUCCESS\x10\x00\x12\t\n\x05\x45RROR\x10\x01\"^\n\rRelatedEntity\x12\n\n\x02id\x18\x01 \x01(\t\x12\x0f\n\x07\x61\x64\x64ress\x18\x02 \x01(\t\x12\x0c\n\x04port\x18\x03 \x01(\r\x12\x15\n\x08shard_id\x18\x04 \x01(\rH\x00\x88\x01\x01\x42\x0b\n\t_shard_id\"\xa0\x02\n\x10\x44\x61taRequestProto\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\x12\x35\n\x04\x64\x61ta\x18\x04 \x01(\x0b\x32\'.jina.DataRequestProto.DataContentProto\x1a\x63\n\x10\x44\x61taContentProto\x12,\n\x04\x64ocs\x18\x01 \x01(\x0b\x32\x1c.docarray.DocumentArrayProtoH\x00\x12\x14\n\ndocs_bytes\x18\x02 \x01(\x0cH\x00\x42\x0b\n\tdocuments\"\x8a\x01\n\x16\x44\x61taRequestProtoWoData\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\"@\n\x14\x44\x61taRequestListProto\x12(\n\x08requests\x18\x01 \x03(\x0b\x32\x16.jina.DataRequestProto\"\x1b\n\nSnapshotId\x12\r\n\x05value\x18\x01 \x01(\t\"\x1a\n\tRestoreId\x12\r\n\x05value\x18\x01 \x01(\t\"\xef\x01\n\x13SnapshotStatusProto\x12\x1c\n\x02id\x18\x01 \x01(\x0b\x32\x10.jina.SnapshotId\x12\x30\n\x06status\x18\x02 \x01(\x0e\x32 .jina.SnapshotStatusProto.Status\x12\x15\n\rsnapshot_file\x18\x03 \x01(\t\"q\n\x06Status\x12\x15\n\x11STATE_UNSPECIFIED\x10\x00\x12\n\n\x06QUEUED\x10\x01\x12\r\n\tSCHEDULED\x10\x02\x12\x0b\n\x07RUNNING\x10\x03\x12\r\n\tSUCCEEDED\x10\x04\x12\n\n\x06\x46\x41ILED\x10\x05\x12\r\n\tNOT_FOUND\x10\x06\"\xca\x01\n\x1aRestoreSnapshotStatusProto\x12\x1b\n\x02id\x18\x01 \x01(\x0b\x32\x0f.jina.RestoreId\x12\x37\n\x06status\x18\x02 \x01(\x0e\x32\'.jina.RestoreSnapshotStatusProto.Status\"V\n\x06Status\x12\x15\n\x11STATE_UNSPECIFIED\x10\x00\x12\x0b\n\x07RUNNING\x10\x01\x12\r\n\tSUCCEEDED\x10\x02\x12\n\n\x06\x46\x41ILED\x10\x03\x12\r\n\tNOT_FOUND\x10\x06\"/\n\x16RestoreSnapshotCommand\x12\x15\n\rsnapshot_file\x18\x01 \x01(\t2Z\n\x12JinaDataRequestRPC\x12\x44\n\x0cprocess_data\x12\x1a.jina.DataRequestListProto\x1a\x16.jina.DataRequestProto\"\x00\x32\x63\n\x18JinaSingleDataRequestRPC\x12G\n\x13process_single_data\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00\x32G\n\x07JinaRPC\x12<\n\x04\x43\x61ll\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00(\x01\x30\x01\x32`\n\x18JinaDiscoverEndpointsRPC\x12\x44\n\x12\x65ndpoint_discovery\x12\x16.google.protobuf.Empty\x1a\x14.jina.EndpointsProto\"\x00\x32N\n\x14JinaGatewayDryRunRPC\x12\x36\n\x07\x64ry_run\x12\x16.google.protobuf.Empty\x1a\x11.jina.StatusProto\"\x00\x32G\n\x0bJinaInfoRPC\x12\x38\n\x07_status\x12\x16.google.protobuf.Empty\x1a\x13.jina.JinaInfoProto\"\x00\x32W\n\x14JinaExecutorSnapshot\x12?\n\x08snapshot\x12\x16.google.protobuf.Empty\x1a\x19.jina.SnapshotStatusProto\"\x00\x32`\n\x1cJinaExecutorSnapshotProgress\x12@\n\x0fsnapshot_status\x12\x10.jina.SnapshotId\x1a\x19.jina.SnapshotStatusProto\"\x00\x32\x62\n\x13JinaExecutorRestore\x12K\n\x07restore\x12\x1c.jina.RestoreSnapshotCommand\x1a .jina.RestoreSnapshotStatusProto\"\x00\x32\x64\n\x1bJinaExecutorRestoreProgress\x12\x45\n\x0erestore_status\x12\x0f.jina.RestoreId\x1a .jina.RestoreSnapshotStatusProto\"\x00\x42#Z!github.com/jina-ai/jina-raft/jinab\x06proto3')
 
 _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
 _builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'jina_pb2', globals())
 if _descriptor._USE_C_DESCRIPTORS == False:
 
   DESCRIPTOR._options = None
+  DESCRIPTOR._serialized_options = b'Z!github.com/jina-ai/jina-raft/jina'
   _JINAINFOPROTO_JINAENTRY._options = None
   _JINAINFOPROTO_JINAENTRY._serialized_options = b'8\001'
   _JINAINFOPROTO_ENVSENTRY._options = None
@@ -39,33 +40,55 @@
   _HEADERPROTO._serialized_start=489
   _HEADERPROTO._serialized_end=687
   _ENDPOINTSPROTO._serialized_start=689
-  _ENDPOINTSPROTO._serialized_end=724
-  _STATUSPROTO._serialized_start=727
-  _STATUSPROTO._serialized_end=976
-  _STATUSPROTO_EXCEPTIONPROTO._serialized_start=860
-  _STATUSPROTO_EXCEPTIONPROTO._serialized_end=938
-  _STATUSPROTO_STATUSCODE._serialized_start=940
-  _STATUSPROTO_STATUSCODE._serialized_end=976
-  _RELATEDENTITY._serialized_start=978
-  _RELATEDENTITY._serialized_end=1072
-  _DATAREQUESTPROTO._serialized_start=1075
-  _DATAREQUESTPROTO._serialized_end=1363
-  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_start=1264
-  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_end=1363
-  _DATAREQUESTPROTOWODATA._serialized_start=1366
-  _DATAREQUESTPROTOWODATA._serialized_end=1504
-  _DATAREQUESTLISTPROTO._serialized_start=1506
-  _DATAREQUESTLISTPROTO._serialized_end=1570
-  _JINADATAREQUESTRPC._serialized_start=1572
-  _JINADATAREQUESTRPC._serialized_end=1662
-  _JINASINGLEDATAREQUESTRPC._serialized_start=1664
-  _JINASINGLEDATAREQUESTRPC._serialized_end=1763
-  _JINARPC._serialized_start=1765
-  _JINARPC._serialized_end=1836
-  _JINADISCOVERENDPOINTSRPC._serialized_start=1838
-  _JINADISCOVERENDPOINTSRPC._serialized_end=1934
-  _JINAGATEWAYDRYRUNRPC._serialized_start=1936
-  _JINAGATEWAYDRYRUNRPC._serialized_end=2014
-  _JINAINFORPC._serialized_start=2016
-  _JINAINFORPC._serialized_end=2087
+  _ENDPOINTSPROTO._serialized_end=749
+  _STATUSPROTO._serialized_start=752
+  _STATUSPROTO._serialized_end=1001
+  _STATUSPROTO_EXCEPTIONPROTO._serialized_start=885
+  _STATUSPROTO_EXCEPTIONPROTO._serialized_end=963
+  _STATUSPROTO_STATUSCODE._serialized_start=965
+  _STATUSPROTO_STATUSCODE._serialized_end=1001
+  _RELATEDENTITY._serialized_start=1003
+  _RELATEDENTITY._serialized_end=1097
+  _DATAREQUESTPROTO._serialized_start=1100
+  _DATAREQUESTPROTO._serialized_end=1388
+  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_start=1289
+  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_end=1388
+  _DATAREQUESTPROTOWODATA._serialized_start=1391
+  _DATAREQUESTPROTOWODATA._serialized_end=1529
+  _DATAREQUESTLISTPROTO._serialized_start=1531
+  _DATAREQUESTLISTPROTO._serialized_end=1595
+  _SNAPSHOTID._serialized_start=1597
+  _SNAPSHOTID._serialized_end=1624
+  _RESTOREID._serialized_start=1626
+  _RESTOREID._serialized_end=1652
+  _SNAPSHOTSTATUSPROTO._serialized_start=1655
+  _SNAPSHOTSTATUSPROTO._serialized_end=1894
+  _SNAPSHOTSTATUSPROTO_STATUS._serialized_start=1781
+  _SNAPSHOTSTATUSPROTO_STATUS._serialized_end=1894
+  _RESTORESNAPSHOTSTATUSPROTO._serialized_start=1897
+  _RESTORESNAPSHOTSTATUSPROTO._serialized_end=2099
+  _RESTORESNAPSHOTSTATUSPROTO_STATUS._serialized_start=2013
+  _RESTORESNAPSHOTSTATUSPROTO_STATUS._serialized_end=2099
+  _RESTORESNAPSHOTCOMMAND._serialized_start=2101
+  _RESTORESNAPSHOTCOMMAND._serialized_end=2148
+  _JINADATAREQUESTRPC._serialized_start=2150
+  _JINADATAREQUESTRPC._serialized_end=2240
+  _JINASINGLEDATAREQUESTRPC._serialized_start=2242
+  _JINASINGLEDATAREQUESTRPC._serialized_end=2341
+  _JINARPC._serialized_start=2343
+  _JINARPC._serialized_end=2414
+  _JINADISCOVERENDPOINTSRPC._serialized_start=2416
+  _JINADISCOVERENDPOINTSRPC._serialized_end=2512
+  _JINAGATEWAYDRYRUNRPC._serialized_start=2514
+  _JINAGATEWAYDRYRUNRPC._serialized_end=2592
+  _JINAINFORPC._serialized_start=2594
+  _JINAINFORPC._serialized_end=2665
+  _JINAEXECUTORSNAPSHOT._serialized_start=2667
+  _JINAEXECUTORSNAPSHOT._serialized_end=2754
+  _JINAEXECUTORSNAPSHOTPROGRESS._serialized_start=2756
+  _JINAEXECUTORSNAPSHOTPROGRESS._serialized_end=2852
+  _JINAEXECUTORRESTORE._serialized_start=2854
+  _JINAEXECUTORRESTORE._serialized_end=2952
+  _JINAEXECUTORRESTOREPROGRESS._serialized_start=2954
+  _JINAEXECUTORRESTOREPROGRESS._serialized_end=3054
 # @@protoc_insertion_point(module_scope)
diff --git a/jina/proto/docarray_v1/pb/jina_pb2_grpc.py b/jina/proto/docarray_v1/pb/jina_pb2_grpc.py
index b1fb12b313eb4..2aa864543aee9 100644
--- a/jina/proto/docarray_v1/pb/jina_pb2_grpc.py
+++ b/jina/proto/docarray_v1/pb/jina_pb2_grpc.py
@@ -412,3 +412,271 @@ def _status(request,
             jina__pb2.JinaInfoProto.FromString,
             options, channel_credentials,
             insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorSnapshotStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.snapshot = channel.unary_unary(
+                '/jina.JinaExecutorSnapshot/snapshot',
+                request_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+                response_deserializer=jina__pb2.SnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorSnapshotServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def snapshot(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorSnapshotServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'snapshot': grpc.unary_unary_rpc_method_handler(
+                    servicer.snapshot,
+                    request_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+                    response_serializer=jina__pb2.SnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorSnapshot', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorSnapshot(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def snapshot(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorSnapshot/snapshot',
+            google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+            jina__pb2.SnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorSnapshotProgressStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.snapshot_status = channel.unary_unary(
+                '/jina.JinaExecutorSnapshotProgress/snapshot_status',
+                request_serializer=jina__pb2.SnapshotId.SerializeToString,
+                response_deserializer=jina__pb2.SnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorSnapshotProgressServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def snapshot_status(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorSnapshotProgressServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'snapshot_status': grpc.unary_unary_rpc_method_handler(
+                    servicer.snapshot_status,
+                    request_deserializer=jina__pb2.SnapshotId.FromString,
+                    response_serializer=jina__pb2.SnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorSnapshotProgress', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorSnapshotProgress(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def snapshot_status(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorSnapshotProgress/snapshot_status',
+            jina__pb2.SnapshotId.SerializeToString,
+            jina__pb2.SnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorRestoreStub(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.restore = channel.unary_unary(
+                '/jina.JinaExecutorRestore/restore',
+                request_serializer=jina__pb2.RestoreSnapshotCommand.SerializeToString,
+                response_deserializer=jina__pb2.RestoreSnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorRestoreServicer(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    def restore(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorRestoreServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'restore': grpc.unary_unary_rpc_method_handler(
+                    servicer.restore,
+                    request_deserializer=jina__pb2.RestoreSnapshotCommand.FromString,
+                    response_serializer=jina__pb2.RestoreSnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorRestore', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorRestore(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    @staticmethod
+    def restore(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorRestore/restore',
+            jina__pb2.RestoreSnapshotCommand.SerializeToString,
+            jina__pb2.RestoreSnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorRestoreProgressStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.restore_status = channel.unary_unary(
+                '/jina.JinaExecutorRestoreProgress/restore_status',
+                request_serializer=jina__pb2.RestoreId.SerializeToString,
+                response_deserializer=jina__pb2.RestoreSnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorRestoreProgressServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def restore_status(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorRestoreProgressServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'restore_status': grpc.unary_unary_rpc_method_handler(
+                    servicer.restore_status,
+                    request_deserializer=jina__pb2.RestoreId.FromString,
+                    response_serializer=jina__pb2.RestoreSnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorRestoreProgress', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorRestoreProgress(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def restore_status(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorRestoreProgress/restore_status',
+            jina__pb2.RestoreId.SerializeToString,
+            jina__pb2.RestoreSnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
diff --git a/jina/proto/docarray_v1/pb2/jina_pb2.py b/jina/proto/docarray_v1/pb2/jina_pb2.py
index e7f9c2ff0a7de..c3dfaaeaeb479 100644
--- a/jina/proto/docarray_v1/pb2/jina_pb2.py
+++ b/jina/proto/docarray_v1/pb2/jina_pb2.py
@@ -7,18 +7,18 @@
 from google.protobuf import message as _message
 from google.protobuf import reflection as _reflection
 from google.protobuf import symbol_database as _symbol_database
-
 # @@protoc_insertion_point(imports)
 
 _sym_db = _symbol_database.Default()
 
 
-import docarray.proto.pb2.docarray_pb2 as docarray__pb2
-from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
-from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
 from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+import docarray.proto.pb2.docarray_pb2 as docarray__pb2
+
 
-DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\njina.proto\x12\x04jina\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x0e\x64ocarray.proto\"\x9f\x01\n\nRouteProto\x12\x10\n\x08\x65xecutor\x18\x01 \x01(\t\x12.\n\nstart_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12!\n\x06status\x18\x04 \x01(\x0b\x32\x11.jina.StatusProto\"\xc3\x01\n\rJinaInfoProto\x12+\n\x04jina\x18\x01 \x03(\x0b\x32\x1d.jina.JinaInfoProto.JinaEntry\x12+\n\x04\x65nvs\x18\x02 \x03(\x0b\x32\x1d.jina.JinaInfoProto.EnvsEntry\x1a+\n\tJinaEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x1a+\n\tEnvsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xc6\x01\n\x0bHeaderProto\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12!\n\x06status\x18\x02 \x01(\x0b\x32\x11.jina.StatusProto\x12\x1a\n\rexec_endpoint\x18\x03 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0ftarget_executor\x18\x04 \x01(\tH\x01\x88\x01\x01\x12\x14\n\x07timeout\x18\x05 \x01(\rH\x02\x88\x01\x01\x42\x10\n\x0e_exec_endpointB\x12\n\x10_target_executorB\n\n\x08_timeout\"#\n\x0e\x45ndpointsProto\x12\x11\n\tendpoints\x18\x01 \x03(\t\"\xf9\x01\n\x0bStatusProto\x12*\n\x04\x63ode\x18\x01 \x01(\x0e\x32\x1c.jina.StatusProto.StatusCode\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x33\n\texception\x18\x03 \x01(\x0b\x32 .jina.StatusProto.ExceptionProto\x1aN\n\x0e\x45xceptionProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04\x61rgs\x18\x02 \x03(\t\x12\x0e\n\x06stacks\x18\x03 \x03(\t\x12\x10\n\x08\x65xecutor\x18\x04 \x01(\t\"$\n\nStatusCode\x12\x0b\n\x07SUCCESS\x10\x00\x12\t\n\x05\x45RROR\x10\x01\"^\n\rRelatedEntity\x12\n\n\x02id\x18\x01 \x01(\t\x12\x0f\n\x07\x61\x64\x64ress\x18\x02 \x01(\t\x12\x0c\n\x04port\x18\x03 \x01(\r\x12\x15\n\x08shard_id\x18\x04 \x01(\rH\x00\x88\x01\x01\x42\x0b\n\t_shard_id\"\xa0\x02\n\x10\x44\x61taRequestProto\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\x12\x35\n\x04\x64\x61ta\x18\x04 \x01(\x0b\x32\'.jina.DataRequestProto.DataContentProto\x1a\x63\n\x10\x44\x61taContentProto\x12,\n\x04\x64ocs\x18\x01 \x01(\x0b\x32\x1c.docarray.DocumentArrayProtoH\x00\x12\x14\n\ndocs_bytes\x18\x02 \x01(\x0cH\x00\x42\x0b\n\tdocuments\"\x8a\x01\n\x16\x44\x61taRequestProtoWoData\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\"@\n\x14\x44\x61taRequestListProto\x12(\n\x08requests\x18\x01 \x03(\x0b\x32\x16.jina.DataRequestProto2Z\n\x12JinaDataRequestRPC\x12\x44\n\x0cprocess_data\x12\x1a.jina.DataRequestListProto\x1a\x16.jina.DataRequestProto\"\x00\x32\x63\n\x18JinaSingleDataRequestRPC\x12G\n\x13process_single_data\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00\x32G\n\x07JinaRPC\x12<\n\x04\x43\x61ll\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00(\x01\x30\x01\x32`\n\x18JinaDiscoverEndpointsRPC\x12\x44\n\x12\x65ndpoint_discovery\x12\x16.google.protobuf.Empty\x1a\x14.jina.EndpointsProto\"\x00\x32N\n\x14JinaGatewayDryRunRPC\x12\x36\n\x07\x64ry_run\x12\x16.google.protobuf.Empty\x1a\x11.jina.StatusProto\"\x00\x32G\n\x0bJinaInfoRPC\x12\x38\n\x07_status\x12\x16.google.protobuf.Empty\x1a\x13.jina.JinaInfoProto\"\x00\x62\x06proto3')
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\njina.proto\x12\x04jina\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x0e\x64ocarray.proto\"\x9f\x01\n\nRouteProto\x12\x10\n\x08\x65xecutor\x18\x01 \x01(\t\x12.\n\nstart_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12!\n\x06status\x18\x04 \x01(\x0b\x32\x11.jina.StatusProto\"\xc3\x01\n\rJinaInfoProto\x12+\n\x04jina\x18\x01 \x03(\x0b\x32\x1d.jina.JinaInfoProto.JinaEntry\x12+\n\x04\x65nvs\x18\x02 \x03(\x0b\x32\x1d.jina.JinaInfoProto.EnvsEntry\x1a+\n\tJinaEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x1a+\n\tEnvsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xc6\x01\n\x0bHeaderProto\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12!\n\x06status\x18\x02 \x01(\x0b\x32\x11.jina.StatusProto\x12\x1a\n\rexec_endpoint\x18\x03 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0ftarget_executor\x18\x04 \x01(\tH\x01\x88\x01\x01\x12\x14\n\x07timeout\x18\x05 \x01(\rH\x02\x88\x01\x01\x42\x10\n\x0e_exec_endpointB\x12\n\x10_target_executorB\n\n\x08_timeout\"<\n\x0e\x45ndpointsProto\x12\x11\n\tendpoints\x18\x01 \x03(\t\x12\x17\n\x0fwrite_endpoints\x18\x02 \x03(\t\"\xf9\x01\n\x0bStatusProto\x12*\n\x04\x63ode\x18\x01 \x01(\x0e\x32\x1c.jina.StatusProto.StatusCode\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x33\n\texception\x18\x03 \x01(\x0b\x32 .jina.StatusProto.ExceptionProto\x1aN\n\x0e\x45xceptionProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04\x61rgs\x18\x02 \x03(\t\x12\x0e\n\x06stacks\x18\x03 \x03(\t\x12\x10\n\x08\x65xecutor\x18\x04 \x01(\t\"$\n\nStatusCode\x12\x0b\n\x07SUCCESS\x10\x00\x12\t\n\x05\x45RROR\x10\x01\"^\n\rRelatedEntity\x12\n\n\x02id\x18\x01 \x01(\t\x12\x0f\n\x07\x61\x64\x64ress\x18\x02 \x01(\t\x12\x0c\n\x04port\x18\x03 \x01(\r\x12\x15\n\x08shard_id\x18\x04 \x01(\rH\x00\x88\x01\x01\x42\x0b\n\t_shard_id\"\xa0\x02\n\x10\x44\x61taRequestProto\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\x12\x35\n\x04\x64\x61ta\x18\x04 \x01(\x0b\x32\'.jina.DataRequestProto.DataContentProto\x1a\x63\n\x10\x44\x61taContentProto\x12,\n\x04\x64ocs\x18\x01 \x01(\x0b\x32\x1c.docarray.DocumentArrayProtoH\x00\x12\x14\n\ndocs_bytes\x18\x02 \x01(\x0cH\x00\x42\x0b\n\tdocuments\"\x8a\x01\n\x16\x44\x61taRequestProtoWoData\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\"@\n\x14\x44\x61taRequestListProto\x12(\n\x08requests\x18\x01 \x03(\x0b\x32\x16.jina.DataRequestProto\"\x1b\n\nSnapshotId\x12\r\n\x05value\x18\x01 \x01(\t\"\x1a\n\tRestoreId\x12\r\n\x05value\x18\x01 \x01(\t\"\xef\x01\n\x13SnapshotStatusProto\x12\x1c\n\x02id\x18\x01 \x01(\x0b\x32\x10.jina.SnapshotId\x12\x30\n\x06status\x18\x02 \x01(\x0e\x32 .jina.SnapshotStatusProto.Status\x12\x15\n\rsnapshot_file\x18\x03 \x01(\t\"q\n\x06Status\x12\x15\n\x11STATE_UNSPECIFIED\x10\x00\x12\n\n\x06QUEUED\x10\x01\x12\r\n\tSCHEDULED\x10\x02\x12\x0b\n\x07RUNNING\x10\x03\x12\r\n\tSUCCEEDED\x10\x04\x12\n\n\x06\x46\x41ILED\x10\x05\x12\r\n\tNOT_FOUND\x10\x06\"\xca\x01\n\x1aRestoreSnapshotStatusProto\x12\x1b\n\x02id\x18\x01 \x01(\x0b\x32\x0f.jina.RestoreId\x12\x37\n\x06status\x18\x02 \x01(\x0e\x32\'.jina.RestoreSnapshotStatusProto.Status\"V\n\x06Status\x12\x15\n\x11STATE_UNSPECIFIED\x10\x00\x12\x0b\n\x07RUNNING\x10\x01\x12\r\n\tSUCCEEDED\x10\x02\x12\n\n\x06\x46\x41ILED\x10\x03\x12\r\n\tNOT_FOUND\x10\x06\"/\n\x16RestoreSnapshotCommand\x12\x15\n\rsnapshot_file\x18\x01 \x01(\t2Z\n\x12JinaDataRequestRPC\x12\x44\n\x0cprocess_data\x12\x1a.jina.DataRequestListProto\x1a\x16.jina.DataRequestProto\"\x00\x32\x63\n\x18JinaSingleDataRequestRPC\x12G\n\x13process_single_data\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00\x32G\n\x07JinaRPC\x12<\n\x04\x43\x61ll\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00(\x01\x30\x01\x32`\n\x18JinaDiscoverEndpointsRPC\x12\x44\n\x12\x65ndpoint_discovery\x12\x16.google.protobuf.Empty\x1a\x14.jina.EndpointsProto\"\x00\x32N\n\x14JinaGatewayDryRunRPC\x12\x36\n\x07\x64ry_run\x12\x16.google.protobuf.Empty\x1a\x11.jina.StatusProto\"\x00\x32G\n\x0bJinaInfoRPC\x12\x38\n\x07_status\x12\x16.google.protobuf.Empty\x1a\x13.jina.JinaInfoProto\"\x00\x32W\n\x14JinaExecutorSnapshot\x12?\n\x08snapshot\x12\x16.google.protobuf.Empty\x1a\x19.jina.SnapshotStatusProto\"\x00\x32`\n\x1cJinaExecutorSnapshotProgress\x12@\n\x0fsnapshot_status\x12\x10.jina.SnapshotId\x1a\x19.jina.SnapshotStatusProto\"\x00\x32\x62\n\x13JinaExecutorRestore\x12K\n\x07restore\x12\x1c.jina.RestoreSnapshotCommand\x1a .jina.RestoreSnapshotStatusProto\"\x00\x32\x64\n\x1bJinaExecutorRestoreProgress\x12\x45\n\x0erestore_status\x12\x0f.jina.RestoreId\x1a .jina.RestoreSnapshotStatusProto\"\x00\x42#Z!github.com/jina-ai/jina-raft/jinab\x06proto3')
 
 
 
@@ -35,7 +35,14 @@
 _DATAREQUESTPROTO_DATACONTENTPROTO = _DATAREQUESTPROTO.nested_types_by_name['DataContentProto']
 _DATAREQUESTPROTOWODATA = DESCRIPTOR.message_types_by_name['DataRequestProtoWoData']
 _DATAREQUESTLISTPROTO = DESCRIPTOR.message_types_by_name['DataRequestListProto']
+_SNAPSHOTID = DESCRIPTOR.message_types_by_name['SnapshotId']
+_RESTOREID = DESCRIPTOR.message_types_by_name['RestoreId']
+_SNAPSHOTSTATUSPROTO = DESCRIPTOR.message_types_by_name['SnapshotStatusProto']
+_RESTORESNAPSHOTSTATUSPROTO = DESCRIPTOR.message_types_by_name['RestoreSnapshotStatusProto']
+_RESTORESNAPSHOTCOMMAND = DESCRIPTOR.message_types_by_name['RestoreSnapshotCommand']
 _STATUSPROTO_STATUSCODE = _STATUSPROTO.enum_types_by_name['StatusCode']
+_SNAPSHOTSTATUSPROTO_STATUS = _SNAPSHOTSTATUSPROTO.enum_types_by_name['Status']
+_RESTORESNAPSHOTSTATUSPROTO_STATUS = _RESTORESNAPSHOTSTATUSPROTO.enum_types_by_name['Status']
 RouteProto = _reflection.GeneratedProtocolMessageType('RouteProto', (_message.Message,), {
   'DESCRIPTOR' : _ROUTEPROTO,
   '__module__' : 'jina_pb2'
@@ -131,15 +138,55 @@
   })
 _sym_db.RegisterMessage(DataRequestListProto)
 
+SnapshotId = _reflection.GeneratedProtocolMessageType('SnapshotId', (_message.Message,), {
+  'DESCRIPTOR' : _SNAPSHOTID,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.SnapshotId)
+  })
+_sym_db.RegisterMessage(SnapshotId)
+
+RestoreId = _reflection.GeneratedProtocolMessageType('RestoreId', (_message.Message,), {
+  'DESCRIPTOR' : _RESTOREID,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.RestoreId)
+  })
+_sym_db.RegisterMessage(RestoreId)
+
+SnapshotStatusProto = _reflection.GeneratedProtocolMessageType('SnapshotStatusProto', (_message.Message,), {
+  'DESCRIPTOR' : _SNAPSHOTSTATUSPROTO,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.SnapshotStatusProto)
+  })
+_sym_db.RegisterMessage(SnapshotStatusProto)
+
+RestoreSnapshotStatusProto = _reflection.GeneratedProtocolMessageType('RestoreSnapshotStatusProto', (_message.Message,), {
+  'DESCRIPTOR' : _RESTORESNAPSHOTSTATUSPROTO,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.RestoreSnapshotStatusProto)
+  })
+_sym_db.RegisterMessage(RestoreSnapshotStatusProto)
+
+RestoreSnapshotCommand = _reflection.GeneratedProtocolMessageType('RestoreSnapshotCommand', (_message.Message,), {
+  'DESCRIPTOR' : _RESTORESNAPSHOTCOMMAND,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.RestoreSnapshotCommand)
+  })
+_sym_db.RegisterMessage(RestoreSnapshotCommand)
+
 _JINADATAREQUESTRPC = DESCRIPTOR.services_by_name['JinaDataRequestRPC']
 _JINASINGLEDATAREQUESTRPC = DESCRIPTOR.services_by_name['JinaSingleDataRequestRPC']
 _JINARPC = DESCRIPTOR.services_by_name['JinaRPC']
 _JINADISCOVERENDPOINTSRPC = DESCRIPTOR.services_by_name['JinaDiscoverEndpointsRPC']
 _JINAGATEWAYDRYRUNRPC = DESCRIPTOR.services_by_name['JinaGatewayDryRunRPC']
 _JINAINFORPC = DESCRIPTOR.services_by_name['JinaInfoRPC']
+_JINAEXECUTORSNAPSHOT = DESCRIPTOR.services_by_name['JinaExecutorSnapshot']
+_JINAEXECUTORSNAPSHOTPROGRESS = DESCRIPTOR.services_by_name['JinaExecutorSnapshotProgress']
+_JINAEXECUTORRESTORE = DESCRIPTOR.services_by_name['JinaExecutorRestore']
+_JINAEXECUTORRESTOREPROGRESS = DESCRIPTOR.services_by_name['JinaExecutorRestoreProgress']
 if _descriptor._USE_C_DESCRIPTORS == False:
 
   DESCRIPTOR._options = None
+  DESCRIPTOR._serialized_options = b'Z!github.com/jina-ai/jina-raft/jina'
   _JINAINFOPROTO_JINAENTRY._options = None
   _JINAINFOPROTO_JINAENTRY._serialized_options = b'8\001'
   _JINAINFOPROTO_ENVSENTRY._options = None
@@ -155,33 +202,55 @@
   _HEADERPROTO._serialized_start=489
   _HEADERPROTO._serialized_end=687
   _ENDPOINTSPROTO._serialized_start=689
-  _ENDPOINTSPROTO._serialized_end=724
-  _STATUSPROTO._serialized_start=727
-  _STATUSPROTO._serialized_end=976
-  _STATUSPROTO_EXCEPTIONPROTO._serialized_start=860
-  _STATUSPROTO_EXCEPTIONPROTO._serialized_end=938
-  _STATUSPROTO_STATUSCODE._serialized_start=940
-  _STATUSPROTO_STATUSCODE._serialized_end=976
-  _RELATEDENTITY._serialized_start=978
-  _RELATEDENTITY._serialized_end=1072
-  _DATAREQUESTPROTO._serialized_start=1075
-  _DATAREQUESTPROTO._serialized_end=1363
-  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_start=1264
-  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_end=1363
-  _DATAREQUESTPROTOWODATA._serialized_start=1366
-  _DATAREQUESTPROTOWODATA._serialized_end=1504
-  _DATAREQUESTLISTPROTO._serialized_start=1506
-  _DATAREQUESTLISTPROTO._serialized_end=1570
-  _JINADATAREQUESTRPC._serialized_start=1572
-  _JINADATAREQUESTRPC._serialized_end=1662
-  _JINASINGLEDATAREQUESTRPC._serialized_start=1664
-  _JINASINGLEDATAREQUESTRPC._serialized_end=1763
-  _JINARPC._serialized_start=1765
-  _JINARPC._serialized_end=1836
-  _JINADISCOVERENDPOINTSRPC._serialized_start=1838
-  _JINADISCOVERENDPOINTSRPC._serialized_end=1934
-  _JINAGATEWAYDRYRUNRPC._serialized_start=1936
-  _JINAGATEWAYDRYRUNRPC._serialized_end=2014
-  _JINAINFORPC._serialized_start=2016
-  _JINAINFORPC._serialized_end=2087
+  _ENDPOINTSPROTO._serialized_end=749
+  _STATUSPROTO._serialized_start=752
+  _STATUSPROTO._serialized_end=1001
+  _STATUSPROTO_EXCEPTIONPROTO._serialized_start=885
+  _STATUSPROTO_EXCEPTIONPROTO._serialized_end=963
+  _STATUSPROTO_STATUSCODE._serialized_start=965
+  _STATUSPROTO_STATUSCODE._serialized_end=1001
+  _RELATEDENTITY._serialized_start=1003
+  _RELATEDENTITY._serialized_end=1097
+  _DATAREQUESTPROTO._serialized_start=1100
+  _DATAREQUESTPROTO._serialized_end=1388
+  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_start=1289
+  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_end=1388
+  _DATAREQUESTPROTOWODATA._serialized_start=1391
+  _DATAREQUESTPROTOWODATA._serialized_end=1529
+  _DATAREQUESTLISTPROTO._serialized_start=1531
+  _DATAREQUESTLISTPROTO._serialized_end=1595
+  _SNAPSHOTID._serialized_start=1597
+  _SNAPSHOTID._serialized_end=1624
+  _RESTOREID._serialized_start=1626
+  _RESTOREID._serialized_end=1652
+  _SNAPSHOTSTATUSPROTO._serialized_start=1655
+  _SNAPSHOTSTATUSPROTO._serialized_end=1894
+  _SNAPSHOTSTATUSPROTO_STATUS._serialized_start=1781
+  _SNAPSHOTSTATUSPROTO_STATUS._serialized_end=1894
+  _RESTORESNAPSHOTSTATUSPROTO._serialized_start=1897
+  _RESTORESNAPSHOTSTATUSPROTO._serialized_end=2099
+  _RESTORESNAPSHOTSTATUSPROTO_STATUS._serialized_start=2013
+  _RESTORESNAPSHOTSTATUSPROTO_STATUS._serialized_end=2099
+  _RESTORESNAPSHOTCOMMAND._serialized_start=2101
+  _RESTORESNAPSHOTCOMMAND._serialized_end=2148
+  _JINADATAREQUESTRPC._serialized_start=2150
+  _JINADATAREQUESTRPC._serialized_end=2240
+  _JINASINGLEDATAREQUESTRPC._serialized_start=2242
+  _JINASINGLEDATAREQUESTRPC._serialized_end=2341
+  _JINARPC._serialized_start=2343
+  _JINARPC._serialized_end=2414
+  _JINADISCOVERENDPOINTSRPC._serialized_start=2416
+  _JINADISCOVERENDPOINTSRPC._serialized_end=2512
+  _JINAGATEWAYDRYRUNRPC._serialized_start=2514
+  _JINAGATEWAYDRYRUNRPC._serialized_end=2592
+  _JINAINFORPC._serialized_start=2594
+  _JINAINFORPC._serialized_end=2665
+  _JINAEXECUTORSNAPSHOT._serialized_start=2667
+  _JINAEXECUTORSNAPSHOT._serialized_end=2754
+  _JINAEXECUTORSNAPSHOTPROGRESS._serialized_start=2756
+  _JINAEXECUTORSNAPSHOTPROGRESS._serialized_end=2852
+  _JINAEXECUTORRESTORE._serialized_start=2854
+  _JINAEXECUTORRESTORE._serialized_end=2952
+  _JINAEXECUTORRESTOREPROGRESS._serialized_start=2954
+  _JINAEXECUTORRESTOREPROGRESS._serialized_end=3054
 # @@protoc_insertion_point(module_scope)
diff --git a/jina/proto/docarray_v1/pb2/jina_pb2_grpc.py b/jina/proto/docarray_v1/pb2/jina_pb2_grpc.py
index 9f25df0c8d7ed..2aa864543aee9 100644
--- a/jina/proto/docarray_v1/pb2/jina_pb2_grpc.py
+++ b/jina/proto/docarray_v1/pb2/jina_pb2_grpc.py
@@ -1,8 +1,8 @@
 # Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
 """Client and server classes corresponding to protobuf-defined services."""
 import grpc
-from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
 
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
 from ... import serializer as jina__pb2
 
 
@@ -412,3 +412,271 @@ def _status(request,
             jina__pb2.JinaInfoProto.FromString,
             options, channel_credentials,
             insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorSnapshotStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.snapshot = channel.unary_unary(
+                '/jina.JinaExecutorSnapshot/snapshot',
+                request_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+                response_deserializer=jina__pb2.SnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorSnapshotServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def snapshot(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorSnapshotServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'snapshot': grpc.unary_unary_rpc_method_handler(
+                    servicer.snapshot,
+                    request_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+                    response_serializer=jina__pb2.SnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorSnapshot', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorSnapshot(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def snapshot(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorSnapshot/snapshot',
+            google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+            jina__pb2.SnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorSnapshotProgressStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.snapshot_status = channel.unary_unary(
+                '/jina.JinaExecutorSnapshotProgress/snapshot_status',
+                request_serializer=jina__pb2.SnapshotId.SerializeToString,
+                response_deserializer=jina__pb2.SnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorSnapshotProgressServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def snapshot_status(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorSnapshotProgressServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'snapshot_status': grpc.unary_unary_rpc_method_handler(
+                    servicer.snapshot_status,
+                    request_deserializer=jina__pb2.SnapshotId.FromString,
+                    response_serializer=jina__pb2.SnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorSnapshotProgress', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorSnapshotProgress(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def snapshot_status(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorSnapshotProgress/snapshot_status',
+            jina__pb2.SnapshotId.SerializeToString,
+            jina__pb2.SnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorRestoreStub(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.restore = channel.unary_unary(
+                '/jina.JinaExecutorRestore/restore',
+                request_serializer=jina__pb2.RestoreSnapshotCommand.SerializeToString,
+                response_deserializer=jina__pb2.RestoreSnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorRestoreServicer(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    def restore(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorRestoreServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'restore': grpc.unary_unary_rpc_method_handler(
+                    servicer.restore,
+                    request_deserializer=jina__pb2.RestoreSnapshotCommand.FromString,
+                    response_serializer=jina__pb2.RestoreSnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorRestore', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorRestore(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    @staticmethod
+    def restore(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorRestore/restore',
+            jina__pb2.RestoreSnapshotCommand.SerializeToString,
+            jina__pb2.RestoreSnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorRestoreProgressStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.restore_status = channel.unary_unary(
+                '/jina.JinaExecutorRestoreProgress/restore_status',
+                request_serializer=jina__pb2.RestoreId.SerializeToString,
+                response_deserializer=jina__pb2.RestoreSnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorRestoreProgressServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def restore_status(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorRestoreProgressServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'restore_status': grpc.unary_unary_rpc_method_handler(
+                    servicer.restore_status,
+                    request_deserializer=jina__pb2.RestoreId.FromString,
+                    response_serializer=jina__pb2.RestoreSnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorRestoreProgress', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorRestoreProgress(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def restore_status(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorRestoreProgress/restore_status',
+            jina__pb2.RestoreId.SerializeToString,
+            jina__pb2.RestoreSnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
diff --git a/jina/proto/docarray_v2/jina.proto b/jina/proto/docarray_v2/jina.proto
index 4f6ad144e0aae..536460db8b5bd 100644
--- a/jina/proto/docarray_v2/jina.proto
+++ b/jina/proto/docarray_v2/jina.proto
@@ -52,6 +52,7 @@ message EndpointsProto {
 
     // list of endpoints exposed by an Executor
     repeated string endpoints = 1;
+    repeated string write_endpoints = 2;
 }
 
 /**
@@ -193,3 +194,110 @@ service JinaInfoRPC {
     }
 }
 
+/**
+* An ID representation for the Snapshot Job
+*/
+message SnapshotId {
+    string value = 1;
+}
+
+/**
+* An ID representation for the Restore Job
+*/
+message RestoreId {
+    string value = 1;
+}
+
+
+/**
+* Represents the status of a Snapshot.
+*/
+message SnapshotStatusProto {
+
+    SnapshotId id = 1; // An ID for the Snapshot Job.
+
+    enum Status {
+        STATE_UNSPECIFIED = 0;
+        
+        QUEUED = 1; // Snapshot Job is admitted (validated and persisted) and waiting for resources.
+        
+        SCHEDULED = 2; // Snapshot Job is scheduled to run as soon as resource allocation is ready.
+        
+        RUNNING = 3; // Resource allocation has been successful. Snapshot Job is RUNNING.
+        
+        SUCCEEDED = 4; // Snapshot Job has finished successfully.
+        
+        FAILED = 5; // Snapshot Job has failed.
+
+        NOT_FOUND = 6; // If the specific snapshot id is unknown or not found
+    }
+
+    // status code
+    Status status = 2;
+    
+    // snapshot file
+    string snapshot_file = 3;
+}
+
+/**
+* jina gRPC service to trigger a snapshot at the Executor Runtime.
+*/
+service JinaExecutorSnapshot {
+    rpc snapshot (google.protobuf.Empty) returns (SnapshotStatusProto) {
+    }
+}
+
+/**
+* jina gRPC service to trigger a snapshot at the Executor Runtime.
+*/
+service JinaExecutorSnapshotProgress {
+    rpc snapshot_status (SnapshotId) returns (SnapshotStatusProto) {
+    }
+}
+
+/**
+* Represents the status of a Restore.
+*/
+message RestoreSnapshotStatusProto {
+
+    RestoreId id = 1; // An ID for the Restore Job.
+
+    enum Status {
+        STATE_UNSPECIFIED = 0;
+
+        RUNNING = 1; // Resource allocation has been successful. Restore Job is RUNNING.
+
+        SUCCEEDED = 2; // Resource Job has finished successfully.
+
+        FAILED = 3; // Resource Job has failed.
+
+        NOT_FOUND = 6; // If the specific restore id is unknown or not found
+    }
+
+    // status code
+    Status status = 2;
+}
+
+/**
+* Commands to restore an Executor from a snapshot file
+*/
+message RestoreSnapshotCommand {
+    // snapshot file
+    string snapshot_file = 1;
+}
+
+/**
+* jina gRPC service to trigger a restore at the Executor Runtime.
+*/
+service JinaExecutorRestore {
+    rpc restore (RestoreSnapshotCommand) returns (RestoreSnapshotStatusProto) {
+    }
+}
+
+/**
+* jina gRPC service to trigger a snapshot at the Executor Runtime.
+*/
+service JinaExecutorRestoreProgress {
+    rpc restore_status (RestoreId) returns (RestoreSnapshotStatusProto) {
+    }
+}
diff --git a/jina/proto/docarray_v2/pb/jina_pb2.py b/jina/proto/docarray_v2/pb/jina_pb2.py
index d4ec187acb8d0..6e9cd330212bf 100644
--- a/jina/proto/docarray_v2/pb/jina_pb2.py
+++ b/jina/proto/docarray_v2/pb/jina_pb2.py
@@ -17,13 +17,14 @@
 import docarray.proto.pb.docarray_pb2 as docarray__pb2
 
 
-DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\njina.proto\x12\x04jina\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x0e\x64ocarray.proto\"\x9f\x01\n\nRouteProto\x12\x10\n\x08\x65xecutor\x18\x01 \x01(\t\x12.\n\nstart_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12!\n\x06status\x18\x04 \x01(\x0b\x32\x11.jina.StatusProto\"\xc3\x01\n\rJinaInfoProto\x12+\n\x04jina\x18\x01 \x03(\x0b\x32\x1d.jina.JinaInfoProto.JinaEntry\x12+\n\x04\x65nvs\x18\x02 \x03(\x0b\x32\x1d.jina.JinaInfoProto.EnvsEntry\x1a+\n\tJinaEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x1a+\n\tEnvsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xc6\x01\n\x0bHeaderProto\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12!\n\x06status\x18\x02 \x01(\x0b\x32\x11.jina.StatusProto\x12\x1a\n\rexec_endpoint\x18\x03 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0ftarget_executor\x18\x04 \x01(\tH\x01\x88\x01\x01\x12\x14\n\x07timeout\x18\x05 \x01(\rH\x02\x88\x01\x01\x42\x10\n\x0e_exec_endpointB\x12\n\x10_target_executorB\n\n\x08_timeout\"#\n\x0e\x45ndpointsProto\x12\x11\n\tendpoints\x18\x01 \x03(\t\"\xf9\x01\n\x0bStatusProto\x12*\n\x04\x63ode\x18\x01 \x01(\x0e\x32\x1c.jina.StatusProto.StatusCode\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x33\n\texception\x18\x03 \x01(\x0b\x32 .jina.StatusProto.ExceptionProto\x1aN\n\x0e\x45xceptionProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04\x61rgs\x18\x02 \x03(\t\x12\x0e\n\x06stacks\x18\x03 \x03(\t\x12\x10\n\x08\x65xecutor\x18\x04 \x01(\t\"$\n\nStatusCode\x12\x0b\n\x07SUCCESS\x10\x00\x12\t\n\x05\x45RROR\x10\x01\"^\n\rRelatedEntity\x12\n\n\x02id\x18\x01 \x01(\t\x12\x0f\n\x07\x61\x64\x64ress\x18\x02 \x01(\t\x12\x0c\n\x04port\x18\x03 \x01(\r\x12\x15\n\x08shard_id\x18\x04 \x01(\rH\x00\x88\x01\x01\x42\x0b\n\t_shard_id\"\x9a\x02\n\x10\x44\x61taRequestProto\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\x12\x35\n\x04\x64\x61ta\x18\x04 \x01(\x0b\x32\'.jina.DataRequestProto.DataContentProto\x1a]\n\x10\x44\x61taContentProto\x12&\n\x04\x64ocs\x18\x01 \x01(\x0b\x32\x16.docarray.DocListProtoH\x00\x12\x14\n\ndocs_bytes\x18\x02 \x01(\x0cH\x00\x42\x0b\n\tdocuments\"\x8a\x01\n\x16\x44\x61taRequestProtoWoData\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\"@\n\x14\x44\x61taRequestListProto\x12(\n\x08requests\x18\x01 \x03(\x0b\x32\x16.jina.DataRequestProto2Z\n\x12JinaDataRequestRPC\x12\x44\n\x0cprocess_data\x12\x1a.jina.DataRequestListProto\x1a\x16.jina.DataRequestProto\"\x00\x32\x63\n\x18JinaSingleDataRequestRPC\x12G\n\x13process_single_data\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00\x32G\n\x07JinaRPC\x12<\n\x04\x43\x61ll\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00(\x01\x30\x01\x32`\n\x18JinaDiscoverEndpointsRPC\x12\x44\n\x12\x65ndpoint_discovery\x12\x16.google.protobuf.Empty\x1a\x14.jina.EndpointsProto\"\x00\x32N\n\x14JinaGatewayDryRunRPC\x12\x36\n\x07\x64ry_run\x12\x16.google.protobuf.Empty\x1a\x11.jina.StatusProto\"\x00\x32G\n\x0bJinaInfoRPC\x12\x38\n\x07_status\x12\x16.google.protobuf.Empty\x1a\x13.jina.JinaInfoProto\"\x00\x62\x06proto3')
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\njina.proto\x12\x04jina\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x0e\x64ocarray.proto\"\x9f\x01\n\nRouteProto\x12\x10\n\x08\x65xecutor\x18\x01 \x01(\t\x12.\n\nstart_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12!\n\x06status\x18\x04 \x01(\x0b\x32\x11.jina.StatusProto\"\xc3\x01\n\rJinaInfoProto\x12+\n\x04jina\x18\x01 \x03(\x0b\x32\x1d.jina.JinaInfoProto.JinaEntry\x12+\n\x04\x65nvs\x18\x02 \x03(\x0b\x32\x1d.jina.JinaInfoProto.EnvsEntry\x1a+\n\tJinaEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x1a+\n\tEnvsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xc6\x01\n\x0bHeaderProto\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12!\n\x06status\x18\x02 \x01(\x0b\x32\x11.jina.StatusProto\x12\x1a\n\rexec_endpoint\x18\x03 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0ftarget_executor\x18\x04 \x01(\tH\x01\x88\x01\x01\x12\x14\n\x07timeout\x18\x05 \x01(\rH\x02\x88\x01\x01\x42\x10\n\x0e_exec_endpointB\x12\n\x10_target_executorB\n\n\x08_timeout\"<\n\x0e\x45ndpointsProto\x12\x11\n\tendpoints\x18\x01 \x03(\t\x12\x17\n\x0fwrite_endpoints\x18\x02 \x03(\t\"\xf9\x01\n\x0bStatusProto\x12*\n\x04\x63ode\x18\x01 \x01(\x0e\x32\x1c.jina.StatusProto.StatusCode\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x33\n\texception\x18\x03 \x01(\x0b\x32 .jina.StatusProto.ExceptionProto\x1aN\n\x0e\x45xceptionProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04\x61rgs\x18\x02 \x03(\t\x12\x0e\n\x06stacks\x18\x03 \x03(\t\x12\x10\n\x08\x65xecutor\x18\x04 \x01(\t\"$\n\nStatusCode\x12\x0b\n\x07SUCCESS\x10\x00\x12\t\n\x05\x45RROR\x10\x01\"^\n\rRelatedEntity\x12\n\n\x02id\x18\x01 \x01(\t\x12\x0f\n\x07\x61\x64\x64ress\x18\x02 \x01(\t\x12\x0c\n\x04port\x18\x03 \x01(\r\x12\x15\n\x08shard_id\x18\x04 \x01(\rH\x00\x88\x01\x01\x42\x0b\n\t_shard_id\"\x9a\x02\n\x10\x44\x61taRequestProto\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\x12\x35\n\x04\x64\x61ta\x18\x04 \x01(\x0b\x32\'.jina.DataRequestProto.DataContentProto\x1a]\n\x10\x44\x61taContentProto\x12&\n\x04\x64ocs\x18\x01 \x01(\x0b\x32\x16.docarray.DocListProtoH\x00\x12\x14\n\ndocs_bytes\x18\x02 \x01(\x0cH\x00\x42\x0b\n\tdocuments\"\x8a\x01\n\x16\x44\x61taRequestProtoWoData\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\"@\n\x14\x44\x61taRequestListProto\x12(\n\x08requests\x18\x01 \x03(\x0b\x32\x16.jina.DataRequestProto\"\x1b\n\nSnapshotId\x12\r\n\x05value\x18\x01 \x01(\t\"\x1a\n\tRestoreId\x12\r\n\x05value\x18\x01 \x01(\t\"\xef\x01\n\x13SnapshotStatusProto\x12\x1c\n\x02id\x18\x01 \x01(\x0b\x32\x10.jina.SnapshotId\x12\x30\n\x06status\x18\x02 \x01(\x0e\x32 .jina.SnapshotStatusProto.Status\x12\x15\n\rsnapshot_file\x18\x03 \x01(\t\"q\n\x06Status\x12\x15\n\x11STATE_UNSPECIFIED\x10\x00\x12\n\n\x06QUEUED\x10\x01\x12\r\n\tSCHEDULED\x10\x02\x12\x0b\n\x07RUNNING\x10\x03\x12\r\n\tSUCCEEDED\x10\x04\x12\n\n\x06\x46\x41ILED\x10\x05\x12\r\n\tNOT_FOUND\x10\x06\"\xca\x01\n\x1aRestoreSnapshotStatusProto\x12\x1b\n\x02id\x18\x01 \x01(\x0b\x32\x0f.jina.RestoreId\x12\x37\n\x06status\x18\x02 \x01(\x0e\x32\'.jina.RestoreSnapshotStatusProto.Status\"V\n\x06Status\x12\x15\n\x11STATE_UNSPECIFIED\x10\x00\x12\x0b\n\x07RUNNING\x10\x01\x12\r\n\tSUCCEEDED\x10\x02\x12\n\n\x06\x46\x41ILED\x10\x03\x12\r\n\tNOT_FOUND\x10\x06\"/\n\x16RestoreSnapshotCommand\x12\x15\n\rsnapshot_file\x18\x01 \x01(\t2Z\n\x12JinaDataRequestRPC\x12\x44\n\x0cprocess_data\x12\x1a.jina.DataRequestListProto\x1a\x16.jina.DataRequestProto\"\x00\x32\x63\n\x18JinaSingleDataRequestRPC\x12G\n\x13process_single_data\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00\x32G\n\x07JinaRPC\x12<\n\x04\x43\x61ll\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00(\x01\x30\x01\x32`\n\x18JinaDiscoverEndpointsRPC\x12\x44\n\x12\x65ndpoint_discovery\x12\x16.google.protobuf.Empty\x1a\x14.jina.EndpointsProto\"\x00\x32N\n\x14JinaGatewayDryRunRPC\x12\x36\n\x07\x64ry_run\x12\x16.google.protobuf.Empty\x1a\x11.jina.StatusProto\"\x00\x32G\n\x0bJinaInfoRPC\x12\x38\n\x07_status\x12\x16.google.protobuf.Empty\x1a\x13.jina.JinaInfoProto\"\x00\x32W\n\x14JinaExecutorSnapshot\x12?\n\x08snapshot\x12\x16.google.protobuf.Empty\x1a\x19.jina.SnapshotStatusProto\"\x00\x32`\n\x1cJinaExecutorSnapshotProgress\x12@\n\x0fsnapshot_status\x12\x10.jina.SnapshotId\x1a\x19.jina.SnapshotStatusProto\"\x00\x32\x62\n\x13JinaExecutorRestore\x12K\n\x07restore\x12\x1c.jina.RestoreSnapshotCommand\x1a .jina.RestoreSnapshotStatusProto\"\x00\x32\x64\n\x1bJinaExecutorRestoreProgress\x12\x45\n\x0erestore_status\x12\x0f.jina.RestoreId\x1a .jina.RestoreSnapshotStatusProto\"\x00\x42#Z!github.com/jina-ai/jina-raft/jinab\x06proto3')
 
 _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
 _builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'jina_pb2', globals())
 if _descriptor._USE_C_DESCRIPTORS == False:
 
   DESCRIPTOR._options = None
+  DESCRIPTOR._serialized_options = b'Z!github.com/jina-ai/jina-raft/jina'
   _JINAINFOPROTO_JINAENTRY._options = None
   _JINAINFOPROTO_JINAENTRY._serialized_options = b'8\001'
   _JINAINFOPROTO_ENVSENTRY._options = None
@@ -39,33 +40,55 @@
   _HEADERPROTO._serialized_start=489
   _HEADERPROTO._serialized_end=687
   _ENDPOINTSPROTO._serialized_start=689
-  _ENDPOINTSPROTO._serialized_end=724
-  _STATUSPROTO._serialized_start=727
-  _STATUSPROTO._serialized_end=976
-  _STATUSPROTO_EXCEPTIONPROTO._serialized_start=860
-  _STATUSPROTO_EXCEPTIONPROTO._serialized_end=938
-  _STATUSPROTO_STATUSCODE._serialized_start=940
-  _STATUSPROTO_STATUSCODE._serialized_end=976
-  _RELATEDENTITY._serialized_start=978
-  _RELATEDENTITY._serialized_end=1072
-  _DATAREQUESTPROTO._serialized_start=1075
-  _DATAREQUESTPROTO._serialized_end=1357
-  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_start=1264
-  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_end=1357
-  _DATAREQUESTPROTOWODATA._serialized_start=1360
-  _DATAREQUESTPROTOWODATA._serialized_end=1498
-  _DATAREQUESTLISTPROTO._serialized_start=1500
-  _DATAREQUESTLISTPROTO._serialized_end=1564
-  _JINADATAREQUESTRPC._serialized_start=1566
-  _JINADATAREQUESTRPC._serialized_end=1656
-  _JINASINGLEDATAREQUESTRPC._serialized_start=1658
-  _JINASINGLEDATAREQUESTRPC._serialized_end=1757
-  _JINARPC._serialized_start=1759
-  _JINARPC._serialized_end=1830
-  _JINADISCOVERENDPOINTSRPC._serialized_start=1832
-  _JINADISCOVERENDPOINTSRPC._serialized_end=1928
-  _JINAGATEWAYDRYRUNRPC._serialized_start=1930
-  _JINAGATEWAYDRYRUNRPC._serialized_end=2008
-  _JINAINFORPC._serialized_start=2010
-  _JINAINFORPC._serialized_end=2081
+  _ENDPOINTSPROTO._serialized_end=749
+  _STATUSPROTO._serialized_start=752
+  _STATUSPROTO._serialized_end=1001
+  _STATUSPROTO_EXCEPTIONPROTO._serialized_start=885
+  _STATUSPROTO_EXCEPTIONPROTO._serialized_end=963
+  _STATUSPROTO_STATUSCODE._serialized_start=965
+  _STATUSPROTO_STATUSCODE._serialized_end=1001
+  _RELATEDENTITY._serialized_start=1003
+  _RELATEDENTITY._serialized_end=1097
+  _DATAREQUESTPROTO._serialized_start=1100
+  _DATAREQUESTPROTO._serialized_end=1382
+  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_start=1289
+  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_end=1382
+  _DATAREQUESTPROTOWODATA._serialized_start=1385
+  _DATAREQUESTPROTOWODATA._serialized_end=1523
+  _DATAREQUESTLISTPROTO._serialized_start=1525
+  _DATAREQUESTLISTPROTO._serialized_end=1589
+  _SNAPSHOTID._serialized_start=1591
+  _SNAPSHOTID._serialized_end=1618
+  _RESTOREID._serialized_start=1620
+  _RESTOREID._serialized_end=1646
+  _SNAPSHOTSTATUSPROTO._serialized_start=1649
+  _SNAPSHOTSTATUSPROTO._serialized_end=1888
+  _SNAPSHOTSTATUSPROTO_STATUS._serialized_start=1775
+  _SNAPSHOTSTATUSPROTO_STATUS._serialized_end=1888
+  _RESTORESNAPSHOTSTATUSPROTO._serialized_start=1891
+  _RESTORESNAPSHOTSTATUSPROTO._serialized_end=2093
+  _RESTORESNAPSHOTSTATUSPROTO_STATUS._serialized_start=2007
+  _RESTORESNAPSHOTSTATUSPROTO_STATUS._serialized_end=2093
+  _RESTORESNAPSHOTCOMMAND._serialized_start=2095
+  _RESTORESNAPSHOTCOMMAND._serialized_end=2142
+  _JINADATAREQUESTRPC._serialized_start=2144
+  _JINADATAREQUESTRPC._serialized_end=2234
+  _JINASINGLEDATAREQUESTRPC._serialized_start=2236
+  _JINASINGLEDATAREQUESTRPC._serialized_end=2335
+  _JINARPC._serialized_start=2337
+  _JINARPC._serialized_end=2408
+  _JINADISCOVERENDPOINTSRPC._serialized_start=2410
+  _JINADISCOVERENDPOINTSRPC._serialized_end=2506
+  _JINAGATEWAYDRYRUNRPC._serialized_start=2508
+  _JINAGATEWAYDRYRUNRPC._serialized_end=2586
+  _JINAINFORPC._serialized_start=2588
+  _JINAINFORPC._serialized_end=2659
+  _JINAEXECUTORSNAPSHOT._serialized_start=2661
+  _JINAEXECUTORSNAPSHOT._serialized_end=2748
+  _JINAEXECUTORSNAPSHOTPROGRESS._serialized_start=2750
+  _JINAEXECUTORSNAPSHOTPROGRESS._serialized_end=2846
+  _JINAEXECUTORRESTORE._serialized_start=2848
+  _JINAEXECUTORRESTORE._serialized_end=2946
+  _JINAEXECUTORRESTOREPROGRESS._serialized_start=2948
+  _JINAEXECUTORRESTOREPROGRESS._serialized_end=3048
 # @@protoc_insertion_point(module_scope)
diff --git a/jina/proto/docarray_v2/pb/jina_pb2_grpc.py b/jina/proto/docarray_v2/pb/jina_pb2_grpc.py
index b1fb12b313eb4..2aa864543aee9 100644
--- a/jina/proto/docarray_v2/pb/jina_pb2_grpc.py
+++ b/jina/proto/docarray_v2/pb/jina_pb2_grpc.py
@@ -412,3 +412,271 @@ def _status(request,
             jina__pb2.JinaInfoProto.FromString,
             options, channel_credentials,
             insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorSnapshotStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.snapshot = channel.unary_unary(
+                '/jina.JinaExecutorSnapshot/snapshot',
+                request_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+                response_deserializer=jina__pb2.SnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorSnapshotServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def snapshot(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorSnapshotServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'snapshot': grpc.unary_unary_rpc_method_handler(
+                    servicer.snapshot,
+                    request_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+                    response_serializer=jina__pb2.SnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorSnapshot', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorSnapshot(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def snapshot(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorSnapshot/snapshot',
+            google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+            jina__pb2.SnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorSnapshotProgressStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.snapshot_status = channel.unary_unary(
+                '/jina.JinaExecutorSnapshotProgress/snapshot_status',
+                request_serializer=jina__pb2.SnapshotId.SerializeToString,
+                response_deserializer=jina__pb2.SnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorSnapshotProgressServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def snapshot_status(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorSnapshotProgressServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'snapshot_status': grpc.unary_unary_rpc_method_handler(
+                    servicer.snapshot_status,
+                    request_deserializer=jina__pb2.SnapshotId.FromString,
+                    response_serializer=jina__pb2.SnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorSnapshotProgress', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorSnapshotProgress(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def snapshot_status(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorSnapshotProgress/snapshot_status',
+            jina__pb2.SnapshotId.SerializeToString,
+            jina__pb2.SnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorRestoreStub(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.restore = channel.unary_unary(
+                '/jina.JinaExecutorRestore/restore',
+                request_serializer=jina__pb2.RestoreSnapshotCommand.SerializeToString,
+                response_deserializer=jina__pb2.RestoreSnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorRestoreServicer(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    def restore(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorRestoreServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'restore': grpc.unary_unary_rpc_method_handler(
+                    servicer.restore,
+                    request_deserializer=jina__pb2.RestoreSnapshotCommand.FromString,
+                    response_serializer=jina__pb2.RestoreSnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorRestore', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorRestore(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    @staticmethod
+    def restore(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorRestore/restore',
+            jina__pb2.RestoreSnapshotCommand.SerializeToString,
+            jina__pb2.RestoreSnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorRestoreProgressStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.restore_status = channel.unary_unary(
+                '/jina.JinaExecutorRestoreProgress/restore_status',
+                request_serializer=jina__pb2.RestoreId.SerializeToString,
+                response_deserializer=jina__pb2.RestoreSnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorRestoreProgressServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def restore_status(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorRestoreProgressServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'restore_status': grpc.unary_unary_rpc_method_handler(
+                    servicer.restore_status,
+                    request_deserializer=jina__pb2.RestoreId.FromString,
+                    response_serializer=jina__pb2.RestoreSnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorRestoreProgress', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorRestoreProgress(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def restore_status(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorRestoreProgress/restore_status',
+            jina__pb2.RestoreId.SerializeToString,
+            jina__pb2.RestoreSnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
diff --git a/jina/proto/docarray_v2/pb2/jina_pb2.py b/jina/proto/docarray_v2/pb2/jina_pb2.py
index ee86a533b9883..8a22c335ad328 100644
--- a/jina/proto/docarray_v2/pb2/jina_pb2.py
+++ b/jina/proto/docarray_v2/pb2/jina_pb2.py
@@ -7,18 +7,18 @@
 from google.protobuf import message as _message
 from google.protobuf import reflection as _reflection
 from google.protobuf import symbol_database as _symbol_database
-
 # @@protoc_insertion_point(imports)
 
 _sym_db = _symbol_database.Default()
 
 
-import docarray.proto.pb2.docarray_pb2 as docarray__pb2
-from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
-from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
 from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2
+from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
+import docarray.proto.pb2.docarray_pb2 as docarray__pb2
+
 
-DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\njina.proto\x12\x04jina\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x0e\x64ocarray.proto\"\x9f\x01\n\nRouteProto\x12\x10\n\x08\x65xecutor\x18\x01 \x01(\t\x12.\n\nstart_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12!\n\x06status\x18\x04 \x01(\x0b\x32\x11.jina.StatusProto\"\xc3\x01\n\rJinaInfoProto\x12+\n\x04jina\x18\x01 \x03(\x0b\x32\x1d.jina.JinaInfoProto.JinaEntry\x12+\n\x04\x65nvs\x18\x02 \x03(\x0b\x32\x1d.jina.JinaInfoProto.EnvsEntry\x1a+\n\tJinaEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x1a+\n\tEnvsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xc6\x01\n\x0bHeaderProto\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12!\n\x06status\x18\x02 \x01(\x0b\x32\x11.jina.StatusProto\x12\x1a\n\rexec_endpoint\x18\x03 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0ftarget_executor\x18\x04 \x01(\tH\x01\x88\x01\x01\x12\x14\n\x07timeout\x18\x05 \x01(\rH\x02\x88\x01\x01\x42\x10\n\x0e_exec_endpointB\x12\n\x10_target_executorB\n\n\x08_timeout\"#\n\x0e\x45ndpointsProto\x12\x11\n\tendpoints\x18\x01 \x03(\t\"\xf9\x01\n\x0bStatusProto\x12*\n\x04\x63ode\x18\x01 \x01(\x0e\x32\x1c.jina.StatusProto.StatusCode\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x33\n\texception\x18\x03 \x01(\x0b\x32 .jina.StatusProto.ExceptionProto\x1aN\n\x0e\x45xceptionProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04\x61rgs\x18\x02 \x03(\t\x12\x0e\n\x06stacks\x18\x03 \x03(\t\x12\x10\n\x08\x65xecutor\x18\x04 \x01(\t\"$\n\nStatusCode\x12\x0b\n\x07SUCCESS\x10\x00\x12\t\n\x05\x45RROR\x10\x01\"^\n\rRelatedEntity\x12\n\n\x02id\x18\x01 \x01(\t\x12\x0f\n\x07\x61\x64\x64ress\x18\x02 \x01(\t\x12\x0c\n\x04port\x18\x03 \x01(\r\x12\x15\n\x08shard_id\x18\x04 \x01(\rH\x00\x88\x01\x01\x42\x0b\n\t_shard_id\"\x9a\x02\n\x10\x44\x61taRequestProto\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\x12\x35\n\x04\x64\x61ta\x18\x04 \x01(\x0b\x32\'.jina.DataRequestProto.DataContentProto\x1a]\n\x10\x44\x61taContentProto\x12&\n\x04\x64ocs\x18\x01 \x01(\x0b\x32\x16.docarray.DocListProtoH\x00\x12\x14\n\ndocs_bytes\x18\x02 \x01(\x0cH\x00\x42\x0b\n\tdocuments\"\x8a\x01\n\x16\x44\x61taRequestProtoWoData\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\"@\n\x14\x44\x61taRequestListProto\x12(\n\x08requests\x18\x01 \x03(\x0b\x32\x16.jina.DataRequestProto2Z\n\x12JinaDataRequestRPC\x12\x44\n\x0cprocess_data\x12\x1a.jina.DataRequestListProto\x1a\x16.jina.DataRequestProto\"\x00\x32\x63\n\x18JinaSingleDataRequestRPC\x12G\n\x13process_single_data\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00\x32G\n\x07JinaRPC\x12<\n\x04\x43\x61ll\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00(\x01\x30\x01\x32`\n\x18JinaDiscoverEndpointsRPC\x12\x44\n\x12\x65ndpoint_discovery\x12\x16.google.protobuf.Empty\x1a\x14.jina.EndpointsProto\"\x00\x32N\n\x14JinaGatewayDryRunRPC\x12\x36\n\x07\x64ry_run\x12\x16.google.protobuf.Empty\x1a\x11.jina.StatusProto\"\x00\x32G\n\x0bJinaInfoRPC\x12\x38\n\x07_status\x12\x16.google.protobuf.Empty\x1a\x13.jina.JinaInfoProto\"\x00\x62\x06proto3')
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\njina.proto\x12\x04jina\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1cgoogle/protobuf/struct.proto\x1a\x1bgoogle/protobuf/empty.proto\x1a\x0e\x64ocarray.proto\"\x9f\x01\n\nRouteProto\x12\x10\n\x08\x65xecutor\x18\x01 \x01(\t\x12.\n\nstart_time\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x08\x65nd_time\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12!\n\x06status\x18\x04 \x01(\x0b\x32\x11.jina.StatusProto\"\xc3\x01\n\rJinaInfoProto\x12+\n\x04jina\x18\x01 \x03(\x0b\x32\x1d.jina.JinaInfoProto.JinaEntry\x12+\n\x04\x65nvs\x18\x02 \x03(\x0b\x32\x1d.jina.JinaInfoProto.EnvsEntry\x1a+\n\tJinaEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x1a+\n\tEnvsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xc6\x01\n\x0bHeaderProto\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12!\n\x06status\x18\x02 \x01(\x0b\x32\x11.jina.StatusProto\x12\x1a\n\rexec_endpoint\x18\x03 \x01(\tH\x00\x88\x01\x01\x12\x1c\n\x0ftarget_executor\x18\x04 \x01(\tH\x01\x88\x01\x01\x12\x14\n\x07timeout\x18\x05 \x01(\rH\x02\x88\x01\x01\x42\x10\n\x0e_exec_endpointB\x12\n\x10_target_executorB\n\n\x08_timeout\"<\n\x0e\x45ndpointsProto\x12\x11\n\tendpoints\x18\x01 \x03(\t\x12\x17\n\x0fwrite_endpoints\x18\x02 \x03(\t\"\xf9\x01\n\x0bStatusProto\x12*\n\x04\x63ode\x18\x01 \x01(\x0e\x32\x1c.jina.StatusProto.StatusCode\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x33\n\texception\x18\x03 \x01(\x0b\x32 .jina.StatusProto.ExceptionProto\x1aN\n\x0e\x45xceptionProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04\x61rgs\x18\x02 \x03(\t\x12\x0e\n\x06stacks\x18\x03 \x03(\t\x12\x10\n\x08\x65xecutor\x18\x04 \x01(\t\"$\n\nStatusCode\x12\x0b\n\x07SUCCESS\x10\x00\x12\t\n\x05\x45RROR\x10\x01\"^\n\rRelatedEntity\x12\n\n\x02id\x18\x01 \x01(\t\x12\x0f\n\x07\x61\x64\x64ress\x18\x02 \x01(\t\x12\x0c\n\x04port\x18\x03 \x01(\r\x12\x15\n\x08shard_id\x18\x04 \x01(\rH\x00\x88\x01\x01\x42\x0b\n\t_shard_id\"\x9a\x02\n\x10\x44\x61taRequestProto\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\x12\x35\n\x04\x64\x61ta\x18\x04 \x01(\x0b\x32\'.jina.DataRequestProto.DataContentProto\x1a]\n\x10\x44\x61taContentProto\x12&\n\x04\x64ocs\x18\x01 \x01(\x0b\x32\x16.docarray.DocListProtoH\x00\x12\x14\n\ndocs_bytes\x18\x02 \x01(\x0cH\x00\x42\x0b\n\tdocuments\"\x8a\x01\n\x16\x44\x61taRequestProtoWoData\x12!\n\x06header\x18\x01 \x01(\x0b\x32\x11.jina.HeaderProto\x12+\n\nparameters\x18\x02 \x01(\x0b\x32\x17.google.protobuf.Struct\x12 \n\x06routes\x18\x03 \x03(\x0b\x32\x10.jina.RouteProto\"@\n\x14\x44\x61taRequestListProto\x12(\n\x08requests\x18\x01 \x03(\x0b\x32\x16.jina.DataRequestProto\"\x1b\n\nSnapshotId\x12\r\n\x05value\x18\x01 \x01(\t\"\x1a\n\tRestoreId\x12\r\n\x05value\x18\x01 \x01(\t\"\xef\x01\n\x13SnapshotStatusProto\x12\x1c\n\x02id\x18\x01 \x01(\x0b\x32\x10.jina.SnapshotId\x12\x30\n\x06status\x18\x02 \x01(\x0e\x32 .jina.SnapshotStatusProto.Status\x12\x15\n\rsnapshot_file\x18\x03 \x01(\t\"q\n\x06Status\x12\x15\n\x11STATE_UNSPECIFIED\x10\x00\x12\n\n\x06QUEUED\x10\x01\x12\r\n\tSCHEDULED\x10\x02\x12\x0b\n\x07RUNNING\x10\x03\x12\r\n\tSUCCEEDED\x10\x04\x12\n\n\x06\x46\x41ILED\x10\x05\x12\r\n\tNOT_FOUND\x10\x06\"\xca\x01\n\x1aRestoreSnapshotStatusProto\x12\x1b\n\x02id\x18\x01 \x01(\x0b\x32\x0f.jina.RestoreId\x12\x37\n\x06status\x18\x02 \x01(\x0e\x32\'.jina.RestoreSnapshotStatusProto.Status\"V\n\x06Status\x12\x15\n\x11STATE_UNSPECIFIED\x10\x00\x12\x0b\n\x07RUNNING\x10\x01\x12\r\n\tSUCCEEDED\x10\x02\x12\n\n\x06\x46\x41ILED\x10\x03\x12\r\n\tNOT_FOUND\x10\x06\"/\n\x16RestoreSnapshotCommand\x12\x15\n\rsnapshot_file\x18\x01 \x01(\t2Z\n\x12JinaDataRequestRPC\x12\x44\n\x0cprocess_data\x12\x1a.jina.DataRequestListProto\x1a\x16.jina.DataRequestProto\"\x00\x32\x63\n\x18JinaSingleDataRequestRPC\x12G\n\x13process_single_data\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00\x32G\n\x07JinaRPC\x12<\n\x04\x43\x61ll\x12\x16.jina.DataRequestProto\x1a\x16.jina.DataRequestProto\"\x00(\x01\x30\x01\x32`\n\x18JinaDiscoverEndpointsRPC\x12\x44\n\x12\x65ndpoint_discovery\x12\x16.google.protobuf.Empty\x1a\x14.jina.EndpointsProto\"\x00\x32N\n\x14JinaGatewayDryRunRPC\x12\x36\n\x07\x64ry_run\x12\x16.google.protobuf.Empty\x1a\x11.jina.StatusProto\"\x00\x32G\n\x0bJinaInfoRPC\x12\x38\n\x07_status\x12\x16.google.protobuf.Empty\x1a\x13.jina.JinaInfoProto\"\x00\x32W\n\x14JinaExecutorSnapshot\x12?\n\x08snapshot\x12\x16.google.protobuf.Empty\x1a\x19.jina.SnapshotStatusProto\"\x00\x32`\n\x1cJinaExecutorSnapshotProgress\x12@\n\x0fsnapshot_status\x12\x10.jina.SnapshotId\x1a\x19.jina.SnapshotStatusProto\"\x00\x32\x62\n\x13JinaExecutorRestore\x12K\n\x07restore\x12\x1c.jina.RestoreSnapshotCommand\x1a .jina.RestoreSnapshotStatusProto\"\x00\x32\x64\n\x1bJinaExecutorRestoreProgress\x12\x45\n\x0erestore_status\x12\x0f.jina.RestoreId\x1a .jina.RestoreSnapshotStatusProto\"\x00\x42#Z!github.com/jina-ai/jina-raft/jinab\x06proto3')
 
 
 
@@ -35,7 +35,14 @@
 _DATAREQUESTPROTO_DATACONTENTPROTO = _DATAREQUESTPROTO.nested_types_by_name['DataContentProto']
 _DATAREQUESTPROTOWODATA = DESCRIPTOR.message_types_by_name['DataRequestProtoWoData']
 _DATAREQUESTLISTPROTO = DESCRIPTOR.message_types_by_name['DataRequestListProto']
+_SNAPSHOTID = DESCRIPTOR.message_types_by_name['SnapshotId']
+_RESTOREID = DESCRIPTOR.message_types_by_name['RestoreId']
+_SNAPSHOTSTATUSPROTO = DESCRIPTOR.message_types_by_name['SnapshotStatusProto']
+_RESTORESNAPSHOTSTATUSPROTO = DESCRIPTOR.message_types_by_name['RestoreSnapshotStatusProto']
+_RESTORESNAPSHOTCOMMAND = DESCRIPTOR.message_types_by_name['RestoreSnapshotCommand']
 _STATUSPROTO_STATUSCODE = _STATUSPROTO.enum_types_by_name['StatusCode']
+_SNAPSHOTSTATUSPROTO_STATUS = _SNAPSHOTSTATUSPROTO.enum_types_by_name['Status']
+_RESTORESNAPSHOTSTATUSPROTO_STATUS = _RESTORESNAPSHOTSTATUSPROTO.enum_types_by_name['Status']
 RouteProto = _reflection.GeneratedProtocolMessageType('RouteProto', (_message.Message,), {
   'DESCRIPTOR' : _ROUTEPROTO,
   '__module__' : 'jina_pb2'
@@ -131,15 +138,55 @@
   })
 _sym_db.RegisterMessage(DataRequestListProto)
 
+SnapshotId = _reflection.GeneratedProtocolMessageType('SnapshotId', (_message.Message,), {
+  'DESCRIPTOR' : _SNAPSHOTID,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.SnapshotId)
+  })
+_sym_db.RegisterMessage(SnapshotId)
+
+RestoreId = _reflection.GeneratedProtocolMessageType('RestoreId', (_message.Message,), {
+  'DESCRIPTOR' : _RESTOREID,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.RestoreId)
+  })
+_sym_db.RegisterMessage(RestoreId)
+
+SnapshotStatusProto = _reflection.GeneratedProtocolMessageType('SnapshotStatusProto', (_message.Message,), {
+  'DESCRIPTOR' : _SNAPSHOTSTATUSPROTO,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.SnapshotStatusProto)
+  })
+_sym_db.RegisterMessage(SnapshotStatusProto)
+
+RestoreSnapshotStatusProto = _reflection.GeneratedProtocolMessageType('RestoreSnapshotStatusProto', (_message.Message,), {
+  'DESCRIPTOR' : _RESTORESNAPSHOTSTATUSPROTO,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.RestoreSnapshotStatusProto)
+  })
+_sym_db.RegisterMessage(RestoreSnapshotStatusProto)
+
+RestoreSnapshotCommand = _reflection.GeneratedProtocolMessageType('RestoreSnapshotCommand', (_message.Message,), {
+  'DESCRIPTOR' : _RESTORESNAPSHOTCOMMAND,
+  '__module__' : 'jina_pb2'
+  # @@protoc_insertion_point(class_scope:jina.RestoreSnapshotCommand)
+  })
+_sym_db.RegisterMessage(RestoreSnapshotCommand)
+
 _JINADATAREQUESTRPC = DESCRIPTOR.services_by_name['JinaDataRequestRPC']
 _JINASINGLEDATAREQUESTRPC = DESCRIPTOR.services_by_name['JinaSingleDataRequestRPC']
 _JINARPC = DESCRIPTOR.services_by_name['JinaRPC']
 _JINADISCOVERENDPOINTSRPC = DESCRIPTOR.services_by_name['JinaDiscoverEndpointsRPC']
 _JINAGATEWAYDRYRUNRPC = DESCRIPTOR.services_by_name['JinaGatewayDryRunRPC']
 _JINAINFORPC = DESCRIPTOR.services_by_name['JinaInfoRPC']
+_JINAEXECUTORSNAPSHOT = DESCRIPTOR.services_by_name['JinaExecutorSnapshot']
+_JINAEXECUTORSNAPSHOTPROGRESS = DESCRIPTOR.services_by_name['JinaExecutorSnapshotProgress']
+_JINAEXECUTORRESTORE = DESCRIPTOR.services_by_name['JinaExecutorRestore']
+_JINAEXECUTORRESTOREPROGRESS = DESCRIPTOR.services_by_name['JinaExecutorRestoreProgress']
 if _descriptor._USE_C_DESCRIPTORS == False:
 
   DESCRIPTOR._options = None
+  DESCRIPTOR._serialized_options = b'Z!github.com/jina-ai/jina-raft/jina'
   _JINAINFOPROTO_JINAENTRY._options = None
   _JINAINFOPROTO_JINAENTRY._serialized_options = b'8\001'
   _JINAINFOPROTO_ENVSENTRY._options = None
@@ -155,33 +202,55 @@
   _HEADERPROTO._serialized_start=489
   _HEADERPROTO._serialized_end=687
   _ENDPOINTSPROTO._serialized_start=689
-  _ENDPOINTSPROTO._serialized_end=724
-  _STATUSPROTO._serialized_start=727
-  _STATUSPROTO._serialized_end=976
-  _STATUSPROTO_EXCEPTIONPROTO._serialized_start=860
-  _STATUSPROTO_EXCEPTIONPROTO._serialized_end=938
-  _STATUSPROTO_STATUSCODE._serialized_start=940
-  _STATUSPROTO_STATUSCODE._serialized_end=976
-  _RELATEDENTITY._serialized_start=978
-  _RELATEDENTITY._serialized_end=1072
-  _DATAREQUESTPROTO._serialized_start=1075
-  _DATAREQUESTPROTO._serialized_end=1357
-  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_start=1264
-  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_end=1357
-  _DATAREQUESTPROTOWODATA._serialized_start=1360
-  _DATAREQUESTPROTOWODATA._serialized_end=1498
-  _DATAREQUESTLISTPROTO._serialized_start=1500
-  _DATAREQUESTLISTPROTO._serialized_end=1564
-  _JINADATAREQUESTRPC._serialized_start=1566
-  _JINADATAREQUESTRPC._serialized_end=1656
-  _JINASINGLEDATAREQUESTRPC._serialized_start=1658
-  _JINASINGLEDATAREQUESTRPC._serialized_end=1757
-  _JINARPC._serialized_start=1759
-  _JINARPC._serialized_end=1830
-  _JINADISCOVERENDPOINTSRPC._serialized_start=1832
-  _JINADISCOVERENDPOINTSRPC._serialized_end=1928
-  _JINAGATEWAYDRYRUNRPC._serialized_start=1930
-  _JINAGATEWAYDRYRUNRPC._serialized_end=2008
-  _JINAINFORPC._serialized_start=2010
-  _JINAINFORPC._serialized_end=2081
+  _ENDPOINTSPROTO._serialized_end=749
+  _STATUSPROTO._serialized_start=752
+  _STATUSPROTO._serialized_end=1001
+  _STATUSPROTO_EXCEPTIONPROTO._serialized_start=885
+  _STATUSPROTO_EXCEPTIONPROTO._serialized_end=963
+  _STATUSPROTO_STATUSCODE._serialized_start=965
+  _STATUSPROTO_STATUSCODE._serialized_end=1001
+  _RELATEDENTITY._serialized_start=1003
+  _RELATEDENTITY._serialized_end=1097
+  _DATAREQUESTPROTO._serialized_start=1100
+  _DATAREQUESTPROTO._serialized_end=1382
+  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_start=1289
+  _DATAREQUESTPROTO_DATACONTENTPROTO._serialized_end=1382
+  _DATAREQUESTPROTOWODATA._serialized_start=1385
+  _DATAREQUESTPROTOWODATA._serialized_end=1523
+  _DATAREQUESTLISTPROTO._serialized_start=1525
+  _DATAREQUESTLISTPROTO._serialized_end=1589
+  _SNAPSHOTID._serialized_start=1591
+  _SNAPSHOTID._serialized_end=1618
+  _RESTOREID._serialized_start=1620
+  _RESTOREID._serialized_end=1646
+  _SNAPSHOTSTATUSPROTO._serialized_start=1649
+  _SNAPSHOTSTATUSPROTO._serialized_end=1888
+  _SNAPSHOTSTATUSPROTO_STATUS._serialized_start=1775
+  _SNAPSHOTSTATUSPROTO_STATUS._serialized_end=1888
+  _RESTORESNAPSHOTSTATUSPROTO._serialized_start=1891
+  _RESTORESNAPSHOTSTATUSPROTO._serialized_end=2093
+  _RESTORESNAPSHOTSTATUSPROTO_STATUS._serialized_start=2007
+  _RESTORESNAPSHOTSTATUSPROTO_STATUS._serialized_end=2093
+  _RESTORESNAPSHOTCOMMAND._serialized_start=2095
+  _RESTORESNAPSHOTCOMMAND._serialized_end=2142
+  _JINADATAREQUESTRPC._serialized_start=2144
+  _JINADATAREQUESTRPC._serialized_end=2234
+  _JINASINGLEDATAREQUESTRPC._serialized_start=2236
+  _JINASINGLEDATAREQUESTRPC._serialized_end=2335
+  _JINARPC._serialized_start=2337
+  _JINARPC._serialized_end=2408
+  _JINADISCOVERENDPOINTSRPC._serialized_start=2410
+  _JINADISCOVERENDPOINTSRPC._serialized_end=2506
+  _JINAGATEWAYDRYRUNRPC._serialized_start=2508
+  _JINAGATEWAYDRYRUNRPC._serialized_end=2586
+  _JINAINFORPC._serialized_start=2588
+  _JINAINFORPC._serialized_end=2659
+  _JINAEXECUTORSNAPSHOT._serialized_start=2661
+  _JINAEXECUTORSNAPSHOT._serialized_end=2748
+  _JINAEXECUTORSNAPSHOTPROGRESS._serialized_start=2750
+  _JINAEXECUTORSNAPSHOTPROGRESS._serialized_end=2846
+  _JINAEXECUTORRESTORE._serialized_start=2848
+  _JINAEXECUTORRESTORE._serialized_end=2946
+  _JINAEXECUTORRESTOREPROGRESS._serialized_start=2948
+  _JINAEXECUTORRESTOREPROGRESS._serialized_end=3048
 # @@protoc_insertion_point(module_scope)
diff --git a/jina/proto/docarray_v2/pb2/jina_pb2_grpc.py b/jina/proto/docarray_v2/pb2/jina_pb2_grpc.py
index 9f25df0c8d7ed..2aa864543aee9 100644
--- a/jina/proto/docarray_v2/pb2/jina_pb2_grpc.py
+++ b/jina/proto/docarray_v2/pb2/jina_pb2_grpc.py
@@ -1,8 +1,8 @@
 # Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
 """Client and server classes corresponding to protobuf-defined services."""
 import grpc
-from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
 
+from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
 from ... import serializer as jina__pb2
 
 
@@ -412,3 +412,271 @@ def _status(request,
             jina__pb2.JinaInfoProto.FromString,
             options, channel_credentials,
             insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorSnapshotStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.snapshot = channel.unary_unary(
+                '/jina.JinaExecutorSnapshot/snapshot',
+                request_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+                response_deserializer=jina__pb2.SnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorSnapshotServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def snapshot(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorSnapshotServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'snapshot': grpc.unary_unary_rpc_method_handler(
+                    servicer.snapshot,
+                    request_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
+                    response_serializer=jina__pb2.SnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorSnapshot', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorSnapshot(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def snapshot(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorSnapshot/snapshot',
+            google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
+            jina__pb2.SnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorSnapshotProgressStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.snapshot_status = channel.unary_unary(
+                '/jina.JinaExecutorSnapshotProgress/snapshot_status',
+                request_serializer=jina__pb2.SnapshotId.SerializeToString,
+                response_deserializer=jina__pb2.SnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorSnapshotProgressServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def snapshot_status(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorSnapshotProgressServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'snapshot_status': grpc.unary_unary_rpc_method_handler(
+                    servicer.snapshot_status,
+                    request_deserializer=jina__pb2.SnapshotId.FromString,
+                    response_serializer=jina__pb2.SnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorSnapshotProgress', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorSnapshotProgress(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def snapshot_status(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorSnapshotProgress/snapshot_status',
+            jina__pb2.SnapshotId.SerializeToString,
+            jina__pb2.SnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorRestoreStub(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.restore = channel.unary_unary(
+                '/jina.JinaExecutorRestore/restore',
+                request_serializer=jina__pb2.RestoreSnapshotCommand.SerializeToString,
+                response_deserializer=jina__pb2.RestoreSnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorRestoreServicer(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    def restore(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorRestoreServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'restore': grpc.unary_unary_rpc_method_handler(
+                    servicer.restore,
+                    request_deserializer=jina__pb2.RestoreSnapshotCommand.FromString,
+                    response_serializer=jina__pb2.RestoreSnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorRestore', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorRestore(object):
+    """*
+    jina gRPC service to trigger a restore at the Executor Runtime.
+    """
+
+    @staticmethod
+    def restore(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorRestore/restore',
+            jina__pb2.RestoreSnapshotCommand.SerializeToString,
+            jina__pb2.RestoreSnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+
+class JinaExecutorRestoreProgressStub(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.restore_status = channel.unary_unary(
+                '/jina.JinaExecutorRestoreProgress/restore_status',
+                request_serializer=jina__pb2.RestoreId.SerializeToString,
+                response_deserializer=jina__pb2.RestoreSnapshotStatusProto.FromString,
+                )
+
+
+class JinaExecutorRestoreProgressServicer(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    def restore_status(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_JinaExecutorRestoreProgressServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'restore_status': grpc.unary_unary_rpc_method_handler(
+                    servicer.restore_status,
+                    request_deserializer=jina__pb2.RestoreId.FromString,
+                    response_serializer=jina__pb2.RestoreSnapshotStatusProto.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'jina.JinaExecutorRestoreProgress', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class JinaExecutorRestoreProgress(object):
+    """*
+    jina gRPC service to trigger a snapshot at the Executor Runtime.
+    """
+
+    @staticmethod
+    def restore_status(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(request, target, '/jina.JinaExecutorRestoreProgress/restore_status',
+            jina__pb2.RestoreId.SerializeToString,
+            jina__pb2.RestoreSnapshotStatusProto.FromString,
+            options, channel_credentials,
+            insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
diff --git a/jina/proto/jina_pb2.py b/jina/proto/jina_pb2.py
index 7ac7b6dc3f9f0..8509b919dfb55 100644
--- a/jina/proto/jina_pb2.py
+++ b/jina/proto/jina_pb2.py
@@ -4,12 +4,12 @@
 
 if __pb__version__.startswith('4'):
     if is_docarray_v2:
-        from .docarray_v2.pb.jina_pb2 import *
+        from jina.proto.docarray_v2.pb.jina_pb2 import *
     else:
-        from .docarray_v1.pb.jina_pb2 import *
+        from jina.proto.docarray_v1.pb.jina_pb2 import *
 
 else:
     if is_docarray_v2:
-        from .docarray_v2.pb2.jina_pb2 import *
+        from jina.proto.docarray_v2.pb2.jina_pb2 import *
     else:
-        from .docarray_v1.pb2.jina_pb2 import *
+        from jina.proto.docarray_v1.pb2.jina_pb2 import *
diff --git a/jina/proto/jina_pb2_grpc.py b/jina/proto/jina_pb2_grpc.py
index 597ac027dcd2d..af3ac94d3b370 100644
--- a/jina/proto/jina_pb2_grpc.py
+++ b/jina/proto/jina_pb2_grpc.py
@@ -4,12 +4,12 @@
 
 if __pb__version__.startswith('4'):
     if is_docarray_v2:
-        from .docarray_v2.pb.jina_pb2_grpc import *
+        from jina.proto.docarray_v2.pb.jina_pb2_grpc import *
     else:
-        from .docarray_v1.pb.jina_pb2_grpc import *
+        from jina.proto.docarray_v1.pb.jina_pb2_grpc import *
 
 else:
     if is_docarray_v2:
-        from .docarray_v2.pb2.jina_pb2_grpc import *
+        from jina.proto.docarray_v2.pb2.jina_pb2_grpc import *
     else:
-        from .docarray_v1.pb2.jina_pb2_grpc import *
+        from jina.proto.docarray_v1.pb2.jina_pb2_grpc import *
diff --git a/jina/proto/serializer.py b/jina/proto/serializer.py
index 92d85d87b9e92..555b2bc4a79e8 100644
--- a/jina/proto/serializer.py
+++ b/jina/proto/serializer.py
@@ -154,3 +154,128 @@ def FromString(x: bytes):
         ip.ParseFromString(x)
 
         return ip
+
+
+class SnapshotId:
+    """Placeholder that delegates the serialization and deserialization to the internal protobuf"""
+
+    @staticmethod
+    def SerializeToString(x):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        return x.SerializeToString()
+
+    @staticmethod
+    def FromString(x: bytes):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        si = jina_pb2.SnapshotId()
+        si.ParseFromString(x)
+
+        return si
+
+
+class SnapshotStatusProto:
+    """Placeholder that delegates the serialization and deserialization to the internal protobuf"""
+
+    @staticmethod
+    def SerializeToString(x):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        return x.SerializeToString()
+
+    @staticmethod
+    def FromString(x: bytes):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        sp = jina_pb2.SnapshotStatusProto()
+        sp.ParseFromString(x)
+
+        return sp
+
+
+class RestoreSnapshotCommand:
+    """Placeholder that delegates the serialization and deserialization to the internal protobuf"""
+
+    @staticmethod
+    def SerializeToString(x):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        return x.SerializeToString()
+
+    @staticmethod
+    def FromString(x: bytes):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        rpcommand = jina_pb2.RestoreSnapshotCommand()
+        rpcommand.ParseFromString(x)
+
+        return rpcommand
+
+
+class RestoreId:
+    """Placeholder that delegates the serialization and deserialization to the internal protobuf"""
+
+    @staticmethod
+    def SerializeToString(x):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        return x.SerializeToString()
+
+    @staticmethod
+    def FromString(x: bytes):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        ri = jina_pb2.RestoreId()
+        ri.ParseFromString(x)
+
+        return ri
+
+
+class RestoreSnapshotStatusProto:
+    """Placeholder that delegates the serialization and deserialization to the internal protobuf"""
+
+    @staticmethod
+    def SerializeToString(x):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        return x.SerializeToString()
+
+    @staticmethod
+    def FromString(x: bytes):
+        """
+        # noqa: DAR101
+        # noqa: DAR102
+        # noqa: DAR201
+        """
+        rsp = jina_pb2.RestoreSnapshotStatusProto()
+        rsp.ParseFromString(x)
+
+        return rsp
diff --git a/jina/serve/consensus/README.md b/jina/serve/consensus/README.md
new file mode 100644
index 0000000000000..c8b95db1621bb
--- /dev/null
+++ b/jina/serve/consensus/README.md
@@ -0,0 +1,124 @@
+# Raft implementation for Jina Docarray
+
+## Setup Go
+
+The default project and folder setup is for the Go project files. Install the Go version >= 1.19. To install all dependencies run:
+
+```shell
+go get
+```
+
+## Setup Python
+
+A copy of the Jina library is placed under the `jina-core` folder. 
+- Create a virtual environment with pipenv: 
+```shell
+pipenv --python 3.7
+```
+- Install Jina in the virtual environment:
+```shell
+pipenv install ".[devel]"
+```
+- To ensure that the correct jina binary is being used, run:
+```shell
+source $(pipenv --venv)/bin/activate
+# check jina location
+# the location should belong the the virtual env
+which jina
+```
+
+## Create local protogen docker
+
+A local docker version needs to be built to correctly re-build jina proto files if modified.
+
+```shell
+docker build -f Dockerfiles/protogen.Dockerfile -t proto/jina-raft-proto .
+```
+
+### Generate Proto for Python
+
+```shell
+docker run --rm -v $(pwd)/jina/proto:/jina/proto proto/jina-raft-proto
+```
+
+### Generate Proto for Go
+
+```shell
+scripts/protogen.sh
+```
+
+# Start Flows
+
+Each flow needs to be started in a separate terminal.
+
+```shell
+scripts/start_executor.sh nodeA
+scripts/start_executor.sh nodeB
+scripts/start_executor.sh nodeC
+```
+
+## Delete Executor workspaces
+
+```shell
+scripts/delete_executor_workspaces.sh
+```
+
+# Install raftadmin locally
+
+Due to the stale go version in the raftadmin repository, the `go install github.com/Jille/raftadmin` doesn't install the raftadmin binary. Follow the below steps to install the binary locally.
+
+```shell
+mkdir -p $GOPATH/src/github.com/Jille
+cd $GOPATH/src/github.com/Jille
+git clone github.com/Jille/raftadmin
+go install cmd/raftadmin/raftadmin.go
+ls $GOPATH/bin # the raftadmin binary should be available
+```
+
+# Create cluster workspace
+
+```shell
+rm -rf /tmp/jina-raft-cluster/node* && mkdir -p /tmp/jina-raft-cluster/node{A,B,C}
+```
+
+# Start your own cluster
+
+```shell
+go build # build a local binary executable
+# open new terminal
+scripts/start_nodeA.sh
+# open new terminal
+scripts/start_nodeB.sh
+# open new terminal
+scripts/start_nodeC.sh
+# open new terminal
+scripts/admin_add_nodes.sh
+```
+
+*Note: The `--raft_bootstrap` argument is required only for the first run when creating a cluster from scratch.*
+
+### Execute a single request
+
+```shell
+go run client/client.go --target=localhost:50051
+```
+
+### Execute snapshot on a raft node
+
+```shell
+scripts/snapshot.sh
+```
+
+# Debugging scripts
+
+### Executor snapshot and snapshot progress
+
+To run a snapshot on the executor directly:
+```shell
+python scripts/trigger_executor_snapshot.py '0.0.0.0:60061'
+```
+
+To check the status of a snapshot with id '123' on the executor directly:
+```shell
+python scripts/check_snapshot_status.py '0.0.0.0:60061' '123'
+```
diff --git a/jina/serve/consensus/add_voter.go b/jina/serve/consensus/add_voter.go
new file mode 100644
index 0000000000000..806fdd58e8e08
--- /dev/null
+++ b/jina/serve/consensus/add_voter.go
@@ -0,0 +1,80 @@
+package main
+
+import (
+    "context"
+    "os"
+    "errors"
+
+    pb "github.com/Jille/raftadmin/proto"
+    "google.golang.org/grpc"
+    "google.golang.org/protobuf/encoding/prototext"
+    "google.golang.org/protobuf/reflect/protoreflect"
+    hclog "github.com/hashicorp/go-hclog"
+)
+
+func AddVoter(target string, id string, voter_address string) error {
+    logLevel := os.Getenv("JINA_LOG_LEVEL")
+    if logLevel == "" {
+        logLevel = "INFO"
+    }
+    add_voter_logger := hclog.New(&hclog.LoggerOptions{
+                    Name:  "add_voter",
+                    Level: hclog.LevelFromString(logLevel),
+                })
+    ctx := context.Background()
+    m := pb.File_raftadmin_proto.Services().ByName("RaftAdmin").Methods().ByName("AddVoter")
+    // Sort fields by field number.
+    reqDesc := m.Input()
+    unorderedFields := reqDesc.Fields()
+    fields := make([]protoreflect.FieldDescriptor, unorderedFields.Len())
+
+    for i := 0; unorderedFields.Len() > i; i++ {
+        f := unorderedFields.Get(i)
+        fields[f.Number()-1] = f
+    }
+    req := &pb.AddVoterRequest {
+                  Id: id,
+                  Address: voter_address,
+                  PreviousIndex: 0,
+              }
+
+    // Connect and send the RPC.
+    var o grpc.DialOption = grpc.EmptyDialOption{}
+    conn, err := grpc.Dial(target, grpc.WithInsecure(), grpc.WithBlock(), o)
+    if err != nil {
+        add_voter_logger.Error("Error dialing:", "error", err)
+        return err
+    }
+    defer conn.Close()
+
+    add_voter_logger.Debug("Invoking", "method", m.Name(), "with request", prototext.Format(req))
+    futurereq := &pb.Future{}
+    resp := futurereq.ProtoReflect().New().Interface()
+    if err := conn.Invoke(ctx, "/RaftAdmin/" + string(m.Name()), req, resp); err != nil {
+        add_voter_logger.Error("Error invoking", "error", err)
+        return err
+    }
+
+    // This method returned a future. We should call Await to get the result, and then Forget to free up the memory of the server.
+    if f, ok := resp.(*pb.Future); ok {
+        c := pb.NewRaftAdminClient(conn)
+        add_voter_logger.Debug("Awaiting for response")
+        resp, err := c.Await(ctx, f)
+        add_voter_logger.Debug("Response from AddVoter:", "Response", prototext.Format(resp))
+
+        if resp.Error != "" {
+            // handle error
+            add_voter_logger.Error("Error in AddVoter Response:", "error", resp.Error)
+            return errors.New("Error in AddVoter Response: target is not the leader")
+        }
+        if err != nil {
+            add_voter_logger.Error("Error from AddVoter:", "error", err)
+            return err
+        }
+        if _, err := c.Forget(ctx, f); err != nil {
+            add_voter_logger.Error("Returning error", "error", err)
+            return err
+        }
+    }
+    return nil
+}
diff --git a/jina/serve/consensus/docarray-go-proto/docarray.pb.go b/jina/serve/consensus/docarray-go-proto/docarray.pb.go
new file mode 100644
index 0000000000000..b9529a7cd236a
--- /dev/null
+++ b/jina/serve/consensus/docarray-go-proto/docarray.pb.go
@@ -0,0 +1,1200 @@
+// Code generated by protoc-gen-go. DO NOT EDIT.
+// versions:
+// 	protoc-gen-go v1.28.1
+// 	protoc        v3.13.0
+// source: docarray.proto
+
+package docarray_go_proto
+
+import (
+	_struct "github.com/golang/protobuf/ptypes/struct"
+	protoreflect "google.golang.org/protobuf/reflect/protoreflect"
+	protoimpl "google.golang.org/protobuf/runtime/protoimpl"
+	reflect "reflect"
+	sync "sync"
+)
+
+const (
+	// Verify that this generated code is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
+	// Verify that runtime/protoimpl is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
+)
+
+//*
+// Represents a (quantized) dense n-dim array
+type DenseNdArrayProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// the actual array data, in bytes
+	Buffer []byte `protobuf:"bytes,1,opt,name=buffer,proto3" json:"buffer,omitempty"`
+	// the shape (dimensions) of the array
+	Shape []uint32 `protobuf:"varint,2,rep,packed,name=shape,proto3" json:"shape,omitempty"`
+	// the data type of the array
+	Dtype string `protobuf:"bytes,3,opt,name=dtype,proto3" json:"dtype,omitempty"`
+}
+
+func (x *DenseNdArrayProto) Reset() {
+	*x = DenseNdArrayProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[0]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *DenseNdArrayProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*DenseNdArrayProto) ProtoMessage() {}
+
+func (x *DenseNdArrayProto) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[0]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use DenseNdArrayProto.ProtoReflect.Descriptor instead.
+func (*DenseNdArrayProto) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{0}
+}
+
+func (x *DenseNdArrayProto) GetBuffer() []byte {
+	if x != nil {
+		return x.Buffer
+	}
+	return nil
+}
+
+func (x *DenseNdArrayProto) GetShape() []uint32 {
+	if x != nil {
+		return x.Shape
+	}
+	return nil
+}
+
+func (x *DenseNdArrayProto) GetDtype() string {
+	if x != nil {
+		return x.Dtype
+	}
+	return ""
+}
+
+//*
+// Represents a general n-dim array, can be either dense or sparse
+type NdArrayProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Dense      *DenseNdArrayProto `protobuf:"bytes,1,opt,name=dense,proto3" json:"dense,omitempty"` // dense representation of the ndarray
+	Parameters *_struct.Struct    `protobuf:"bytes,2,opt,name=parameters,proto3" json:"parameters,omitempty"`
+}
+
+func (x *NdArrayProto) Reset() {
+	*x = NdArrayProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[1]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *NdArrayProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*NdArrayProto) ProtoMessage() {}
+
+func (x *NdArrayProto) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[1]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use NdArrayProto.ProtoReflect.Descriptor instead.
+func (*NdArrayProto) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{1}
+}
+
+func (x *NdArrayProto) GetDense() *DenseNdArrayProto {
+	if x != nil {
+		return x.Dense
+	}
+	return nil
+}
+
+func (x *NdArrayProto) GetParameters() *_struct.Struct {
+	if x != nil {
+		return x.Parameters
+	}
+	return nil
+}
+
+type KeyValuePair struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Key   *_struct.Value `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
+	Value *_struct.Value `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"`
+}
+
+func (x *KeyValuePair) Reset() {
+	*x = KeyValuePair{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[2]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *KeyValuePair) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*KeyValuePair) ProtoMessage() {}
+
+func (x *KeyValuePair) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[2]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use KeyValuePair.ProtoReflect.Descriptor instead.
+func (*KeyValuePair) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{2}
+}
+
+func (x *KeyValuePair) GetKey() *_struct.Value {
+	if x != nil {
+		return x.Key
+	}
+	return nil
+}
+
+func (x *KeyValuePair) GetValue() *_struct.Value {
+	if x != nil {
+		return x.Value
+	}
+	return nil
+}
+
+type GenericDictValue struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Entries []*KeyValuePair `protobuf:"bytes,1,rep,name=entries,proto3" json:"entries,omitempty"`
+}
+
+func (x *GenericDictValue) Reset() {
+	*x = GenericDictValue{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[3]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *GenericDictValue) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*GenericDictValue) ProtoMessage() {}
+
+func (x *GenericDictValue) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[3]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use GenericDictValue.ProtoReflect.Descriptor instead.
+func (*GenericDictValue) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{3}
+}
+
+func (x *GenericDictValue) GetEntries() []*KeyValuePair {
+	if x != nil {
+		return x.Entries
+	}
+	return nil
+}
+
+type NodeProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// Types that are assignable to Content:
+	//	*NodeProto_Text
+	//	*NodeProto_Integer
+	//	*NodeProto_Float
+	//	*NodeProto_Boolean
+	//	*NodeProto_Blob
+	//	*NodeProto_Ndarray
+	//	*NodeProto_Doc
+	//	*NodeProto_DocArray
+	//	*NodeProto_List
+	//	*NodeProto_Set
+	//	*NodeProto_Tuple
+	//	*NodeProto_Dict
+	Content isNodeProto_Content `protobuf_oneof:"content"`
+	// Types that are assignable to DocarrayType:
+	//	*NodeProto_Type
+	DocarrayType isNodeProto_DocarrayType `protobuf_oneof:"docarray_type"`
+}
+
+func (x *NodeProto) Reset() {
+	*x = NodeProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[4]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *NodeProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*NodeProto) ProtoMessage() {}
+
+func (x *NodeProto) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[4]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use NodeProto.ProtoReflect.Descriptor instead.
+func (*NodeProto) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{4}
+}
+
+func (m *NodeProto) GetContent() isNodeProto_Content {
+	if m != nil {
+		return m.Content
+	}
+	return nil
+}
+
+func (x *NodeProto) GetText() string {
+	if x, ok := x.GetContent().(*NodeProto_Text); ok {
+		return x.Text
+	}
+	return ""
+}
+
+func (x *NodeProto) GetInteger() int32 {
+	if x, ok := x.GetContent().(*NodeProto_Integer); ok {
+		return x.Integer
+	}
+	return 0
+}
+
+func (x *NodeProto) GetFloat() float64 {
+	if x, ok := x.GetContent().(*NodeProto_Float); ok {
+		return x.Float
+	}
+	return 0
+}
+
+func (x *NodeProto) GetBoolean() bool {
+	if x, ok := x.GetContent().(*NodeProto_Boolean); ok {
+		return x.Boolean
+	}
+	return false
+}
+
+func (x *NodeProto) GetBlob() []byte {
+	if x, ok := x.GetContent().(*NodeProto_Blob); ok {
+		return x.Blob
+	}
+	return nil
+}
+
+func (x *NodeProto) GetNdarray() *NdArrayProto {
+	if x, ok := x.GetContent().(*NodeProto_Ndarray); ok {
+		return x.Ndarray
+	}
+	return nil
+}
+
+func (x *NodeProto) GetDoc() *DocProto {
+	if x, ok := x.GetContent().(*NodeProto_Doc); ok {
+		return x.Doc
+	}
+	return nil
+}
+
+func (x *NodeProto) GetDocArray() *DocListProto {
+	if x, ok := x.GetContent().(*NodeProto_DocArray); ok {
+		return x.DocArray
+	}
+	return nil
+}
+
+func (x *NodeProto) GetList() *ListOfAnyProto {
+	if x, ok := x.GetContent().(*NodeProto_List); ok {
+		return x.List
+	}
+	return nil
+}
+
+func (x *NodeProto) GetSet() *ListOfAnyProto {
+	if x, ok := x.GetContent().(*NodeProto_Set); ok {
+		return x.Set
+	}
+	return nil
+}
+
+func (x *NodeProto) GetTuple() *ListOfAnyProto {
+	if x, ok := x.GetContent().(*NodeProto_Tuple); ok {
+		return x.Tuple
+	}
+	return nil
+}
+
+func (x *NodeProto) GetDict() *DictOfAnyProto {
+	if x, ok := x.GetContent().(*NodeProto_Dict); ok {
+		return x.Dict
+	}
+	return nil
+}
+
+func (m *NodeProto) GetDocarrayType() isNodeProto_DocarrayType {
+	if m != nil {
+		return m.DocarrayType
+	}
+	return nil
+}
+
+func (x *NodeProto) GetType() string {
+	if x, ok := x.GetDocarrayType().(*NodeProto_Type); ok {
+		return x.Type
+	}
+	return ""
+}
+
+type isNodeProto_Content interface {
+	isNodeProto_Content()
+}
+
+type NodeProto_Text struct {
+	// a text
+	Text string `protobuf:"bytes,1,opt,name=text,proto3,oneof"`
+}
+
+type NodeProto_Integer struct {
+	//an integer
+	Integer int32 `protobuf:"varint,2,opt,name=integer,proto3,oneof"`
+}
+
+type NodeProto_Float struct {
+	// a float
+	Float float64 `protobuf:"fixed64,3,opt,name=float,proto3,oneof"`
+}
+
+type NodeProto_Boolean struct {
+	//a boolean
+	Boolean bool `protobuf:"varint,4,opt,name=boolean,proto3,oneof"`
+}
+
+type NodeProto_Blob struct {
+	// a bytes representation
+	Blob []byte `protobuf:"bytes,5,opt,name=blob,proto3,oneof"`
+}
+
+type NodeProto_Ndarray struct {
+	// the ndarray of the image/audio/video document
+	Ndarray *NdArrayProto `protobuf:"bytes,6,opt,name=ndarray,proto3,oneof"`
+}
+
+type NodeProto_Doc struct {
+	// a sub Document
+	Doc *DocProto `protobuf:"bytes,7,opt,name=doc,proto3,oneof"`
+}
+
+type NodeProto_DocArray struct {
+	// a sub DocArray
+	DocArray *DocListProto `protobuf:"bytes,8,opt,name=doc_array,json=docArray,proto3,oneof"`
+}
+
+type NodeProto_List struct {
+	//any list
+	List *ListOfAnyProto `protobuf:"bytes,9,opt,name=list,proto3,oneof"`
+}
+
+type NodeProto_Set struct {
+	//any set
+	Set *ListOfAnyProto `protobuf:"bytes,10,opt,name=set,proto3,oneof"`
+}
+
+type NodeProto_Tuple struct {
+	//any tuple
+	Tuple *ListOfAnyProto `protobuf:"bytes,11,opt,name=tuple,proto3,oneof"`
+}
+
+type NodeProto_Dict struct {
+	// dictionary with string as keys
+	Dict *DictOfAnyProto `protobuf:"bytes,12,opt,name=dict,proto3,oneof"`
+}
+
+func (*NodeProto_Text) isNodeProto_Content() {}
+
+func (*NodeProto_Integer) isNodeProto_Content() {}
+
+func (*NodeProto_Float) isNodeProto_Content() {}
+
+func (*NodeProto_Boolean) isNodeProto_Content() {}
+
+func (*NodeProto_Blob) isNodeProto_Content() {}
+
+func (*NodeProto_Ndarray) isNodeProto_Content() {}
+
+func (*NodeProto_Doc) isNodeProto_Content() {}
+
+func (*NodeProto_DocArray) isNodeProto_Content() {}
+
+func (*NodeProto_List) isNodeProto_Content() {}
+
+func (*NodeProto_Set) isNodeProto_Content() {}
+
+func (*NodeProto_Tuple) isNodeProto_Content() {}
+
+func (*NodeProto_Dict) isNodeProto_Content() {}
+
+type isNodeProto_DocarrayType interface {
+	isNodeProto_DocarrayType()
+}
+
+type NodeProto_Type struct {
+	Type string `protobuf:"bytes,13,opt,name=type,proto3,oneof"`
+}
+
+func (*NodeProto_Type) isNodeProto_DocarrayType() {}
+
+//*
+// Represents a Document
+type DocProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Data map[string]*NodeProto `protobuf:"bytes,1,rep,name=data,proto3" json:"data,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
+}
+
+func (x *DocProto) Reset() {
+	*x = DocProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[5]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *DocProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*DocProto) ProtoMessage() {}
+
+func (x *DocProto) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[5]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use DocProto.ProtoReflect.Descriptor instead.
+func (*DocProto) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{5}
+}
+
+func (x *DocProto) GetData() map[string]*NodeProto {
+	if x != nil {
+		return x.Data
+	}
+	return nil
+}
+
+type DictOfAnyProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Data map[string]*NodeProto `protobuf:"bytes,1,rep,name=data,proto3" json:"data,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
+}
+
+func (x *DictOfAnyProto) Reset() {
+	*x = DictOfAnyProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[6]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *DictOfAnyProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*DictOfAnyProto) ProtoMessage() {}
+
+func (x *DictOfAnyProto) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[6]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use DictOfAnyProto.ProtoReflect.Descriptor instead.
+func (*DictOfAnyProto) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{6}
+}
+
+func (x *DictOfAnyProto) GetData() map[string]*NodeProto {
+	if x != nil {
+		return x.Data
+	}
+	return nil
+}
+
+type ListOfAnyProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Data []*NodeProto `protobuf:"bytes,1,rep,name=data,proto3" json:"data,omitempty"`
+}
+
+func (x *ListOfAnyProto) Reset() {
+	*x = ListOfAnyProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[7]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *ListOfAnyProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ListOfAnyProto) ProtoMessage() {}
+
+func (x *ListOfAnyProto) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[7]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ListOfAnyProto.ProtoReflect.Descriptor instead.
+func (*ListOfAnyProto) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{7}
+}
+
+func (x *ListOfAnyProto) GetData() []*NodeProto {
+	if x != nil {
+		return x.Data
+	}
+	return nil
+}
+
+type DocListProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Docs []*DocProto `protobuf:"bytes,1,rep,name=docs,proto3" json:"docs,omitempty"` // a list of Documents
+}
+
+func (x *DocListProto) Reset() {
+	*x = DocListProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[8]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *DocListProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*DocListProto) ProtoMessage() {}
+
+func (x *DocListProto) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[8]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use DocListProto.ProtoReflect.Descriptor instead.
+func (*DocListProto) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{8}
+}
+
+func (x *DocListProto) GetDocs() []*DocProto {
+	if x != nil {
+		return x.Docs
+	}
+	return nil
+}
+
+type ListOfDocArrayProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Data []*DocListProto `protobuf:"bytes,1,rep,name=data,proto3" json:"data,omitempty"`
+}
+
+func (x *ListOfDocArrayProto) Reset() {
+	*x = ListOfDocArrayProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[9]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *ListOfDocArrayProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ListOfDocArrayProto) ProtoMessage() {}
+
+func (x *ListOfDocArrayProto) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[9]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ListOfDocArrayProto.ProtoReflect.Descriptor instead.
+func (*ListOfDocArrayProto) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{9}
+}
+
+func (x *ListOfDocArrayProto) GetData() []*DocListProto {
+	if x != nil {
+		return x.Data
+	}
+	return nil
+}
+
+type DocVecProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	TensorColumns  map[string]*NdArrayProto        `protobuf:"bytes,1,rep,name=tensor_columns,json=tensorColumns,proto3" json:"tensor_columns,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`      // a dict of document columns
+	DocColumns     map[string]*DocVecProto         `protobuf:"bytes,2,rep,name=doc_columns,json=docColumns,proto3" json:"doc_columns,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`               // a dict of tensor columns
+	DocsVecColumns map[string]*ListOfDocArrayProto `protobuf:"bytes,3,rep,name=docs_vec_columns,json=docsVecColumns,proto3" json:"docs_vec_columns,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // a dict of document array columns
+	AnyColumns     map[string]*ListOfAnyProto      `protobuf:"bytes,4,rep,name=any_columns,json=anyColumns,proto3" json:"any_columns,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`               // a dict of any columns. Used for the rest of the data
+}
+
+func (x *DocVecProto) Reset() {
+	*x = DocVecProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_docarray_proto_msgTypes[10]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *DocVecProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*DocVecProto) ProtoMessage() {}
+
+func (x *DocVecProto) ProtoReflect() protoreflect.Message {
+	mi := &file_docarray_proto_msgTypes[10]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use DocVecProto.ProtoReflect.Descriptor instead.
+func (*DocVecProto) Descriptor() ([]byte, []int) {
+	return file_docarray_proto_rawDescGZIP(), []int{10}
+}
+
+func (x *DocVecProto) GetTensorColumns() map[string]*NdArrayProto {
+	if x != nil {
+		return x.TensorColumns
+	}
+	return nil
+}
+
+func (x *DocVecProto) GetDocColumns() map[string]*DocVecProto {
+	if x != nil {
+		return x.DocColumns
+	}
+	return nil
+}
+
+func (x *DocVecProto) GetDocsVecColumns() map[string]*ListOfDocArrayProto {
+	if x != nil {
+		return x.DocsVecColumns
+	}
+	return nil
+}
+
+func (x *DocVecProto) GetAnyColumns() map[string]*ListOfAnyProto {
+	if x != nil {
+		return x.AnyColumns
+	}
+	return nil
+}
+
+var File_docarray_proto protoreflect.FileDescriptor
+
+var file_docarray_proto_rawDesc = []byte{
+	0x0a, 0x0e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f,
+	0x12, 0x08, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x1a, 0x1c, 0x67, 0x6f, 0x6f, 0x67,
+	0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x73, 0x74, 0x72, 0x75,
+	0x63, 0x74, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x57, 0x0a, 0x11, 0x44, 0x65, 0x6e, 0x73,
+	0x65, 0x4e, 0x64, 0x41, 0x72, 0x72, 0x61, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x16, 0x0a,
+	0x06, 0x62, 0x75, 0x66, 0x66, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x06, 0x62,
+	0x75, 0x66, 0x66, 0x65, 0x72, 0x12, 0x14, 0x0a, 0x05, 0x73, 0x68, 0x61, 0x70, 0x65, 0x18, 0x02,
+	0x20, 0x03, 0x28, 0x0d, 0x52, 0x05, 0x73, 0x68, 0x61, 0x70, 0x65, 0x12, 0x14, 0x0a, 0x05, 0x64,
+	0x74, 0x79, 0x70, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x64, 0x74, 0x79, 0x70,
+	0x65, 0x22, 0x7a, 0x0a, 0x0c, 0x4e, 0x64, 0x41, 0x72, 0x72, 0x61, 0x79, 0x50, 0x72, 0x6f, 0x74,
+	0x6f, 0x12, 0x31, 0x0a, 0x05, 0x64, 0x65, 0x6e, 0x73, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b,
+	0x32, 0x1b, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x44, 0x65, 0x6e, 0x73,
+	0x65, 0x4e, 0x64, 0x41, 0x72, 0x72, 0x61, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x05, 0x64,
+	0x65, 0x6e, 0x73, 0x65, 0x12, 0x37, 0x0a, 0x0a, 0x70, 0x61, 0x72, 0x61, 0x6d, 0x65, 0x74, 0x65,
+	0x72, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c,
+	0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x53, 0x74, 0x72, 0x75, 0x63,
+	0x74, 0x52, 0x0a, 0x70, 0x61, 0x72, 0x61, 0x6d, 0x65, 0x74, 0x65, 0x72, 0x73, 0x22, 0x66, 0x0a,
+	0x0c, 0x4b, 0x65, 0x79, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x50, 0x61, 0x69, 0x72, 0x12, 0x28, 0x0a,
+	0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x67, 0x6f, 0x6f,
+	0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x56, 0x61, 0x6c,
+	0x75, 0x65, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x2c, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65,
+	0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e,
+	0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05,
+	0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, 0x44, 0x0a, 0x10, 0x47, 0x65, 0x6e, 0x65, 0x72, 0x69, 0x63,
+	0x44, 0x69, 0x63, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, 0x30, 0x0a, 0x07, 0x65, 0x6e, 0x74,
+	0x72, 0x69, 0x65, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x64, 0x6f, 0x63,
+	0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x4b, 0x65, 0x79, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x50, 0x61,
+	0x69, 0x72, 0x52, 0x07, 0x65, 0x6e, 0x74, 0x72, 0x69, 0x65, 0x73, 0x22, 0x8c, 0x04, 0x0a, 0x09,
+	0x4e, 0x6f, 0x64, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x14, 0x0a, 0x04, 0x74, 0x65, 0x78,
+	0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x48, 0x00, 0x52, 0x04, 0x74, 0x65, 0x78, 0x74, 0x12,
+	0x1a, 0x0a, 0x07, 0x69, 0x6e, 0x74, 0x65, 0x67, 0x65, 0x72, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05,
+	0x48, 0x00, 0x52, 0x07, 0x69, 0x6e, 0x74, 0x65, 0x67, 0x65, 0x72, 0x12, 0x16, 0x0a, 0x05, 0x66,
+	0x6c, 0x6f, 0x61, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x01, 0x48, 0x00, 0x52, 0x05, 0x66, 0x6c,
+	0x6f, 0x61, 0x74, 0x12, 0x1a, 0x0a, 0x07, 0x62, 0x6f, 0x6f, 0x6c, 0x65, 0x61, 0x6e, 0x18, 0x04,
+	0x20, 0x01, 0x28, 0x08, 0x48, 0x00, 0x52, 0x07, 0x62, 0x6f, 0x6f, 0x6c, 0x65, 0x61, 0x6e, 0x12,
+	0x14, 0x0a, 0x04, 0x62, 0x6c, 0x6f, 0x62, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0c, 0x48, 0x00, 0x52,
+	0x04, 0x62, 0x6c, 0x6f, 0x62, 0x12, 0x32, 0x0a, 0x07, 0x6e, 0x64, 0x61, 0x72, 0x72, 0x61, 0x79,
+	0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61,
+	0x79, 0x2e, 0x4e, 0x64, 0x41, 0x72, 0x72, 0x61, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x48, 0x00,
+	0x52, 0x07, 0x6e, 0x64, 0x61, 0x72, 0x72, 0x61, 0x79, 0x12, 0x26, 0x0a, 0x03, 0x64, 0x6f, 0x63,
+	0x18, 0x07, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61,
+	0x79, 0x2e, 0x44, 0x6f, 0x63, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x48, 0x00, 0x52, 0x03, 0x64, 0x6f,
+	0x63, 0x12, 0x35, 0x0a, 0x09, 0x64, 0x6f, 0x63, 0x5f, 0x61, 0x72, 0x72, 0x61, 0x79, 0x18, 0x08,
+	0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e,
+	0x44, 0x6f, 0x63, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x48, 0x00, 0x52, 0x08,
+	0x64, 0x6f, 0x63, 0x41, 0x72, 0x72, 0x61, 0x79, 0x12, 0x2e, 0x0a, 0x04, 0x6c, 0x69, 0x73, 0x74,
+	0x18, 0x09, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61,
+	0x79, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x4f, 0x66, 0x41, 0x6e, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f,
+	0x48, 0x00, 0x52, 0x04, 0x6c, 0x69, 0x73, 0x74, 0x12, 0x2c, 0x0a, 0x03, 0x73, 0x65, 0x74, 0x18,
+	0x0a, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79,
+	0x2e, 0x4c, 0x69, 0x73, 0x74, 0x4f, 0x66, 0x41, 0x6e, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x48,
+	0x00, 0x52, 0x03, 0x73, 0x65, 0x74, 0x12, 0x30, 0x0a, 0x05, 0x74, 0x75, 0x70, 0x6c, 0x65, 0x18,
+	0x0b, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79,
+	0x2e, 0x4c, 0x69, 0x73, 0x74, 0x4f, 0x66, 0x41, 0x6e, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x48,
+	0x00, 0x52, 0x05, 0x74, 0x75, 0x70, 0x6c, 0x65, 0x12, 0x2e, 0x0a, 0x04, 0x64, 0x69, 0x63, 0x74,
+	0x18, 0x0c, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61,
+	0x79, 0x2e, 0x44, 0x69, 0x63, 0x74, 0x4f, 0x66, 0x41, 0x6e, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f,
+	0x48, 0x00, 0x52, 0x04, 0x64, 0x69, 0x63, 0x74, 0x12, 0x14, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65,
+	0x18, 0x0d, 0x20, 0x01, 0x28, 0x09, 0x48, 0x01, 0x52, 0x04, 0x74, 0x79, 0x70, 0x65, 0x42, 0x09,
+	0x0a, 0x07, 0x63, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x42, 0x0f, 0x0a, 0x0d, 0x64, 0x6f, 0x63,
+	0x61, 0x72, 0x72, 0x61, 0x79, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x22, 0x8a, 0x01, 0x0a, 0x08, 0x44,
+	0x6f, 0x63, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x30, 0x0a, 0x04, 0x64, 0x61, 0x74, 0x61, 0x18,
+	0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79,
+	0x2e, 0x44, 0x6f, 0x63, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x45, 0x6e,
+	0x74, 0x72, 0x79, 0x52, 0x04, 0x64, 0x61, 0x74, 0x61, 0x1a, 0x4c, 0x0a, 0x09, 0x44, 0x61, 0x74,
+	0x61, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20,
+	0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x29, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75,
+	0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x13, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72,
+	0x61, 0x79, 0x2e, 0x4e, 0x6f, 0x64, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x05, 0x76, 0x61,
+	0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x96, 0x01, 0x0a, 0x0e, 0x44, 0x69, 0x63, 0x74,
+	0x4f, 0x66, 0x41, 0x6e, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x36, 0x0a, 0x04, 0x64, 0x61,
+	0x74, 0x61, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72,
+	0x72, 0x61, 0x79, 0x2e, 0x44, 0x69, 0x63, 0x74, 0x4f, 0x66, 0x41, 0x6e, 0x79, 0x50, 0x72, 0x6f,
+	0x74, 0x6f, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x04, 0x64, 0x61,
+	0x74, 0x61, 0x1a, 0x4c, 0x0a, 0x09, 0x44, 0x61, 0x74, 0x61, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12,
+	0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65,
+	0x79, 0x12, 0x29, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b,
+	0x32, 0x13, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x4e, 0x6f, 0x64, 0x65,
+	0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01,
+	0x22, 0x39, 0x0a, 0x0e, 0x4c, 0x69, 0x73, 0x74, 0x4f, 0x66, 0x41, 0x6e, 0x79, 0x50, 0x72, 0x6f,
+	0x74, 0x6f, 0x12, 0x27, 0x0a, 0x04, 0x64, 0x61, 0x74, 0x61, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b,
+	0x32, 0x13, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x4e, 0x6f, 0x64, 0x65,
+	0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x04, 0x64, 0x61, 0x74, 0x61, 0x22, 0x36, 0x0a, 0x0c, 0x44,
+	0x6f, 0x63, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x26, 0x0a, 0x04, 0x64,
+	0x6f, 0x63, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x64, 0x6f, 0x63, 0x61,
+	0x72, 0x72, 0x61, 0x79, 0x2e, 0x44, 0x6f, 0x63, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x04, 0x64,
+	0x6f, 0x63, 0x73, 0x22, 0x41, 0x0a, 0x13, 0x4c, 0x69, 0x73, 0x74, 0x4f, 0x66, 0x44, 0x6f, 0x63,
+	0x41, 0x72, 0x72, 0x61, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x2a, 0x0a, 0x04, 0x64, 0x61,
+	0x74, 0x61, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72,
+	0x72, 0x61, 0x79, 0x2e, 0x44, 0x6f, 0x63, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f,
+	0x52, 0x04, 0x64, 0x61, 0x74, 0x61, 0x22, 0xae, 0x05, 0x0a, 0x0b, 0x44, 0x6f, 0x63, 0x56, 0x65,
+	0x63, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x4f, 0x0a, 0x0e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72,
+	0x5f, 0x63, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x28,
+	0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x44, 0x6f, 0x63, 0x56, 0x65, 0x63,
+	0x50, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x43, 0x6f, 0x6c, 0x75,
+	0x6d, 0x6e, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x0d, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72,
+	0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x12, 0x46, 0x0a, 0x0b, 0x64, 0x6f, 0x63, 0x5f, 0x63,
+	0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x25, 0x2e, 0x64,
+	0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x44, 0x6f, 0x63, 0x56, 0x65, 0x63, 0x50, 0x72,
+	0x6f, 0x74, 0x6f, 0x2e, 0x44, 0x6f, 0x63, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x45, 0x6e,
+	0x74, 0x72, 0x79, 0x52, 0x0a, 0x64, 0x6f, 0x63, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x12,
+	0x53, 0x0a, 0x10, 0x64, 0x6f, 0x63, 0x73, 0x5f, 0x76, 0x65, 0x63, 0x5f, 0x63, 0x6f, 0x6c, 0x75,
+	0x6d, 0x6e, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x29, 0x2e, 0x64, 0x6f, 0x63, 0x61,
+	0x72, 0x72, 0x61, 0x79, 0x2e, 0x44, 0x6f, 0x63, 0x56, 0x65, 0x63, 0x50, 0x72, 0x6f, 0x74, 0x6f,
+	0x2e, 0x44, 0x6f, 0x63, 0x73, 0x56, 0x65, 0x63, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x45,
+	0x6e, 0x74, 0x72, 0x79, 0x52, 0x0e, 0x64, 0x6f, 0x63, 0x73, 0x56, 0x65, 0x63, 0x43, 0x6f, 0x6c,
+	0x75, 0x6d, 0x6e, 0x73, 0x12, 0x46, 0x0a, 0x0b, 0x61, 0x6e, 0x79, 0x5f, 0x63, 0x6f, 0x6c, 0x75,
+	0x6d, 0x6e, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x25, 0x2e, 0x64, 0x6f, 0x63, 0x61,
+	0x72, 0x72, 0x61, 0x79, 0x2e, 0x44, 0x6f, 0x63, 0x56, 0x65, 0x63, 0x50, 0x72, 0x6f, 0x74, 0x6f,
+	0x2e, 0x41, 0x6e, 0x79, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79,
+	0x52, 0x0a, 0x61, 0x6e, 0x79, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x1a, 0x58, 0x0a, 0x12,
+	0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x45, 0x6e, 0x74,
+	0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52,
+	0x03, 0x6b, 0x65, 0x79, 0x12, 0x2c, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20,
+	0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x4e,
+	0x64, 0x41, 0x72, 0x72, 0x61, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x05, 0x76, 0x61, 0x6c,
+	0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x1a, 0x54, 0x0a, 0x0f, 0x44, 0x6f, 0x63, 0x43, 0x6f, 0x6c,
+	0x75, 0x6d, 0x6e, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79,
+	0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x2b, 0x0a, 0x05, 0x76,
+	0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x64, 0x6f, 0x63,
+	0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x44, 0x6f, 0x63, 0x56, 0x65, 0x63, 0x50, 0x72, 0x6f, 0x74,
+	0x6f, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x1a, 0x60, 0x0a, 0x13,
+	0x44, 0x6f, 0x63, 0x73, 0x56, 0x65, 0x63, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x45, 0x6e,
+	0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09,
+	0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x33, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02,
+	0x20, 0x01, 0x28, 0x0b, 0x32, 0x1d, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e,
+	0x4c, 0x69, 0x73, 0x74, 0x4f, 0x66, 0x44, 0x6f, 0x63, 0x41, 0x72, 0x72, 0x61, 0x79, 0x50, 0x72,
+	0x6f, 0x74, 0x6f, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x1a, 0x57,
+	0x0a, 0x0f, 0x41, 0x6e, 0x79, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x73, 0x45, 0x6e, 0x74, 0x72,
+	0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03,
+	0x6b, 0x65, 0x79, 0x12, 0x2e, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01,
+	0x28, 0x0b, 0x32, 0x18, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x4c, 0x69,
+	0x73, 0x74, 0x4f, 0x66, 0x41, 0x6e, 0x79, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x05, 0x76, 0x61,
+	0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x42, 0x19, 0x5a, 0x17, 0x6a, 0x72, 0x61, 0x66, 0x74,
+	0x2f, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2d, 0x67, 0x6f, 0x2d, 0x70, 0x72, 0x6f,
+	0x74, 0x6f, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
+}
+
+var (
+	file_docarray_proto_rawDescOnce sync.Once
+	file_docarray_proto_rawDescData = file_docarray_proto_rawDesc
+)
+
+func file_docarray_proto_rawDescGZIP() []byte {
+	file_docarray_proto_rawDescOnce.Do(func() {
+		file_docarray_proto_rawDescData = protoimpl.X.CompressGZIP(file_docarray_proto_rawDescData)
+	})
+	return file_docarray_proto_rawDescData
+}
+
+var file_docarray_proto_msgTypes = make([]protoimpl.MessageInfo, 17)
+var file_docarray_proto_goTypes = []interface{}{
+	(*DenseNdArrayProto)(nil),   // 0: docarray.DenseNdArrayProto
+	(*NdArrayProto)(nil),        // 1: docarray.NdArrayProto
+	(*KeyValuePair)(nil),        // 2: docarray.KeyValuePair
+	(*GenericDictValue)(nil),    // 3: docarray.GenericDictValue
+	(*NodeProto)(nil),           // 4: docarray.NodeProto
+	(*DocProto)(nil),            // 5: docarray.DocProto
+	(*DictOfAnyProto)(nil),      // 6: docarray.DictOfAnyProto
+	(*ListOfAnyProto)(nil),      // 7: docarray.ListOfAnyProto
+	(*DocListProto)(nil),        // 8: docarray.DocListProto
+	(*ListOfDocArrayProto)(nil), // 9: docarray.ListOfDocArrayProto
+	(*DocVecProto)(nil),         // 10: docarray.DocVecProto
+	nil,                         // 11: docarray.DocProto.DataEntry
+	nil,                         // 12: docarray.DictOfAnyProto.DataEntry
+	nil,                         // 13: docarray.DocVecProto.TensorColumnsEntry
+	nil,                         // 14: docarray.DocVecProto.DocColumnsEntry
+	nil,                         // 15: docarray.DocVecProto.DocsVecColumnsEntry
+	nil,                         // 16: docarray.DocVecProto.AnyColumnsEntry
+	(*_struct.Struct)(nil),      // 17: google.protobuf.Struct
+	(*_struct.Value)(nil),       // 18: google.protobuf.Value
+}
+var file_docarray_proto_depIdxs = []int32{
+	0,  // 0: docarray.NdArrayProto.dense:type_name -> docarray.DenseNdArrayProto
+	17, // 1: docarray.NdArrayProto.parameters:type_name -> google.protobuf.Struct
+	18, // 2: docarray.KeyValuePair.key:type_name -> google.protobuf.Value
+	18, // 3: docarray.KeyValuePair.value:type_name -> google.protobuf.Value
+	2,  // 4: docarray.GenericDictValue.entries:type_name -> docarray.KeyValuePair
+	1,  // 5: docarray.NodeProto.ndarray:type_name -> docarray.NdArrayProto
+	5,  // 6: docarray.NodeProto.doc:type_name -> docarray.DocProto
+	8,  // 7: docarray.NodeProto.doc_array:type_name -> docarray.DocListProto
+	7,  // 8: docarray.NodeProto.list:type_name -> docarray.ListOfAnyProto
+	7,  // 9: docarray.NodeProto.set:type_name -> docarray.ListOfAnyProto
+	7,  // 10: docarray.NodeProto.tuple:type_name -> docarray.ListOfAnyProto
+	6,  // 11: docarray.NodeProto.dict:type_name -> docarray.DictOfAnyProto
+	11, // 12: docarray.DocProto.data:type_name -> docarray.DocProto.DataEntry
+	12, // 13: docarray.DictOfAnyProto.data:type_name -> docarray.DictOfAnyProto.DataEntry
+	4,  // 14: docarray.ListOfAnyProto.data:type_name -> docarray.NodeProto
+	5,  // 15: docarray.DocListProto.docs:type_name -> docarray.DocProto
+	8,  // 16: docarray.ListOfDocArrayProto.data:type_name -> docarray.DocListProto
+	13, // 17: docarray.DocVecProto.tensor_columns:type_name -> docarray.DocVecProto.TensorColumnsEntry
+	14, // 18: docarray.DocVecProto.doc_columns:type_name -> docarray.DocVecProto.DocColumnsEntry
+	15, // 19: docarray.DocVecProto.docs_vec_columns:type_name -> docarray.DocVecProto.DocsVecColumnsEntry
+	16, // 20: docarray.DocVecProto.any_columns:type_name -> docarray.DocVecProto.AnyColumnsEntry
+	4,  // 21: docarray.DocProto.DataEntry.value:type_name -> docarray.NodeProto
+	4,  // 22: docarray.DictOfAnyProto.DataEntry.value:type_name -> docarray.NodeProto
+	1,  // 23: docarray.DocVecProto.TensorColumnsEntry.value:type_name -> docarray.NdArrayProto
+	10, // 24: docarray.DocVecProto.DocColumnsEntry.value:type_name -> docarray.DocVecProto
+	9,  // 25: docarray.DocVecProto.DocsVecColumnsEntry.value:type_name -> docarray.ListOfDocArrayProto
+	7,  // 26: docarray.DocVecProto.AnyColumnsEntry.value:type_name -> docarray.ListOfAnyProto
+	27, // [27:27] is the sub-list for method output_type
+	27, // [27:27] is the sub-list for method input_type
+	27, // [27:27] is the sub-list for extension type_name
+	27, // [27:27] is the sub-list for extension extendee
+	0,  // [0:27] is the sub-list for field type_name
+}
+
+func init() { file_docarray_proto_init() }
+func file_docarray_proto_init() {
+	if File_docarray_proto != nil {
+		return
+	}
+	if !protoimpl.UnsafeEnabled {
+		file_docarray_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*DenseNdArrayProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*NdArrayProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*KeyValuePair); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*GenericDictValue); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*NodeProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*DocProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*DictOfAnyProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[7].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*ListOfAnyProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[8].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*DocListProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[9].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*ListOfDocArrayProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_docarray_proto_msgTypes[10].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*DocVecProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+	}
+	file_docarray_proto_msgTypes[4].OneofWrappers = []interface{}{
+		(*NodeProto_Text)(nil),
+		(*NodeProto_Integer)(nil),
+		(*NodeProto_Float)(nil),
+		(*NodeProto_Boolean)(nil),
+		(*NodeProto_Blob)(nil),
+		(*NodeProto_Ndarray)(nil),
+		(*NodeProto_Doc)(nil),
+		(*NodeProto_DocArray)(nil),
+		(*NodeProto_List)(nil),
+		(*NodeProto_Set)(nil),
+		(*NodeProto_Tuple)(nil),
+		(*NodeProto_Dict)(nil),
+		(*NodeProto_Type)(nil),
+	}
+	type x struct{}
+	out := protoimpl.TypeBuilder{
+		File: protoimpl.DescBuilder{
+			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
+			RawDescriptor: file_docarray_proto_rawDesc,
+			NumEnums:      0,
+			NumMessages:   17,
+			NumExtensions: 0,
+			NumServices:   0,
+		},
+		GoTypes:           file_docarray_proto_goTypes,
+		DependencyIndexes: file_docarray_proto_depIdxs,
+		MessageInfos:      file_docarray_proto_msgTypes,
+	}.Build()
+	File_docarray_proto = out.File
+	file_docarray_proto_rawDesc = nil
+	file_docarray_proto_goTypes = nil
+	file_docarray_proto_depIdxs = nil
+}
diff --git a/jina/serve/consensus/docarray.proto b/jina/serve/consensus/docarray.proto
new file mode 100644
index 0000000000000..28534f99ca578
--- /dev/null
+++ b/jina/serve/consensus/docarray.proto
@@ -0,0 +1,110 @@
+//protoc --go_out docarray-go-proto --go_opt=paths=source_relative --go_opt=Mdocarray.proto=docarray-go-proto=jraft/docarray-go-proto --go-grpc_out docarray-go-proto --go-grpc_opt=paths=source_relative --experimental_allow_proto3_optional docarray.proto
+syntax = "proto3";
+import "google/protobuf/struct.proto";
+
+package docarray;
+option go_package = "jraft/docarray-go-proto";
+
+/**
+ * Represents a (quantized) dense n-dim array
+ */
+message DenseNdArrayProto {
+  // the actual array data, in bytes
+  bytes buffer = 1;
+
+  // the shape (dimensions) of the array
+  repeated uint32 shape = 2;
+
+  // the data type of the array
+  string dtype = 3;
+}
+
+/**
+ * Represents a general n-dim array, can be either dense or sparse
+ */
+message NdArrayProto {
+  DenseNdArrayProto dense = 1; // dense representation of the ndarray
+
+  google.protobuf.Struct parameters = 2;
+}
+
+message KeyValuePair {
+  google.protobuf.Value key = 1;
+  google.protobuf.Value value = 2;
+}
+
+message GenericDictValue {
+  repeated KeyValuePair entries = 1;
+}
+
+
+
+message NodeProto {
+
+  oneof content {
+    // a text
+    string text = 1;
+    //an integer
+    int32 integer = 2;
+    // a float
+    double float = 3;
+    //a boolean
+    bool boolean = 4;
+    // a bytes representation
+    bytes blob = 5;
+    // the ndarray of the image/audio/video document
+    NdArrayProto ndarray = 6;
+    // a sub Document
+    DocProto doc = 7;
+    // a sub DocArray
+    DocListProto doc_array = 8;
+    //any list
+    ListOfAnyProto list = 9;
+    //any set
+    ListOfAnyProto set = 10;
+    //any tuple
+    ListOfAnyProto tuple = 11;
+    // dictionary with string as keys
+    DictOfAnyProto dict = 12;
+  }
+
+  oneof docarray_type {
+    string type = 13;
+  }
+
+}
+
+/**
+ * Represents a Document
+ */
+message DocProto {
+
+  map<string, NodeProto> data = 1;
+
+}
+
+message DictOfAnyProto {
+
+  map<string, NodeProto> data = 1;
+
+}
+
+message ListOfAnyProto {
+  repeated NodeProto data = 1;
+}
+
+message DocListProto {
+  repeated DocProto docs = 1; // a list of Documents
+}
+
+
+message ListOfDocArrayProto {
+  repeated DocListProto data = 1;
+}
+
+message DocVecProto{
+  map<string, NdArrayProto> tensor_columns = 1; // a dict of document columns
+  map<string, DocVecProto> doc_columns = 2; // a dict of tensor columns
+  map<string, ListOfDocArrayProto> docs_vec_columns = 3; // a dict of document array columns
+  map<string, ListOfAnyProto> any_columns = 4; // a dict of any columns. Used for the rest of the data
+}
\ No newline at end of file
diff --git a/jina/serve/consensus/go.mod b/jina/serve/consensus/go.mod
new file mode 100644
index 0000000000000..cdde96d726db1
--- /dev/null
+++ b/jina/serve/consensus/go.mod
@@ -0,0 +1,30 @@
+module jraft
+
+go 1.19
+
+require (
+	github.com/Jille/raft-grpc-leader-rpc v1.1.0
+	github.com/Jille/raft-grpc-transport v1.1.1
+	github.com/Jille/raftadmin v1.2.0
+	github.com/golang/protobuf v1.5.2
+	github.com/hashicorp/go-hclog v0.16.2
+	github.com/hashicorp/raft v1.3.11
+	github.com/hashicorp/raft-boltdb v0.0.0-20220329195025-15018e9b97e0
+	google.golang.org/grpc v1.47.0
+	google.golang.org/protobuf v1.27.1
+)
+
+require (
+	github.com/armon/go-metrics v0.3.9 // indirect
+	github.com/boltdb/bolt v1.3.1 // indirect
+	github.com/fatih/color v1.12.0 // indirect
+	github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
+	github.com/hashicorp/go-msgpack v0.5.5 // indirect
+	github.com/hashicorp/golang-lru v0.5.4 // indirect
+	github.com/mattn/go-colorable v0.1.8 // indirect
+	github.com/mattn/go-isatty v0.0.14 // indirect
+	golang.org/x/net v0.0.0-20210907225631-ff17edfbf26d // indirect
+	golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34 // indirect
+	golang.org/x/text v0.3.7 // indirect
+	google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83 // indirect
+)
diff --git a/jina/serve/consensus/go.sum b/jina/serve/consensus/go.sum
new file mode 100644
index 0000000000000..51c22e36e8c99
--- /dev/null
+++ b/jina/serve/consensus/go.sum
@@ -0,0 +1,286 @@
+cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/DataDog/datadog-go v2.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3tL4fMGNddJ+vMq1mwgfaqoQ=
+github.com/DataDog/datadog-go v3.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3tL4fMGNddJ+vMq1mwgfaqoQ=
+github.com/Jille/raft-grpc-leader-rpc v1.1.0 h1:u36rmA4tjp+4FSdZ17jg/1sfSCYNQIe5bzzwvW0iVTM=
+github.com/Jille/raft-grpc-leader-rpc v1.1.0/go.mod h1:l+pK+uPuqpFDFcPmyUPSng4257UXrST0Vc3Lo4XwVB0=
+github.com/Jille/raft-grpc-transport v1.1.1 h1:VvFzYU3GuuEtehtBbgfubJnoPNBDHix0DVljV4PkGCc=
+github.com/Jille/raft-grpc-transport v1.1.1/go.mod h1:NG6sOvCFk8j7t93TTfJ40w5it4cbmB1kw0MjoEWzs6o=
+github.com/Jille/raftadmin v1.2.0 h1:hMLFUK7iKpeXP+CoIhNMWj+F53XOLSjMDSia0C60cps=
+github.com/Jille/raftadmin v1.2.0/go.mod h1:vtVEpToPGTUPVwwunypWDpi69JpdnHMhWRUlc/65U+Y=
+github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
+github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
+github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
+github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878/go.mod h1:3AMJUQhVx52RsWOnlkpikZr01T/yAVN2gn0861vByNg=
+github.com/armon/go-metrics v0.3.8/go.mod h1:4O98XIr/9W0sxpJ8UaYkvjk10Iff7SnFrb4QAOwNTFc=
+github.com/armon/go-metrics v0.3.9 h1:O2sNqxBdvq8Eq5xmzljcYzAORli6RWCvEym4cJf9m18=
+github.com/armon/go-metrics v0.3.9/go.mod h1:4O98XIr/9W0sxpJ8UaYkvjk10Iff7SnFrb4QAOwNTFc=
+github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
+github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
+github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4=
+github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps=
+github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
+github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
+github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible/go.mod h1:nmEj6Dob7S7YxXgwXpfOuvO54S+tGdZdw9fuRZt25Ag=
+github.com/circonus-labs/circonusllhist v0.1.3/go.mod h1:kMXHVDlOchFAehlya5ePtbp5jckzBHf4XRpQvBOLI+I=
+github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
+github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
+github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI=
+github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/cncf/xds/go v0.0.0-20211001041855-01bcc9b48dfe/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
+github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
+github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
+github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
+github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
+github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1/go.mod h1:KJwIaB5Mv44NWtYuAOFCVOjcI94vtpEz2JU/D2v6IjE=
+github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
+github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
+github.com/fatih/color v1.12.0 h1:mRhaKNwANqRgUBGKmnI5ZxEk7QXmjQeCcuYFMX2bfcc=
+github.com/fatih/color v1.12.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGEBuJM=
+github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
+github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
+github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
+github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
+github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
+github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
+github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
+github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
+github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
+github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
+github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
+github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
+github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
+github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
+github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
+github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
+github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
+github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
+github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
+github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
+github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
+github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
+github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
+github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
+github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.6 h1:BKbKCqvP6I+rmFHt06ZmyQtvB8xAkWdhFyr0ZUNZcxQ=
+github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
+github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
+github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
+github.com/hashicorp/go-hclog v0.9.1/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
+github.com/hashicorp/go-hclog v0.16.2 h1:K4ev2ib4LdQETX5cSZBG0DVLk1jwGqSPXBjdah3veNs=
+github.com/hashicorp/go-hclog v0.16.2/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
+github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
+github.com/hashicorp/go-immutable-radix v1.3.1 h1:DKHmCUm2hRBK510BaiZlwvpD40f8bJFeZnpfm2KLowc=
+github.com/hashicorp/go-immutable-radix v1.3.1/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
+github.com/hashicorp/go-msgpack v0.5.5 h1:i9R9JSrqIz0QVLz3sz+i3YJdT7TTSLcfLLzJi9aZTuI=
+github.com/hashicorp/go-msgpack v0.5.5/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM=
+github.com/hashicorp/go-retryablehttp v0.5.3/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
+github.com/hashicorp/go-uuid v1.0.0 h1:RS8zrF7PhGwyNPOtxSClXXj9HA8feRnJzgnI1RJCSnM=
+github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
+github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
+github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
+github.com/hashicorp/raft v1.1.0/go.mod h1:4Ak7FSPnuvmb0GV6vgIAJ4vYT4bek9bb6Q+7HVbyzqM=
+github.com/hashicorp/raft v1.1.2/go.mod h1:vPAJM8Asw6u8LxC3eJCUZmRP/E4QmUGE1R7g7k8sG/8=
+github.com/hashicorp/raft v1.3.11 h1:p3v6gf6l3S797NnK5av3HcczOC1T5CLoaRvg0g9ys4A=
+github.com/hashicorp/raft v1.3.11/go.mod h1:J8naEwc6XaaCfts7+28whSeRvCqTd6e20BlCU3LtEO4=
+github.com/hashicorp/raft-boltdb v0.0.0-20171010151810-6e5ba93211ea/go.mod h1:pNv7Wc3ycL6F5oOWn+tPGo2gWD4a5X+yp/ntwdKLjRk=
+github.com/hashicorp/raft-boltdb v0.0.0-20220329195025-15018e9b97e0 h1:CO8dBMLH6dvE1jTn/30ZZw3iuPsNfajshWoJTnVc5cc=
+github.com/hashicorp/raft-boltdb v0.0.0-20220329195025-15018e9b97e0/go.mod h1:nTakvJ4XYq45UXtn0DbwR4aU9ZdjlnIenpbs6Cd+FM0=
+github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
+github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
+github.com/mattn/go-colorable v0.1.8 h1:c1ghPdyEDarC70ftn0y+A/Ee++9zz8ljHG1b13eJ0s8=
+github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
+github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
+github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
+github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
+github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y=
+github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
+github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
+github.com/pascaldekloe/goe v0.1.0 h1:cBOtyMzM9HTpWjXfbbunk26uA6nG3a8n06Wieeh0MwY=
+github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
+github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
+github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM=
+github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
+github.com/prometheus/client_golang v1.4.0/go.mod h1:e9GMxYsXl05ICDXkRhurwBS4Q3OK1iX/F2sw+iXX5zU=
+github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
+github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
+github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
+github.com/prometheus/common v0.9.1/go.mod h1:yhUN8i9wzaXS3w1O07YhxHEBxD+W35wd8bs7vj7HSQ4=
+github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
+github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
+github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
+github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
+github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
+github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
+github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
+github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926/go.mod h1:9ESjWnEqriFuLhtthL60Sar/7RFoluCcXsuvEwTV5KM=
+github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
+go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
+golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
+golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
+golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
+golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
+golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
+golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
+golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
+golang.org/x/net v0.0.0-20210907225631-ff17edfbf26d h1:kuk8nKPQ25KCDODLCDXt99tnTVeOyOM8HGvtJ0NzAvw=
+golang.org/x/net v0.0.0-20210907225631-ff17edfbf26d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
+golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190523142557-0e01d883c5c5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34 h1:GkvMjFtXUmahfDtashnc1mnrCtuBVcwse5QV2lUk/tI=
+golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
+golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
+golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
+google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
+google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
+google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
+google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
+google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
+google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83 h1:3V2dxSZpz4zozWWUq36vUxXEKnSYitEH2LdsAx+RUmg=
+google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY=
+google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
+google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
+google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
+google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
+google.golang.org/grpc v1.31.1/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
+google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0=
+google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
+google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
+google.golang.org/grpc v1.47.0 h1:9n77onPX5F3qfFCqjy9dhn8PbNQsIKeVU04J9G7umt8=
+google.golang.org/grpc v1.47.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk=
+google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
+google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
+google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
+google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
+google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
+google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
+google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
+google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
+google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
+google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
+google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
+google.golang.org/protobuf v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
+google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
diff --git a/jina/serve/consensus/jina-go-proto/jina.pb.go b/jina/serve/consensus/jina-go-proto/jina.pb.go
new file mode 100644
index 0000000000000..664c2a1a3dde4
--- /dev/null
+++ b/jina/serve/consensus/jina-go-proto/jina.pb.go
@@ -0,0 +1,1757 @@
+//protoc --go_out jina-go-proto --go_opt=paths=source_relative --go_opt=Mjina.proto=jina-go-proto=jraft/jina-go-proto --go-grpc_out jina-go-proto --go-grpc_opt=paths=source_relative --experimental_allow_proto3_optional jina.proto
+
+// Code generated by protoc-gen-go. DO NOT EDIT.
+// versions:
+// 	protoc-gen-go v1.28.1
+// 	protoc        v3.13.0
+// source: jina.proto
+
+package jina_go_proto
+
+import (
+	empty "github.com/golang/protobuf/ptypes/empty"
+	_struct "github.com/golang/protobuf/ptypes/struct"
+	timestamp "github.com/golang/protobuf/ptypes/timestamp"
+	protoreflect "google.golang.org/protobuf/reflect/protoreflect"
+	protoimpl "google.golang.org/protobuf/runtime/protoimpl"
+	docarray_go_proto "jraft/docarray-go-proto"
+	reflect "reflect"
+	sync "sync"
+)
+
+const (
+	// Verify that this generated code is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
+	// Verify that runtime/protoimpl is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
+)
+
+type StatusProto_StatusCode int32
+
+const (
+	StatusProto_SUCCESS StatusProto_StatusCode = 0 // success
+	StatusProto_ERROR   StatusProto_StatusCode = 1 // error
+)
+
+// Enum value maps for StatusProto_StatusCode.
+var (
+	StatusProto_StatusCode_name = map[int32]string{
+		0: "SUCCESS",
+		1: "ERROR",
+	}
+	StatusProto_StatusCode_value = map[string]int32{
+		"SUCCESS": 0,
+		"ERROR":   1,
+	}
+)
+
+func (x StatusProto_StatusCode) Enum() *StatusProto_StatusCode {
+	p := new(StatusProto_StatusCode)
+	*p = x
+	return p
+}
+
+func (x StatusProto_StatusCode) String() string {
+	return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
+}
+
+func (StatusProto_StatusCode) Descriptor() protoreflect.EnumDescriptor {
+	return file_jina_proto_enumTypes[0].Descriptor()
+}
+
+func (StatusProto_StatusCode) Type() protoreflect.EnumType {
+	return &file_jina_proto_enumTypes[0]
+}
+
+func (x StatusProto_StatusCode) Number() protoreflect.EnumNumber {
+	return protoreflect.EnumNumber(x)
+}
+
+// Deprecated: Use StatusProto_StatusCode.Descriptor instead.
+func (StatusProto_StatusCode) EnumDescriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{4, 0}
+}
+
+type SnapshotStatusProto_Status int32
+
+const (
+	SnapshotStatusProto_STATE_UNSPECIFIED SnapshotStatusProto_Status = 0
+	SnapshotStatusProto_QUEUED            SnapshotStatusProto_Status = 1 // Snapshot Job is admitted (validated and persisted) and waiting for resources.
+	SnapshotStatusProto_SCHEDULED         SnapshotStatusProto_Status = 2 // Snapshot Job is scheduled to run as soon as resource allocation is ready.
+	SnapshotStatusProto_RUNNING           SnapshotStatusProto_Status = 3 // Resource allocation has been successful. Snapshot Job is RUNNING.
+	SnapshotStatusProto_SUCCEEDED         SnapshotStatusProto_Status = 4 // Snapshot Job has finished successfully.
+	SnapshotStatusProto_FAILED            SnapshotStatusProto_Status = 5 // Snapshot Job has failed.
+	SnapshotStatusProto_NOT_FOUND         SnapshotStatusProto_Status = 6 // If the specific snapshot id is unknown or not found
+)
+
+// Enum value maps for SnapshotStatusProto_Status.
+var (
+	SnapshotStatusProto_Status_name = map[int32]string{
+		0: "STATE_UNSPECIFIED",
+		1: "QUEUED",
+		2: "SCHEDULED",
+		3: "RUNNING",
+		4: "SUCCEEDED",
+		5: "FAILED",
+		6: "NOT_FOUND",
+	}
+	SnapshotStatusProto_Status_value = map[string]int32{
+		"STATE_UNSPECIFIED": 0,
+		"QUEUED":            1,
+		"SCHEDULED":         2,
+		"RUNNING":           3,
+		"SUCCEEDED":         4,
+		"FAILED":            5,
+		"NOT_FOUND":         6,
+	}
+)
+
+func (x SnapshotStatusProto_Status) Enum() *SnapshotStatusProto_Status {
+	p := new(SnapshotStatusProto_Status)
+	*p = x
+	return p
+}
+
+func (x SnapshotStatusProto_Status) String() string {
+	return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
+}
+
+func (SnapshotStatusProto_Status) Descriptor() protoreflect.EnumDescriptor {
+	return file_jina_proto_enumTypes[1].Descriptor()
+}
+
+func (SnapshotStatusProto_Status) Type() protoreflect.EnumType {
+	return &file_jina_proto_enumTypes[1]
+}
+
+func (x SnapshotStatusProto_Status) Number() protoreflect.EnumNumber {
+	return protoreflect.EnumNumber(x)
+}
+
+// Deprecated: Use SnapshotStatusProto_Status.Descriptor instead.
+func (SnapshotStatusProto_Status) EnumDescriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{11, 0}
+}
+
+type RestoreSnapshotStatusProto_Status int32
+
+const (
+	RestoreSnapshotStatusProto_STATE_UNSPECIFIED RestoreSnapshotStatusProto_Status = 0
+	RestoreSnapshotStatusProto_RUNNING           RestoreSnapshotStatusProto_Status = 1 // Resource allocation has been successful. Restore Job is RUNNING.
+	RestoreSnapshotStatusProto_SUCCEEDED         RestoreSnapshotStatusProto_Status = 2 // Resource Job has finished successfully.
+	RestoreSnapshotStatusProto_FAILED            RestoreSnapshotStatusProto_Status = 3 // Resource Job has failed.
+	RestoreSnapshotStatusProto_NOT_FOUND         RestoreSnapshotStatusProto_Status = 6 // If the specific restore id is unknown or not found
+)
+
+// Enum value maps for RestoreSnapshotStatusProto_Status.
+var (
+	RestoreSnapshotStatusProto_Status_name = map[int32]string{
+		0: "STATE_UNSPECIFIED",
+		1: "RUNNING",
+		2: "SUCCEEDED",
+		3: "FAILED",
+		6: "NOT_FOUND",
+	}
+	RestoreSnapshotStatusProto_Status_value = map[string]int32{
+		"STATE_UNSPECIFIED": 0,
+		"RUNNING":           1,
+		"SUCCEEDED":         2,
+		"FAILED":            3,
+		"NOT_FOUND":         6,
+	}
+)
+
+func (x RestoreSnapshotStatusProto_Status) Enum() *RestoreSnapshotStatusProto_Status {
+	p := new(RestoreSnapshotStatusProto_Status)
+	*p = x
+	return p
+}
+
+func (x RestoreSnapshotStatusProto_Status) String() string {
+	return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
+}
+
+func (RestoreSnapshotStatusProto_Status) Descriptor() protoreflect.EnumDescriptor {
+	return file_jina_proto_enumTypes[2].Descriptor()
+}
+
+func (RestoreSnapshotStatusProto_Status) Type() protoreflect.EnumType {
+	return &file_jina_proto_enumTypes[2]
+}
+
+func (x RestoreSnapshotStatusProto_Status) Number() protoreflect.EnumNumber {
+	return protoreflect.EnumNumber(x)
+}
+
+// Deprecated: Use RestoreSnapshotStatusProto_Status.Descriptor instead.
+func (RestoreSnapshotStatusProto_Status) EnumDescriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{12, 0}
+}
+
+//*
+// Represents a the route paths of this message as perceived by the Gateway
+// start_time is set when the Gateway sends a message to a Pod
+// end_time is set when the Gateway receives a message from a Pod
+// thus end_time - start_time includes Executor computation, runtime overhead, serialization and network
+type RouteProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Executor  string               `protobuf:"bytes,1,opt,name=executor,proto3" json:"executor,omitempty"`                    // the name of the BasePod
+	StartTime *timestamp.Timestamp `protobuf:"bytes,2,opt,name=start_time,json=startTime,proto3" json:"start_time,omitempty"` // time when the Gateway starts sending to the Pod
+	EndTime   *timestamp.Timestamp `protobuf:"bytes,3,opt,name=end_time,json=endTime,proto3" json:"end_time,omitempty"`       // time when the Gateway received it from the Pod
+	Status    *StatusProto         `protobuf:"bytes,4,opt,name=status,proto3" json:"status,omitempty"`                        // the status of the execution
+}
+
+func (x *RouteProto) Reset() {
+	*x = RouteProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[0]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *RouteProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*RouteProto) ProtoMessage() {}
+
+func (x *RouteProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[0]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use RouteProto.ProtoReflect.Descriptor instead.
+func (*RouteProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{0}
+}
+
+func (x *RouteProto) GetExecutor() string {
+	if x != nil {
+		return x.Executor
+	}
+	return ""
+}
+
+func (x *RouteProto) GetStartTime() *timestamp.Timestamp {
+	if x != nil {
+		return x.StartTime
+	}
+	return nil
+}
+
+func (x *RouteProto) GetEndTime() *timestamp.Timestamp {
+	if x != nil {
+		return x.EndTime
+	}
+	return nil
+}
+
+func (x *RouteProto) GetStatus() *StatusProto {
+	if x != nil {
+		return x.Status
+	}
+	return nil
+}
+
+type JinaInfoProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Jina map[string]string `protobuf:"bytes,1,rep,name=jina,proto3" json:"jina,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // information about the system running and package version information including jina
+	Envs map[string]string `protobuf:"bytes,2,rep,name=envs,proto3" json:"envs,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // the environment variable setting
+}
+
+func (x *JinaInfoProto) Reset() {
+	*x = JinaInfoProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[1]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *JinaInfoProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*JinaInfoProto) ProtoMessage() {}
+
+func (x *JinaInfoProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[1]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use JinaInfoProto.ProtoReflect.Descriptor instead.
+func (*JinaInfoProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{1}
+}
+
+func (x *JinaInfoProto) GetJina() map[string]string {
+	if x != nil {
+		return x.Jina
+	}
+	return nil
+}
+
+func (x *JinaInfoProto) GetEnvs() map[string]string {
+	if x != nil {
+		return x.Envs
+	}
+	return nil
+}
+
+//*
+// Represents a Header.
+// - The header's content will be defined by the user request.
+// - It will be copied to the envelope.header
+// - In-flow operations will modify the envelope.header
+// - While returning, copy envelope.header back to request.header
+type HeaderProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	RequestId      string       `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`                      // the unique ID of this request. Multiple requests with the same ID will be gathered
+	Status         *StatusProto `protobuf:"bytes,2,opt,name=status,proto3" json:"status,omitempty"`                                             // status info
+	ExecEndpoint   *string      `protobuf:"bytes,3,opt,name=exec_endpoint,json=execEndpoint,proto3,oneof" json:"exec_endpoint,omitempty"`       // the endpoint specified by `@requests(on='/abc')`
+	TargetExecutor *string      `protobuf:"bytes,4,opt,name=target_executor,json=targetExecutor,proto3,oneof" json:"target_executor,omitempty"` // if set, the request is targeted to certain executor, regex strings
+	Timeout        *uint32      `protobuf:"varint,5,opt,name=timeout,proto3,oneof" json:"timeout,omitempty"`                                    // epoch time in seconds after which the request should be dropped
+}
+
+func (x *HeaderProto) Reset() {
+	*x = HeaderProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[2]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *HeaderProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*HeaderProto) ProtoMessage() {}
+
+func (x *HeaderProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[2]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use HeaderProto.ProtoReflect.Descriptor instead.
+func (*HeaderProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{2}
+}
+
+func (x *HeaderProto) GetRequestId() string {
+	if x != nil {
+		return x.RequestId
+	}
+	return ""
+}
+
+func (x *HeaderProto) GetStatus() *StatusProto {
+	if x != nil {
+		return x.Status
+	}
+	return nil
+}
+
+func (x *HeaderProto) GetExecEndpoint() string {
+	if x != nil && x.ExecEndpoint != nil {
+		return *x.ExecEndpoint
+	}
+	return ""
+}
+
+func (x *HeaderProto) GetTargetExecutor() string {
+	if x != nil && x.TargetExecutor != nil {
+		return *x.TargetExecutor
+	}
+	return ""
+}
+
+func (x *HeaderProto) GetTimeout() uint32 {
+	if x != nil && x.Timeout != nil {
+		return *x.Timeout
+	}
+	return 0
+}
+
+//*
+// Represents the set of Endpoints exposed by an Executor
+type EndpointsProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// list of endpoints exposed by an Executor
+	Endpoints      []string `protobuf:"bytes,1,rep,name=endpoints,proto3" json:"endpoints,omitempty"`
+	WriteEndpoints []string `protobuf:"bytes,2,rep,name=write_endpoints,json=writeEndpoints,proto3" json:"write_endpoints,omitempty"`
+}
+
+func (x *EndpointsProto) Reset() {
+	*x = EndpointsProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[3]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *EndpointsProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*EndpointsProto) ProtoMessage() {}
+
+func (x *EndpointsProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[3]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use EndpointsProto.ProtoReflect.Descriptor instead.
+func (*EndpointsProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{3}
+}
+
+func (x *EndpointsProto) GetEndpoints() []string {
+	if x != nil {
+		return x.Endpoints
+	}
+	return nil
+}
+
+func (x *EndpointsProto) GetWriteEndpoints() []string {
+	if x != nil {
+		return x.WriteEndpoints
+	}
+	return nil
+}
+
+//*
+// Represents a Status
+type StatusProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// status code
+	Code StatusProto_StatusCode `protobuf:"varint,1,opt,name=code,proto3,enum=jina.StatusProto_StatusCode" json:"code,omitempty"`
+	// error description of the very first exception
+	Description string `protobuf:"bytes,2,opt,name=description,proto3" json:"description,omitempty"`
+	// the details of the error
+	Exception *StatusProto_ExceptionProto `protobuf:"bytes,3,opt,name=exception,proto3" json:"exception,omitempty"`
+}
+
+func (x *StatusProto) Reset() {
+	*x = StatusProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[4]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *StatusProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*StatusProto) ProtoMessage() {}
+
+func (x *StatusProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[4]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use StatusProto.ProtoReflect.Descriptor instead.
+func (*StatusProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{4}
+}
+
+func (x *StatusProto) GetCode() StatusProto_StatusCode {
+	if x != nil {
+		return x.Code
+	}
+	return StatusProto_SUCCESS
+}
+
+func (x *StatusProto) GetDescription() string {
+	if x != nil {
+		return x.Description
+	}
+	return ""
+}
+
+func (x *StatusProto) GetException() *StatusProto_ExceptionProto {
+	if x != nil {
+		return x.Exception
+	}
+	return nil
+}
+
+//*
+// Represents an entity (like an ExecutorRuntime)
+type RelatedEntity struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Id      string  `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`                                 // unique id of the entity, like the name of a pod
+	Address string  `protobuf:"bytes,2,opt,name=address,proto3" json:"address,omitempty"`                       // address of the entity, could be an IP address, domain name etc, does not include port
+	Port    uint32  `protobuf:"varint,3,opt,name=port,proto3" json:"port,omitempty"`                            // port this entity is listening on
+	ShardId *uint32 `protobuf:"varint,4,opt,name=shard_id,json=shardId,proto3,oneof" json:"shard_id,omitempty"` // the id of the shard it belongs to, if it is a shard
+}
+
+func (x *RelatedEntity) Reset() {
+	*x = RelatedEntity{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[5]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *RelatedEntity) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*RelatedEntity) ProtoMessage() {}
+
+func (x *RelatedEntity) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[5]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use RelatedEntity.ProtoReflect.Descriptor instead.
+func (*RelatedEntity) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{5}
+}
+
+func (x *RelatedEntity) GetId() string {
+	if x != nil {
+		return x.Id
+	}
+	return ""
+}
+
+func (x *RelatedEntity) GetAddress() string {
+	if x != nil {
+		return x.Address
+	}
+	return ""
+}
+
+func (x *RelatedEntity) GetPort() uint32 {
+	if x != nil {
+		return x.Port
+	}
+	return 0
+}
+
+func (x *RelatedEntity) GetShardId() uint32 {
+	if x != nil && x.ShardId != nil {
+		return *x.ShardId
+	}
+	return 0
+}
+
+//*
+// Represents a DataRequest
+type DataRequestProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Header     *HeaderProto                       `protobuf:"bytes,1,opt,name=header,proto3" json:"header,omitempty"`         // header contains meta info defined by the user
+	Parameters *_struct.Struct                    `protobuf:"bytes,2,opt,name=parameters,proto3" json:"parameters,omitempty"` // extra kwargs that will be used in executor
+	Routes     []*RouteProto                      `protobuf:"bytes,3,rep,name=routes,proto3" json:"routes,omitempty"`         // status info on every routes
+	Data       *DataRequestProto_DataContentProto `protobuf:"bytes,4,opt,name=data,proto3" json:"data,omitempty"`             // container for docs and groundtruths
+}
+
+func (x *DataRequestProto) Reset() {
+	*x = DataRequestProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[6]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *DataRequestProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*DataRequestProto) ProtoMessage() {}
+
+func (x *DataRequestProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[6]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use DataRequestProto.ProtoReflect.Descriptor instead.
+func (*DataRequestProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{6}
+}
+
+func (x *DataRequestProto) GetHeader() *HeaderProto {
+	if x != nil {
+		return x.Header
+	}
+	return nil
+}
+
+func (x *DataRequestProto) GetParameters() *_struct.Struct {
+	if x != nil {
+		return x.Parameters
+	}
+	return nil
+}
+
+func (x *DataRequestProto) GetRoutes() []*RouteProto {
+	if x != nil {
+		return x.Routes
+	}
+	return nil
+}
+
+func (x *DataRequestProto) GetData() *DataRequestProto_DataContentProto {
+	if x != nil {
+		return x.Data
+	}
+	return nil
+}
+
+type DataRequestProtoWoData struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Header     *HeaderProto    `protobuf:"bytes,1,opt,name=header,proto3" json:"header,omitempty"`         // header contains meta info defined by the user
+	Parameters *_struct.Struct `protobuf:"bytes,2,opt,name=parameters,proto3" json:"parameters,omitempty"` // extra kwargs that will be used in executor
+	Routes     []*RouteProto   `protobuf:"bytes,3,rep,name=routes,proto3" json:"routes,omitempty"`         // status info on every routes
+}
+
+func (x *DataRequestProtoWoData) Reset() {
+	*x = DataRequestProtoWoData{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[7]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *DataRequestProtoWoData) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*DataRequestProtoWoData) ProtoMessage() {}
+
+func (x *DataRequestProtoWoData) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[7]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use DataRequestProtoWoData.ProtoReflect.Descriptor instead.
+func (*DataRequestProtoWoData) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{7}
+}
+
+func (x *DataRequestProtoWoData) GetHeader() *HeaderProto {
+	if x != nil {
+		return x.Header
+	}
+	return nil
+}
+
+func (x *DataRequestProtoWoData) GetParameters() *_struct.Struct {
+	if x != nil {
+		return x.Parameters
+	}
+	return nil
+}
+
+func (x *DataRequestProtoWoData) GetRoutes() []*RouteProto {
+	if x != nil {
+		return x.Routes
+	}
+	return nil
+}
+
+//*
+// Represents a list of data requests
+// This should be replaced by streaming
+type DataRequestListProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Requests []*DataRequestProto `protobuf:"bytes,1,rep,name=requests,proto3" json:"requests,omitempty"` // requests in this list
+}
+
+func (x *DataRequestListProto) Reset() {
+	*x = DataRequestListProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[8]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *DataRequestListProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*DataRequestListProto) ProtoMessage() {}
+
+func (x *DataRequestListProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[8]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use DataRequestListProto.ProtoReflect.Descriptor instead.
+func (*DataRequestListProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{8}
+}
+
+func (x *DataRequestListProto) GetRequests() []*DataRequestProto {
+	if x != nil {
+		return x.Requests
+	}
+	return nil
+}
+
+//*
+// An ID representation for the Snapshot Job
+type SnapshotId struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Value string `protobuf:"bytes,1,opt,name=value,proto3" json:"value,omitempty"`
+}
+
+func (x *SnapshotId) Reset() {
+	*x = SnapshotId{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[9]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *SnapshotId) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*SnapshotId) ProtoMessage() {}
+
+func (x *SnapshotId) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[9]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use SnapshotId.ProtoReflect.Descriptor instead.
+func (*SnapshotId) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{9}
+}
+
+func (x *SnapshotId) GetValue() string {
+	if x != nil {
+		return x.Value
+	}
+	return ""
+}
+
+//*
+// An ID representation for the Restore Job
+type RestoreId struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Value string `protobuf:"bytes,1,opt,name=value,proto3" json:"value,omitempty"`
+}
+
+func (x *RestoreId) Reset() {
+	*x = RestoreId{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[10]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *RestoreId) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*RestoreId) ProtoMessage() {}
+
+func (x *RestoreId) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[10]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use RestoreId.ProtoReflect.Descriptor instead.
+func (*RestoreId) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{10}
+}
+
+func (x *RestoreId) GetValue() string {
+	if x != nil {
+		return x.Value
+	}
+	return ""
+}
+
+//*
+// Represents the status of a Snapshot.
+type SnapshotStatusProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Id *SnapshotId `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` // An ID for the Snapshot Job.
+	// status code
+	Status SnapshotStatusProto_Status `protobuf:"varint,2,opt,name=status,proto3,enum=jina.SnapshotStatusProto_Status" json:"status,omitempty"`
+	// snapshot file
+	SnapshotFile string `protobuf:"bytes,3,opt,name=snapshot_file,json=snapshotFile,proto3" json:"snapshot_file,omitempty"`
+}
+
+func (x *SnapshotStatusProto) Reset() {
+	*x = SnapshotStatusProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[11]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *SnapshotStatusProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*SnapshotStatusProto) ProtoMessage() {}
+
+func (x *SnapshotStatusProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[11]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use SnapshotStatusProto.ProtoReflect.Descriptor instead.
+func (*SnapshotStatusProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{11}
+}
+
+func (x *SnapshotStatusProto) GetId() *SnapshotId {
+	if x != nil {
+		return x.Id
+	}
+	return nil
+}
+
+func (x *SnapshotStatusProto) GetStatus() SnapshotStatusProto_Status {
+	if x != nil {
+		return x.Status
+	}
+	return SnapshotStatusProto_STATE_UNSPECIFIED
+}
+
+func (x *SnapshotStatusProto) GetSnapshotFile() string {
+	if x != nil {
+		return x.SnapshotFile
+	}
+	return ""
+}
+
+//*
+// Represents the status of a Restore.
+type RestoreSnapshotStatusProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	Id *RestoreId `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` // An ID for the Restore Job.
+	// status code
+	Status RestoreSnapshotStatusProto_Status `protobuf:"varint,2,opt,name=status,proto3,enum=jina.RestoreSnapshotStatusProto_Status" json:"status,omitempty"`
+}
+
+func (x *RestoreSnapshotStatusProto) Reset() {
+	*x = RestoreSnapshotStatusProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[12]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *RestoreSnapshotStatusProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*RestoreSnapshotStatusProto) ProtoMessage() {}
+
+func (x *RestoreSnapshotStatusProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[12]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use RestoreSnapshotStatusProto.ProtoReflect.Descriptor instead.
+func (*RestoreSnapshotStatusProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{12}
+}
+
+func (x *RestoreSnapshotStatusProto) GetId() *RestoreId {
+	if x != nil {
+		return x.Id
+	}
+	return nil
+}
+
+func (x *RestoreSnapshotStatusProto) GetStatus() RestoreSnapshotStatusProto_Status {
+	if x != nil {
+		return x.Status
+	}
+	return RestoreSnapshotStatusProto_STATE_UNSPECIFIED
+}
+
+//*
+// Commands to restore an Executor from a snapshot file
+type RestoreSnapshotCommand struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// snapshot file
+	SnapshotFile string `protobuf:"bytes,1,opt,name=snapshot_file,json=snapshotFile,proto3" json:"snapshot_file,omitempty"`
+}
+
+func (x *RestoreSnapshotCommand) Reset() {
+	*x = RestoreSnapshotCommand{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[13]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *RestoreSnapshotCommand) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*RestoreSnapshotCommand) ProtoMessage() {}
+
+func (x *RestoreSnapshotCommand) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[13]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use RestoreSnapshotCommand.ProtoReflect.Descriptor instead.
+func (*RestoreSnapshotCommand) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{13}
+}
+
+func (x *RestoreSnapshotCommand) GetSnapshotFile() string {
+	if x != nil {
+		return x.SnapshotFile
+	}
+	return ""
+}
+
+type StatusProto_ExceptionProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// the class name of the exception
+	Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
+	// the list of arguments given to the exception constructor.
+	Args []string `protobuf:"bytes,2,rep,name=args,proto3" json:"args,omitempty"`
+	// the exception traceback stacks
+	Stacks []string `protobuf:"bytes,3,rep,name=stacks,proto3" json:"stacks,omitempty"`
+	// the name of the executor bind to that Executor (if applicable)
+	Executor string `protobuf:"bytes,4,opt,name=executor,proto3" json:"executor,omitempty"`
+}
+
+func (x *StatusProto_ExceptionProto) Reset() {
+	*x = StatusProto_ExceptionProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[16]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *StatusProto_ExceptionProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*StatusProto_ExceptionProto) ProtoMessage() {}
+
+func (x *StatusProto_ExceptionProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[16]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use StatusProto_ExceptionProto.ProtoReflect.Descriptor instead.
+func (*StatusProto_ExceptionProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{4, 0}
+}
+
+func (x *StatusProto_ExceptionProto) GetName() string {
+	if x != nil {
+		return x.Name
+	}
+	return ""
+}
+
+func (x *StatusProto_ExceptionProto) GetArgs() []string {
+	if x != nil {
+		return x.Args
+	}
+	return nil
+}
+
+func (x *StatusProto_ExceptionProto) GetStacks() []string {
+	if x != nil {
+		return x.Stacks
+	}
+	return nil
+}
+
+func (x *StatusProto_ExceptionProto) GetExecutor() string {
+	if x != nil {
+		return x.Executor
+	}
+	return ""
+}
+
+type DataRequestProto_DataContentProto struct {
+	state         protoimpl.MessageState
+	sizeCache     protoimpl.SizeCache
+	unknownFields protoimpl.UnknownFields
+
+	// Types that are assignable to Documents:
+	//	*DataRequestProto_DataContentProto_Docs
+	//	*DataRequestProto_DataContentProto_DocsBytes
+	Documents isDataRequestProto_DataContentProto_Documents `protobuf_oneof:"documents"`
+}
+
+func (x *DataRequestProto_DataContentProto) Reset() {
+	*x = DataRequestProto_DataContentProto{}
+	if protoimpl.UnsafeEnabled {
+		mi := &file_jina_proto_msgTypes[17]
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		ms.StoreMessageInfo(mi)
+	}
+}
+
+func (x *DataRequestProto_DataContentProto) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*DataRequestProto_DataContentProto) ProtoMessage() {}
+
+func (x *DataRequestProto_DataContentProto) ProtoReflect() protoreflect.Message {
+	mi := &file_jina_proto_msgTypes[17]
+	if protoimpl.UnsafeEnabled && x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use DataRequestProto_DataContentProto.ProtoReflect.Descriptor instead.
+func (*DataRequestProto_DataContentProto) Descriptor() ([]byte, []int) {
+	return file_jina_proto_rawDescGZIP(), []int{6, 0}
+}
+
+func (m *DataRequestProto_DataContentProto) GetDocuments() isDataRequestProto_DataContentProto_Documents {
+	if m != nil {
+		return m.Documents
+	}
+	return nil
+}
+
+func (x *DataRequestProto_DataContentProto) GetDocs() *docarray_go_proto.DocListProto {
+	if x, ok := x.GetDocuments().(*DataRequestProto_DataContentProto_Docs); ok {
+		return x.Docs
+	}
+	return nil
+}
+
+func (x *DataRequestProto_DataContentProto) GetDocsBytes() []byte {
+	if x, ok := x.GetDocuments().(*DataRequestProto_DataContentProto_DocsBytes); ok {
+		return x.DocsBytes
+	}
+	return nil
+}
+
+type isDataRequestProto_DataContentProto_Documents interface {
+	isDataRequestProto_DataContentProto_Documents()
+}
+
+type DataRequestProto_DataContentProto_Docs struct {
+	Docs *docarray_go_proto.DocListProto `protobuf:"bytes,1,opt,name=docs,proto3,oneof"` // the docs in this request
+}
+
+type DataRequestProto_DataContentProto_DocsBytes struct {
+	DocsBytes []byte `protobuf:"bytes,2,opt,name=docs_bytes,json=docsBytes,proto3,oneof"` // the docs in this request as bytes
+}
+
+func (*DataRequestProto_DataContentProto_Docs) isDataRequestProto_DataContentProto_Documents() {}
+
+func (*DataRequestProto_DataContentProto_DocsBytes) isDataRequestProto_DataContentProto_Documents() {}
+
+var File_jina_proto protoreflect.FileDescriptor
+
+var file_jina_proto_rawDesc = []byte{
+	0x0a, 0x0a, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x04, 0x6a, 0x69,
+	0x6e, 0x61, 0x1a, 0x1f, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f,
+	0x62, 0x75, 0x66, 0x2f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x2e, 0x70, 0x72,
+	0x6f, 0x74, 0x6f, 0x1a, 0x1c, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74,
+	0x6f, 0x62, 0x75, 0x66, 0x2f, 0x73, 0x74, 0x72, 0x75, 0x63, 0x74, 0x2e, 0x70, 0x72, 0x6f, 0x74,
+	0x6f, 0x1a, 0x1b, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62,
+	0x75, 0x66, 0x2f, 0x65, 0x6d, 0x70, 0x74, 0x79, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x0e,
+	0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0xc5,
+	0x01, 0x0a, 0x0a, 0x52, 0x6f, 0x75, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x1a, 0x0a,
+	0x08, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x6f, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52,
+	0x08, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x6f, 0x72, 0x12, 0x39, 0x0a, 0x0a, 0x73, 0x74, 0x61,
+	0x72, 0x74, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e,
+	0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e,
+	0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x09, 0x73, 0x74, 0x61, 0x72, 0x74,
+	0x54, 0x69, 0x6d, 0x65, 0x12, 0x35, 0x0a, 0x08, 0x65, 0x6e, 0x64, 0x5f, 0x74, 0x69, 0x6d, 0x65,
+	0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e,
+	0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61,
+	0x6d, 0x70, 0x52, 0x07, 0x65, 0x6e, 0x64, 0x54, 0x69, 0x6d, 0x65, 0x12, 0x29, 0x0a, 0x06, 0x73,
+	0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x6a, 0x69,
+	0x6e, 0x61, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x06,
+	0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0xe7, 0x01, 0x0a, 0x0d, 0x4a, 0x69, 0x6e, 0x61, 0x49,
+	0x6e, 0x66, 0x6f, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x31, 0x0a, 0x04, 0x6a, 0x69, 0x6e, 0x61,
+	0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1d, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x4a, 0x69,
+	0x6e, 0x61, 0x49, 0x6e, 0x66, 0x6f, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x4a, 0x69, 0x6e, 0x61,
+	0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x04, 0x6a, 0x69, 0x6e, 0x61, 0x12, 0x31, 0x0a, 0x04, 0x65,
+	0x6e, 0x76, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1d, 0x2e, 0x6a, 0x69, 0x6e, 0x61,
+	0x2e, 0x4a, 0x69, 0x6e, 0x61, 0x49, 0x6e, 0x66, 0x6f, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x45,
+	0x6e, 0x76, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x04, 0x65, 0x6e, 0x76, 0x73, 0x1a, 0x37,
+	0x0a, 0x09, 0x4a, 0x69, 0x6e, 0x61, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b,
+	0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a,
+	0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61,
+	0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x1a, 0x37, 0x0a, 0x09, 0x45, 0x6e, 0x76, 0x73, 0x45,
+	0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28,
+	0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18,
+	0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01,
+	0x22, 0x80, 0x02, 0x0a, 0x0b, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x50, 0x72, 0x6f, 0x74, 0x6f,
+	0x12, 0x1d, 0x0a, 0x0a, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01,
+	0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, 0x12,
+	0x29, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32,
+	0x11, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f,
+	0x74, 0x6f, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x28, 0x0a, 0x0d, 0x65, 0x78,
+	0x65, 0x63, 0x5f, 0x65, 0x6e, 0x64, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28,
+	0x09, 0x48, 0x00, 0x52, 0x0c, 0x65, 0x78, 0x65, 0x63, 0x45, 0x6e, 0x64, 0x70, 0x6f, 0x69, 0x6e,
+	0x74, 0x88, 0x01, 0x01, 0x12, 0x2c, 0x0a, 0x0f, 0x74, 0x61, 0x72, 0x67, 0x65, 0x74, 0x5f, 0x65,
+	0x78, 0x65, 0x63, 0x75, 0x74, 0x6f, 0x72, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x48, 0x01, 0x52,
+	0x0e, 0x74, 0x61, 0x72, 0x67, 0x65, 0x74, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x6f, 0x72, 0x88,
+	0x01, 0x01, 0x12, 0x1d, 0x0a, 0x07, 0x74, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x18, 0x05, 0x20,
+	0x01, 0x28, 0x0d, 0x48, 0x02, 0x52, 0x07, 0x74, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x88, 0x01,
+	0x01, 0x42, 0x10, 0x0a, 0x0e, 0x5f, 0x65, 0x78, 0x65, 0x63, 0x5f, 0x65, 0x6e, 0x64, 0x70, 0x6f,
+	0x69, 0x6e, 0x74, 0x42, 0x12, 0x0a, 0x10, 0x5f, 0x74, 0x61, 0x72, 0x67, 0x65, 0x74, 0x5f, 0x65,
+	0x78, 0x65, 0x63, 0x75, 0x74, 0x6f, 0x72, 0x42, 0x0a, 0x0a, 0x08, 0x5f, 0x74, 0x69, 0x6d, 0x65,
+	0x6f, 0x75, 0x74, 0x22, 0x57, 0x0a, 0x0e, 0x45, 0x6e, 0x64, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0x73,
+	0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x1c, 0x0a, 0x09, 0x65, 0x6e, 0x64, 0x70, 0x6f, 0x69, 0x6e,
+	0x74, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, 0x52, 0x09, 0x65, 0x6e, 0x64, 0x70, 0x6f, 0x69,
+	0x6e, 0x74, 0x73, 0x12, 0x27, 0x0a, 0x0f, 0x77, 0x72, 0x69, 0x74, 0x65, 0x5f, 0x65, 0x6e, 0x64,
+	0x70, 0x6f, 0x69, 0x6e, 0x74, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x09, 0x52, 0x0e, 0x77, 0x72,
+	0x69, 0x74, 0x65, 0x45, 0x6e, 0x64, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0x73, 0x22, 0xb5, 0x02, 0x0a,
+	0x0b, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x30, 0x0a, 0x04,
+	0x63, 0x6f, 0x64, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1c, 0x2e, 0x6a, 0x69, 0x6e,
+	0x61, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x53, 0x74,
+	0x61, 0x74, 0x75, 0x73, 0x43, 0x6f, 0x64, 0x65, 0x52, 0x04, 0x63, 0x6f, 0x64, 0x65, 0x12, 0x20,
+	0x0a, 0x0b, 0x64, 0x65, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20,
+	0x01, 0x28, 0x09, 0x52, 0x0b, 0x64, 0x65, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e,
+	0x12, 0x3e, 0x0a, 0x09, 0x65, 0x78, 0x63, 0x65, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20,
+	0x01, 0x28, 0x0b, 0x32, 0x20, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75,
+	0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x45, 0x78, 0x63, 0x65, 0x70, 0x74, 0x69, 0x6f, 0x6e,
+	0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x09, 0x65, 0x78, 0x63, 0x65, 0x70, 0x74, 0x69, 0x6f, 0x6e,
+	0x1a, 0x6c, 0x0a, 0x0e, 0x45, 0x78, 0x63, 0x65, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x50, 0x72, 0x6f,
+	0x74, 0x6f, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09,
+	0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x61, 0x72, 0x67, 0x73, 0x18, 0x02,
+	0x20, 0x03, 0x28, 0x09, 0x52, 0x04, 0x61, 0x72, 0x67, 0x73, 0x12, 0x16, 0x0a, 0x06, 0x73, 0x74,
+	0x61, 0x63, 0x6b, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x09, 0x52, 0x06, 0x73, 0x74, 0x61, 0x63,
+	0x6b, 0x73, 0x12, 0x1a, 0x0a, 0x08, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x6f, 0x72, 0x18, 0x04,
+	0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x65, 0x78, 0x65, 0x63, 0x75, 0x74, 0x6f, 0x72, 0x22, 0x24,
+	0x0a, 0x0a, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x43, 0x6f, 0x64, 0x65, 0x12, 0x0b, 0x0a, 0x07,
+	0x53, 0x55, 0x43, 0x43, 0x45, 0x53, 0x53, 0x10, 0x00, 0x12, 0x09, 0x0a, 0x05, 0x45, 0x52, 0x52,
+	0x4f, 0x52, 0x10, 0x01, 0x22, 0x7a, 0x0a, 0x0d, 0x52, 0x65, 0x6c, 0x61, 0x74, 0x65, 0x64, 0x45,
+	0x6e, 0x74, 0x69, 0x74, 0x79, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28,
+	0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x18, 0x0a, 0x07, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73,
+	0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x12,
+	0x12, 0x0a, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0d, 0x52, 0x04, 0x70,
+	0x6f, 0x72, 0x74, 0x12, 0x1e, 0x0a, 0x08, 0x73, 0x68, 0x61, 0x72, 0x64, 0x5f, 0x69, 0x64, 0x18,
+	0x04, 0x20, 0x01, 0x28, 0x0d, 0x48, 0x00, 0x52, 0x07, 0x73, 0x68, 0x61, 0x72, 0x64, 0x49, 0x64,
+	0x88, 0x01, 0x01, 0x42, 0x0b, 0x0a, 0x09, 0x5f, 0x73, 0x68, 0x61, 0x72, 0x64, 0x5f, 0x69, 0x64,
+	0x22, 0xcd, 0x02, 0x0a, 0x10, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74,
+	0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x29, 0x0a, 0x06, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x18,
+	0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x48, 0x65, 0x61,
+	0x64, 0x65, 0x72, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x06, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72,
+	0x12, 0x37, 0x0a, 0x0a, 0x70, 0x61, 0x72, 0x61, 0x6d, 0x65, 0x74, 0x65, 0x72, 0x73, 0x18, 0x02,
+	0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72,
+	0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x53, 0x74, 0x72, 0x75, 0x63, 0x74, 0x52, 0x0a, 0x70,
+	0x61, 0x72, 0x61, 0x6d, 0x65, 0x74, 0x65, 0x72, 0x73, 0x12, 0x28, 0x0a, 0x06, 0x72, 0x6f, 0x75,
+	0x74, 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x10, 0x2e, 0x6a, 0x69, 0x6e, 0x61,
+	0x2e, 0x52, 0x6f, 0x75, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x06, 0x72, 0x6f, 0x75,
+	0x74, 0x65, 0x73, 0x12, 0x3b, 0x0a, 0x04, 0x64, 0x61, 0x74, 0x61, 0x18, 0x04, 0x20, 0x01, 0x28,
+	0x0b, 0x32, 0x27, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71,
+	0x75, 0x65, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x43, 0x6f,
+	0x6e, 0x74, 0x65, 0x6e, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x04, 0x64, 0x61, 0x74, 0x61,
+	0x1a, 0x6e, 0x0a, 0x10, 0x44, 0x61, 0x74, 0x61, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x50,
+	0x72, 0x6f, 0x74, 0x6f, 0x12, 0x2c, 0x0a, 0x04, 0x64, 0x6f, 0x63, 0x73, 0x18, 0x01, 0x20, 0x01,
+	0x28, 0x0b, 0x32, 0x16, 0x2e, 0x64, 0x6f, 0x63, 0x61, 0x72, 0x72, 0x61, 0x79, 0x2e, 0x44, 0x6f,
+	0x63, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x48, 0x00, 0x52, 0x04, 0x64, 0x6f,
+	0x63, 0x73, 0x12, 0x1f, 0x0a, 0x0a, 0x64, 0x6f, 0x63, 0x73, 0x5f, 0x62, 0x79, 0x74, 0x65, 0x73,
+	0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x48, 0x00, 0x52, 0x09, 0x64, 0x6f, 0x63, 0x73, 0x42, 0x79,
+	0x74, 0x65, 0x73, 0x42, 0x0b, 0x0a, 0x09, 0x64, 0x6f, 0x63, 0x75, 0x6d, 0x65, 0x6e, 0x74, 0x73,
+	0x22, 0xa6, 0x01, 0x0a, 0x16, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74,
+	0x50, 0x72, 0x6f, 0x74, 0x6f, 0x57, 0x6f, 0x44, 0x61, 0x74, 0x61, 0x12, 0x29, 0x0a, 0x06, 0x68,
+	0x65, 0x61, 0x64, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x6a, 0x69,
+	0x6e, 0x61, 0x2e, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x06,
+	0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x37, 0x0a, 0x0a, 0x70, 0x61, 0x72, 0x61, 0x6d, 0x65,
+	0x74, 0x65, 0x72, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x67, 0x6f, 0x6f,
+	0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x53, 0x74, 0x72,
+	0x75, 0x63, 0x74, 0x52, 0x0a, 0x70, 0x61, 0x72, 0x61, 0x6d, 0x65, 0x74, 0x65, 0x72, 0x73, 0x12,
+	0x28, 0x0a, 0x06, 0x72, 0x6f, 0x75, 0x74, 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32,
+	0x10, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x52, 0x6f, 0x75, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x74,
+	0x6f, 0x52, 0x06, 0x72, 0x6f, 0x75, 0x74, 0x65, 0x73, 0x22, 0x4a, 0x0a, 0x14, 0x44, 0x61, 0x74,
+	0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74,
+	0x6f, 0x12, 0x32, 0x0a, 0x08, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x73, 0x18, 0x01, 0x20,
+	0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52,
+	0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x52, 0x08, 0x72, 0x65, 0x71,
+	0x75, 0x65, 0x73, 0x74, 0x73, 0x22, 0x22, 0x0a, 0x0a, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f,
+	0x74, 0x49, 0x64, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x01, 0x20, 0x01,
+	0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, 0x21, 0x0a, 0x09, 0x52, 0x65, 0x73,
+	0x74, 0x6f, 0x72, 0x65, 0x49, 0x64, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18,
+	0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, 0x89, 0x02, 0x0a,
+	0x13, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x50,
+	0x72, 0x6f, 0x74, 0x6f, 0x12, 0x20, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b,
+	0x32, 0x10, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74,
+	0x49, 0x64, 0x52, 0x02, 0x69, 0x64, 0x12, 0x38, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73,
+	0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x20, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x53, 0x6e,
+	0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f, 0x74,
+	0x6f, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73,
+	0x12, 0x23, 0x0a, 0x0d, 0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x66, 0x69, 0x6c,
+	0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0c, 0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f,
+	0x74, 0x46, 0x69, 0x6c, 0x65, 0x22, 0x71, 0x0a, 0x06, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12,
+	0x15, 0x0a, 0x11, 0x53, 0x54, 0x41, 0x54, 0x45, 0x5f, 0x55, 0x4e, 0x53, 0x50, 0x45, 0x43, 0x49,
+	0x46, 0x49, 0x45, 0x44, 0x10, 0x00, 0x12, 0x0a, 0x0a, 0x06, 0x51, 0x55, 0x45, 0x55, 0x45, 0x44,
+	0x10, 0x01, 0x12, 0x0d, 0x0a, 0x09, 0x53, 0x43, 0x48, 0x45, 0x44, 0x55, 0x4c, 0x45, 0x44, 0x10,
+	0x02, 0x12, 0x0b, 0x0a, 0x07, 0x52, 0x55, 0x4e, 0x4e, 0x49, 0x4e, 0x47, 0x10, 0x03, 0x12, 0x0d,
+	0x0a, 0x09, 0x53, 0x55, 0x43, 0x43, 0x45, 0x45, 0x44, 0x45, 0x44, 0x10, 0x04, 0x12, 0x0a, 0x0a,
+	0x06, 0x46, 0x41, 0x49, 0x4c, 0x45, 0x44, 0x10, 0x05, 0x12, 0x0d, 0x0a, 0x09, 0x4e, 0x4f, 0x54,
+	0x5f, 0x46, 0x4f, 0x55, 0x4e, 0x44, 0x10, 0x06, 0x22, 0xd6, 0x01, 0x0a, 0x1a, 0x52, 0x65, 0x73,
+	0x74, 0x6f, 0x72, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x53, 0x74, 0x61, 0x74,
+	0x75, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x1f, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20,
+	0x01, 0x28, 0x0b, 0x32, 0x0f, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x52, 0x65, 0x73, 0x74, 0x6f,
+	0x72, 0x65, 0x49, 0x64, 0x52, 0x02, 0x69, 0x64, 0x12, 0x3f, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74,
+	0x75, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x27, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e,
+	0x52, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x53,
+	0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75,
+	0x73, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x56, 0x0a, 0x06, 0x53, 0x74, 0x61,
+	0x74, 0x75, 0x73, 0x12, 0x15, 0x0a, 0x11, 0x53, 0x54, 0x41, 0x54, 0x45, 0x5f, 0x55, 0x4e, 0x53,
+	0x50, 0x45, 0x43, 0x49, 0x46, 0x49, 0x45, 0x44, 0x10, 0x00, 0x12, 0x0b, 0x0a, 0x07, 0x52, 0x55,
+	0x4e, 0x4e, 0x49, 0x4e, 0x47, 0x10, 0x01, 0x12, 0x0d, 0x0a, 0x09, 0x53, 0x55, 0x43, 0x43, 0x45,
+	0x45, 0x44, 0x45, 0x44, 0x10, 0x02, 0x12, 0x0a, 0x0a, 0x06, 0x46, 0x41, 0x49, 0x4c, 0x45, 0x44,
+	0x10, 0x03, 0x12, 0x0d, 0x0a, 0x09, 0x4e, 0x4f, 0x54, 0x5f, 0x46, 0x4f, 0x55, 0x4e, 0x44, 0x10,
+	0x06, 0x22, 0x3d, 0x0a, 0x16, 0x52, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x53, 0x6e, 0x61, 0x70,
+	0x73, 0x68, 0x6f, 0x74, 0x43, 0x6f, 0x6d, 0x6d, 0x61, 0x6e, 0x64, 0x12, 0x23, 0x0a, 0x0d, 0x73,
+	0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x66, 0x69, 0x6c, 0x65, 0x18, 0x01, 0x20, 0x01,
+	0x28, 0x09, 0x52, 0x0c, 0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x46, 0x69, 0x6c, 0x65,
+	0x32, 0x5a, 0x0a, 0x12, 0x4a, 0x69, 0x6e, 0x61, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75,
+	0x65, 0x73, 0x74, 0x52, 0x50, 0x43, 0x12, 0x44, 0x0a, 0x0c, 0x70, 0x72, 0x6f, 0x63, 0x65, 0x73,
+	0x73, 0x5f, 0x64, 0x61, 0x74, 0x61, 0x12, 0x1a, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x44, 0x61,
+	0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f,
+	0x74, 0x6f, 0x1a, 0x16, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65,
+	0x71, 0x75, 0x65, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x00, 0x32, 0x63, 0x0a, 0x18,
+	0x4a, 0x69, 0x6e, 0x61, 0x53, 0x69, 0x6e, 0x67, 0x6c, 0x65, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65,
+	0x71, 0x75, 0x65, 0x73, 0x74, 0x52, 0x50, 0x43, 0x12, 0x47, 0x0a, 0x13, 0x70, 0x72, 0x6f, 0x63,
+	0x65, 0x73, 0x73, 0x5f, 0x73, 0x69, 0x6e, 0x67, 0x6c, 0x65, 0x5f, 0x64, 0x61, 0x74, 0x61, 0x12,
+	0x16, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65,
+	0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x16, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x44,
+	0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x22,
+	0x00, 0x32, 0x47, 0x0a, 0x07, 0x4a, 0x69, 0x6e, 0x61, 0x52, 0x50, 0x43, 0x12, 0x3c, 0x0a, 0x04,
+	0x43, 0x61, 0x6c, 0x6c, 0x12, 0x16, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x44, 0x61, 0x74, 0x61,
+	0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x16, 0x2e, 0x6a,
+	0x69, 0x6e, 0x61, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x50,
+	0x72, 0x6f, 0x74, 0x6f, 0x22, 0x00, 0x28, 0x01, 0x30, 0x01, 0x32, 0x60, 0x0a, 0x18, 0x4a, 0x69,
+	0x6e, 0x61, 0x44, 0x69, 0x73, 0x63, 0x6f, 0x76, 0x65, 0x72, 0x45, 0x6e, 0x64, 0x70, 0x6f, 0x69,
+	0x6e, 0x74, 0x73, 0x52, 0x50, 0x43, 0x12, 0x44, 0x0a, 0x12, 0x65, 0x6e, 0x64, 0x70, 0x6f, 0x69,
+	0x6e, 0x74, 0x5f, 0x64, 0x69, 0x73, 0x63, 0x6f, 0x76, 0x65, 0x72, 0x79, 0x12, 0x16, 0x2e, 0x67,
+	0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x45,
+	0x6d, 0x70, 0x74, 0x79, 0x1a, 0x14, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x45, 0x6e, 0x64, 0x70,
+	0x6f, 0x69, 0x6e, 0x74, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x00, 0x32, 0x4e, 0x0a, 0x14,
+	0x4a, 0x69, 0x6e, 0x61, 0x47, 0x61, 0x74, 0x65, 0x77, 0x61, 0x79, 0x44, 0x72, 0x79, 0x52, 0x75,
+	0x6e, 0x52, 0x50, 0x43, 0x12, 0x36, 0x0a, 0x07, 0x64, 0x72, 0x79, 0x5f, 0x72, 0x75, 0x6e, 0x12,
+	0x16, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75,
+	0x66, 0x2e, 0x45, 0x6d, 0x70, 0x74, 0x79, 0x1a, 0x11, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x53,
+	0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x00, 0x32, 0x47, 0x0a, 0x0b,
+	0x4a, 0x69, 0x6e, 0x61, 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x50, 0x43, 0x12, 0x38, 0x0a, 0x07, 0x5f,
+	0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x16, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e,
+	0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x45, 0x6d, 0x70, 0x74, 0x79, 0x1a, 0x13,
+	0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x4a, 0x69, 0x6e, 0x61, 0x49, 0x6e, 0x66, 0x6f, 0x50, 0x72,
+	0x6f, 0x74, 0x6f, 0x22, 0x00, 0x32, 0x57, 0x0a, 0x14, 0x4a, 0x69, 0x6e, 0x61, 0x45, 0x78, 0x65,
+	0x63, 0x75, 0x74, 0x6f, 0x72, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x12, 0x3f, 0x0a,
+	0x08, 0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x12, 0x16, 0x2e, 0x67, 0x6f, 0x6f, 0x67,
+	0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x45, 0x6d, 0x70, 0x74,
+	0x79, 0x1a, 0x19, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f,
+	0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x00, 0x32, 0x60,
+	0x0a, 0x1c, 0x4a, 0x69, 0x6e, 0x61, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x6f, 0x72, 0x53, 0x6e,
+	0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x50, 0x72, 0x6f, 0x67, 0x72, 0x65, 0x73, 0x73, 0x12, 0x40,
+	0x0a, 0x0f, 0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x75,
+	0x73, 0x12, 0x10, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f,
+	0x74, 0x49, 0x64, 0x1a, 0x19, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x53, 0x6e, 0x61, 0x70, 0x73,
+	0x68, 0x6f, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x00,
+	0x32, 0x62, 0x0a, 0x13, 0x4a, 0x69, 0x6e, 0x61, 0x45, 0x78, 0x65, 0x63, 0x75, 0x74, 0x6f, 0x72,
+	0x52, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x12, 0x4b, 0x0a, 0x07, 0x72, 0x65, 0x73, 0x74, 0x6f,
+	0x72, 0x65, 0x12, 0x1c, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x52, 0x65, 0x73, 0x74, 0x6f, 0x72,
+	0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x43, 0x6f, 0x6d, 0x6d, 0x61, 0x6e, 0x64,
+	0x1a, 0x20, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x52, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x53,
+	0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x50, 0x72, 0x6f,
+	0x74, 0x6f, 0x22, 0x00, 0x32, 0x64, 0x0a, 0x1b, 0x4a, 0x69, 0x6e, 0x61, 0x45, 0x78, 0x65, 0x63,
+	0x75, 0x74, 0x6f, 0x72, 0x52, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x50, 0x72, 0x6f, 0x67, 0x72,
+	0x65, 0x73, 0x73, 0x12, 0x45, 0x0a, 0x0e, 0x72, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x5f, 0x73,
+	0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x0f, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x52, 0x65, 0x73,
+	0x74, 0x6f, 0x72, 0x65, 0x49, 0x64, 0x1a, 0x20, 0x2e, 0x6a, 0x69, 0x6e, 0x61, 0x2e, 0x52, 0x65,
+	0x73, 0x74, 0x6f, 0x72, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x53, 0x74, 0x61,
+	0x74, 0x75, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x00, 0x42, 0x15, 0x5a, 0x13, 0x6a, 0x72,
+	0x61, 0x66, 0x74, 0x2f, 0x6a, 0x69, 0x6e, 0x61, 0x2d, 0x67, 0x6f, 0x2d, 0x70, 0x72, 0x6f, 0x74,
+	0x6f, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
+}
+
+var (
+	file_jina_proto_rawDescOnce sync.Once
+	file_jina_proto_rawDescData = file_jina_proto_rawDesc
+)
+
+func file_jina_proto_rawDescGZIP() []byte {
+	file_jina_proto_rawDescOnce.Do(func() {
+		file_jina_proto_rawDescData = protoimpl.X.CompressGZIP(file_jina_proto_rawDescData)
+	})
+	return file_jina_proto_rawDescData
+}
+
+var file_jina_proto_enumTypes = make([]protoimpl.EnumInfo, 3)
+var file_jina_proto_msgTypes = make([]protoimpl.MessageInfo, 18)
+var file_jina_proto_goTypes = []interface{}{
+	(StatusProto_StatusCode)(0),               // 0: jina.StatusProto.StatusCode
+	(SnapshotStatusProto_Status)(0),           // 1: jina.SnapshotStatusProto.Status
+	(RestoreSnapshotStatusProto_Status)(0),    // 2: jina.RestoreSnapshotStatusProto.Status
+	(*RouteProto)(nil),                        // 3: jina.RouteProto
+	(*JinaInfoProto)(nil),                     // 4: jina.JinaInfoProto
+	(*HeaderProto)(nil),                       // 5: jina.HeaderProto
+	(*EndpointsProto)(nil),                    // 6: jina.EndpointsProto
+	(*StatusProto)(nil),                       // 7: jina.StatusProto
+	(*RelatedEntity)(nil),                     // 8: jina.RelatedEntity
+	(*DataRequestProto)(nil),                  // 9: jina.DataRequestProto
+	(*DataRequestProtoWoData)(nil),            // 10: jina.DataRequestProtoWoData
+	(*DataRequestListProto)(nil),              // 11: jina.DataRequestListProto
+	(*SnapshotId)(nil),                        // 12: jina.SnapshotId
+	(*RestoreId)(nil),                         // 13: jina.RestoreId
+	(*SnapshotStatusProto)(nil),               // 14: jina.SnapshotStatusProto
+	(*RestoreSnapshotStatusProto)(nil),        // 15: jina.RestoreSnapshotStatusProto
+	(*RestoreSnapshotCommand)(nil),            // 16: jina.RestoreSnapshotCommand
+	nil,                                       // 17: jina.JinaInfoProto.JinaEntry
+	nil,                                       // 18: jina.JinaInfoProto.EnvsEntry
+	(*StatusProto_ExceptionProto)(nil),        // 19: jina.StatusProto.ExceptionProto
+	(*DataRequestProto_DataContentProto)(nil), // 20: jina.DataRequestProto.DataContentProto
+	(*timestamp.Timestamp)(nil),               // 21: google.protobuf.Timestamp
+	(*_struct.Struct)(nil),                    // 22: google.protobuf.Struct
+	(*docarray_go_proto.DocListProto)(nil),    // 23: docarray.DocListProto
+	(*empty.Empty)(nil),                       // 24: google.protobuf.Empty
+}
+var file_jina_proto_depIdxs = []int32{
+	21, // 0: jina.RouteProto.start_time:type_name -> google.protobuf.Timestamp
+	21, // 1: jina.RouteProto.end_time:type_name -> google.protobuf.Timestamp
+	7,  // 2: jina.RouteProto.status:type_name -> jina.StatusProto
+	17, // 3: jina.JinaInfoProto.jina:type_name -> jina.JinaInfoProto.JinaEntry
+	18, // 4: jina.JinaInfoProto.envs:type_name -> jina.JinaInfoProto.EnvsEntry
+	7,  // 5: jina.HeaderProto.status:type_name -> jina.StatusProto
+	0,  // 6: jina.StatusProto.code:type_name -> jina.StatusProto.StatusCode
+	19, // 7: jina.StatusProto.exception:type_name -> jina.StatusProto.ExceptionProto
+	5,  // 8: jina.DataRequestProto.header:type_name -> jina.HeaderProto
+	22, // 9: jina.DataRequestProto.parameters:type_name -> google.protobuf.Struct
+	3,  // 10: jina.DataRequestProto.routes:type_name -> jina.RouteProto
+	20, // 11: jina.DataRequestProto.data:type_name -> jina.DataRequestProto.DataContentProto
+	5,  // 12: jina.DataRequestProtoWoData.header:type_name -> jina.HeaderProto
+	22, // 13: jina.DataRequestProtoWoData.parameters:type_name -> google.protobuf.Struct
+	3,  // 14: jina.DataRequestProtoWoData.routes:type_name -> jina.RouteProto
+	9,  // 15: jina.DataRequestListProto.requests:type_name -> jina.DataRequestProto
+	12, // 16: jina.SnapshotStatusProto.id:type_name -> jina.SnapshotId
+	1,  // 17: jina.SnapshotStatusProto.status:type_name -> jina.SnapshotStatusProto.Status
+	13, // 18: jina.RestoreSnapshotStatusProto.id:type_name -> jina.RestoreId
+	2,  // 19: jina.RestoreSnapshotStatusProto.status:type_name -> jina.RestoreSnapshotStatusProto.Status
+	23, // 20: jina.DataRequestProto.DataContentProto.docs:type_name -> docarray.DocListProto
+	11, // 21: jina.JinaDataRequestRPC.process_data:input_type -> jina.DataRequestListProto
+	9,  // 22: jina.JinaSingleDataRequestRPC.process_single_data:input_type -> jina.DataRequestProto
+	9,  // 23: jina.JinaRPC.Call:input_type -> jina.DataRequestProto
+	24, // 24: jina.JinaDiscoverEndpointsRPC.endpoint_discovery:input_type -> google.protobuf.Empty
+	24, // 25: jina.JinaGatewayDryRunRPC.dry_run:input_type -> google.protobuf.Empty
+	24, // 26: jina.JinaInfoRPC._status:input_type -> google.protobuf.Empty
+	24, // 27: jina.JinaExecutorSnapshot.snapshot:input_type -> google.protobuf.Empty
+	12, // 28: jina.JinaExecutorSnapshotProgress.snapshot_status:input_type -> jina.SnapshotId
+	16, // 29: jina.JinaExecutorRestore.restore:input_type -> jina.RestoreSnapshotCommand
+	13, // 30: jina.JinaExecutorRestoreProgress.restore_status:input_type -> jina.RestoreId
+	9,  // 31: jina.JinaDataRequestRPC.process_data:output_type -> jina.DataRequestProto
+	9,  // 32: jina.JinaSingleDataRequestRPC.process_single_data:output_type -> jina.DataRequestProto
+	9,  // 33: jina.JinaRPC.Call:output_type -> jina.DataRequestProto
+	6,  // 34: jina.JinaDiscoverEndpointsRPC.endpoint_discovery:output_type -> jina.EndpointsProto
+	7,  // 35: jina.JinaGatewayDryRunRPC.dry_run:output_type -> jina.StatusProto
+	4,  // 36: jina.JinaInfoRPC._status:output_type -> jina.JinaInfoProto
+	14, // 37: jina.JinaExecutorSnapshot.snapshot:output_type -> jina.SnapshotStatusProto
+	14, // 38: jina.JinaExecutorSnapshotProgress.snapshot_status:output_type -> jina.SnapshotStatusProto
+	15, // 39: jina.JinaExecutorRestore.restore:output_type -> jina.RestoreSnapshotStatusProto
+	15, // 40: jina.JinaExecutorRestoreProgress.restore_status:output_type -> jina.RestoreSnapshotStatusProto
+	31, // [31:41] is the sub-list for method output_type
+	21, // [21:31] is the sub-list for method input_type
+	21, // [21:21] is the sub-list for extension type_name
+	21, // [21:21] is the sub-list for extension extendee
+	0,  // [0:21] is the sub-list for field type_name
+}
+
+func init() { file_jina_proto_init() }
+func file_jina_proto_init() {
+	if File_jina_proto != nil {
+		return
+	}
+	if !protoimpl.UnsafeEnabled {
+		file_jina_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*RouteProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*JinaInfoProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*HeaderProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*EndpointsProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*StatusProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*RelatedEntity); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*DataRequestProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[7].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*DataRequestProtoWoData); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[8].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*DataRequestListProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[9].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*SnapshotId); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[10].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*RestoreId); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[11].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*SnapshotStatusProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[12].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*RestoreSnapshotStatusProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[13].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*RestoreSnapshotCommand); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[16].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*StatusProto_ExceptionProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+		file_jina_proto_msgTypes[17].Exporter = func(v interface{}, i int) interface{} {
+			switch v := v.(*DataRequestProto_DataContentProto); i {
+			case 0:
+				return &v.state
+			case 1:
+				return &v.sizeCache
+			case 2:
+				return &v.unknownFields
+			default:
+				return nil
+			}
+		}
+	}
+	file_jina_proto_msgTypes[2].OneofWrappers = []interface{}{}
+	file_jina_proto_msgTypes[5].OneofWrappers = []interface{}{}
+	file_jina_proto_msgTypes[17].OneofWrappers = []interface{}{
+		(*DataRequestProto_DataContentProto_Docs)(nil),
+		(*DataRequestProto_DataContentProto_DocsBytes)(nil),
+	}
+	type x struct{}
+	out := protoimpl.TypeBuilder{
+		File: protoimpl.DescBuilder{
+			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
+			RawDescriptor: file_jina_proto_rawDesc,
+			NumEnums:      3,
+			NumMessages:   18,
+			NumExtensions: 0,
+			NumServices:   10,
+		},
+		GoTypes:           file_jina_proto_goTypes,
+		DependencyIndexes: file_jina_proto_depIdxs,
+		EnumInfos:         file_jina_proto_enumTypes,
+		MessageInfos:      file_jina_proto_msgTypes,
+	}.Build()
+	File_jina_proto = out.File
+	file_jina_proto_rawDesc = nil
+	file_jina_proto_goTypes = nil
+	file_jina_proto_depIdxs = nil
+}
diff --git a/jina/serve/consensus/jina-go-proto/jina_grpc.pb.go b/jina/serve/consensus/jina-go-proto/jina_grpc.pb.go
new file mode 100644
index 0000000000000..e089a033cdaef
--- /dev/null
+++ b/jina/serve/consensus/jina-go-proto/jina_grpc.pb.go
@@ -0,0 +1,922 @@
+// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
+// versions:
+// - protoc-gen-go-grpc v1.2.0
+// - protoc             v3.13.0
+// source: jina.proto
+
+package jina_go_proto
+
+import (
+	context "context"
+	empty "github.com/golang/protobuf/ptypes/empty"
+	grpc "google.golang.org/grpc"
+	codes "google.golang.org/grpc/codes"
+	status "google.golang.org/grpc/status"
+)
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the grpc package it is being compiled against.
+// Requires gRPC-Go v1.32.0 or later.
+const _ = grpc.SupportPackageIsVersion7
+
+// JinaDataRequestRPCClient is the client API for JinaDataRequestRPC service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaDataRequestRPCClient interface {
+	// Used for passing DataRequests to the Executors
+	ProcessData(ctx context.Context, in *DataRequestListProto, opts ...grpc.CallOption) (*DataRequestProto, error)
+}
+
+type jinaDataRequestRPCClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaDataRequestRPCClient(cc grpc.ClientConnInterface) JinaDataRequestRPCClient {
+	return &jinaDataRequestRPCClient{cc}
+}
+
+func (c *jinaDataRequestRPCClient) ProcessData(ctx context.Context, in *DataRequestListProto, opts ...grpc.CallOption) (*DataRequestProto, error) {
+	out := new(DataRequestProto)
+	err := c.cc.Invoke(ctx, "/jina.JinaDataRequestRPC/process_data", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// JinaDataRequestRPCServer is the server API for JinaDataRequestRPC service.
+// All implementations must embed UnimplementedJinaDataRequestRPCServer
+// for forward compatibility
+type JinaDataRequestRPCServer interface {
+	// Used for passing DataRequests to the Executors
+	ProcessData(context.Context, *DataRequestListProto) (*DataRequestProto, error)
+	mustEmbedUnimplementedJinaDataRequestRPCServer()
+}
+
+// UnimplementedJinaDataRequestRPCServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaDataRequestRPCServer struct {
+}
+
+func (UnimplementedJinaDataRequestRPCServer) ProcessData(context.Context, *DataRequestListProto) (*DataRequestProto, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method ProcessData not implemented")
+}
+func (UnimplementedJinaDataRequestRPCServer) mustEmbedUnimplementedJinaDataRequestRPCServer() {}
+
+// UnsafeJinaDataRequestRPCServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaDataRequestRPCServer will
+// result in compilation errors.
+type UnsafeJinaDataRequestRPCServer interface {
+	mustEmbedUnimplementedJinaDataRequestRPCServer()
+}
+
+func RegisterJinaDataRequestRPCServer(s grpc.ServiceRegistrar, srv JinaDataRequestRPCServer) {
+	s.RegisterService(&JinaDataRequestRPC_ServiceDesc, srv)
+}
+
+func _JinaDataRequestRPC_ProcessData_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(DataRequestListProto)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(JinaDataRequestRPCServer).ProcessData(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/jina.JinaDataRequestRPC/process_data",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(JinaDataRequestRPCServer).ProcessData(ctx, req.(*DataRequestListProto))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// JinaDataRequestRPC_ServiceDesc is the grpc.ServiceDesc for JinaDataRequestRPC service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaDataRequestRPC_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaDataRequestRPC",
+	HandlerType: (*JinaDataRequestRPCServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "process_data",
+			Handler:    _JinaDataRequestRPC_ProcessData_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "jina.proto",
+}
+
+// JinaSingleDataRequestRPCClient is the client API for JinaSingleDataRequestRPC service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaSingleDataRequestRPCClient interface {
+	// Used for passing DataRequests to the Executors
+	ProcessSingleData(ctx context.Context, in *DataRequestProto, opts ...grpc.CallOption) (*DataRequestProto, error)
+}
+
+type jinaSingleDataRequestRPCClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaSingleDataRequestRPCClient(cc grpc.ClientConnInterface) JinaSingleDataRequestRPCClient {
+	return &jinaSingleDataRequestRPCClient{cc}
+}
+
+func (c *jinaSingleDataRequestRPCClient) ProcessSingleData(ctx context.Context, in *DataRequestProto, opts ...grpc.CallOption) (*DataRequestProto, error) {
+	out := new(DataRequestProto)
+	err := c.cc.Invoke(ctx, "/jina.JinaSingleDataRequestRPC/process_single_data", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// JinaSingleDataRequestRPCServer is the server API for JinaSingleDataRequestRPC service.
+// All implementations must embed UnimplementedJinaSingleDataRequestRPCServer
+// for forward compatibility
+type JinaSingleDataRequestRPCServer interface {
+	// Used for passing DataRequests to the Executors
+	ProcessSingleData(context.Context, *DataRequestProto) (*DataRequestProto, error)
+	mustEmbedUnimplementedJinaSingleDataRequestRPCServer()
+}
+
+// UnimplementedJinaSingleDataRequestRPCServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaSingleDataRequestRPCServer struct {
+}
+
+func (UnimplementedJinaSingleDataRequestRPCServer) ProcessSingleData(context.Context, *DataRequestProto) (*DataRequestProto, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method ProcessSingleData not implemented")
+}
+func (UnimplementedJinaSingleDataRequestRPCServer) mustEmbedUnimplementedJinaSingleDataRequestRPCServer() {
+}
+
+// UnsafeJinaSingleDataRequestRPCServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaSingleDataRequestRPCServer will
+// result in compilation errors.
+type UnsafeJinaSingleDataRequestRPCServer interface {
+	mustEmbedUnimplementedJinaSingleDataRequestRPCServer()
+}
+
+func RegisterJinaSingleDataRequestRPCServer(s grpc.ServiceRegistrar, srv JinaSingleDataRequestRPCServer) {
+	s.RegisterService(&JinaSingleDataRequestRPC_ServiceDesc, srv)
+}
+
+func _JinaSingleDataRequestRPC_ProcessSingleData_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(DataRequestProto)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(JinaSingleDataRequestRPCServer).ProcessSingleData(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/jina.JinaSingleDataRequestRPC/process_single_data",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(JinaSingleDataRequestRPCServer).ProcessSingleData(ctx, req.(*DataRequestProto))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// JinaSingleDataRequestRPC_ServiceDesc is the grpc.ServiceDesc for JinaSingleDataRequestRPC service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaSingleDataRequestRPC_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaSingleDataRequestRPC",
+	HandlerType: (*JinaSingleDataRequestRPCServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "process_single_data",
+			Handler:    _JinaSingleDataRequestRPC_ProcessSingleData_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "jina.proto",
+}
+
+// JinaRPCClient is the client API for JinaRPC service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaRPCClient interface {
+	// Pass in a Request and a filled Request with matches will be returned.
+	Call(ctx context.Context, opts ...grpc.CallOption) (JinaRPC_CallClient, error)
+}
+
+type jinaRPCClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaRPCClient(cc grpc.ClientConnInterface) JinaRPCClient {
+	return &jinaRPCClient{cc}
+}
+
+func (c *jinaRPCClient) Call(ctx context.Context, opts ...grpc.CallOption) (JinaRPC_CallClient, error) {
+	stream, err := c.cc.NewStream(ctx, &JinaRPC_ServiceDesc.Streams[0], "/jina.JinaRPC/Call", opts...)
+	if err != nil {
+		return nil, err
+	}
+	x := &jinaRPCCallClient{stream}
+	return x, nil
+}
+
+type JinaRPC_CallClient interface {
+	Send(*DataRequestProto) error
+	Recv() (*DataRequestProto, error)
+	grpc.ClientStream
+}
+
+type jinaRPCCallClient struct {
+	grpc.ClientStream
+}
+
+func (x *jinaRPCCallClient) Send(m *DataRequestProto) error {
+	return x.ClientStream.SendMsg(m)
+}
+
+func (x *jinaRPCCallClient) Recv() (*DataRequestProto, error) {
+	m := new(DataRequestProto)
+	if err := x.ClientStream.RecvMsg(m); err != nil {
+		return nil, err
+	}
+	return m, nil
+}
+
+// JinaRPCServer is the server API for JinaRPC service.
+// All implementations must embed UnimplementedJinaRPCServer
+// for forward compatibility
+type JinaRPCServer interface {
+	// Pass in a Request and a filled Request with matches will be returned.
+	Call(JinaRPC_CallServer) error
+	mustEmbedUnimplementedJinaRPCServer()
+}
+
+// UnimplementedJinaRPCServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaRPCServer struct {
+}
+
+func (UnimplementedJinaRPCServer) Call(JinaRPC_CallServer) error {
+	return status.Errorf(codes.Unimplemented, "method Call not implemented")
+}
+func (UnimplementedJinaRPCServer) mustEmbedUnimplementedJinaRPCServer() {}
+
+// UnsafeJinaRPCServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaRPCServer will
+// result in compilation errors.
+type UnsafeJinaRPCServer interface {
+	mustEmbedUnimplementedJinaRPCServer()
+}
+
+func RegisterJinaRPCServer(s grpc.ServiceRegistrar, srv JinaRPCServer) {
+	s.RegisterService(&JinaRPC_ServiceDesc, srv)
+}
+
+func _JinaRPC_Call_Handler(srv interface{}, stream grpc.ServerStream) error {
+	return srv.(JinaRPCServer).Call(&jinaRPCCallServer{stream})
+}
+
+type JinaRPC_CallServer interface {
+	Send(*DataRequestProto) error
+	Recv() (*DataRequestProto, error)
+	grpc.ServerStream
+}
+
+type jinaRPCCallServer struct {
+	grpc.ServerStream
+}
+
+func (x *jinaRPCCallServer) Send(m *DataRequestProto) error {
+	return x.ServerStream.SendMsg(m)
+}
+
+func (x *jinaRPCCallServer) Recv() (*DataRequestProto, error) {
+	m := new(DataRequestProto)
+	if err := x.ServerStream.RecvMsg(m); err != nil {
+		return nil, err
+	}
+	return m, nil
+}
+
+// JinaRPC_ServiceDesc is the grpc.ServiceDesc for JinaRPC service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaRPC_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaRPC",
+	HandlerType: (*JinaRPCServer)(nil),
+	Methods:     []grpc.MethodDesc{},
+	Streams: []grpc.StreamDesc{
+		{
+			StreamName:    "Call",
+			Handler:       _JinaRPC_Call_Handler,
+			ServerStreams: true,
+			ClientStreams: true,
+		},
+	},
+	Metadata: "jina.proto",
+}
+
+// JinaDiscoverEndpointsRPCClient is the client API for JinaDiscoverEndpointsRPC service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaDiscoverEndpointsRPCClient interface {
+	EndpointDiscovery(ctx context.Context, in *empty.Empty, opts ...grpc.CallOption) (*EndpointsProto, error)
+}
+
+type jinaDiscoverEndpointsRPCClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaDiscoverEndpointsRPCClient(cc grpc.ClientConnInterface) JinaDiscoverEndpointsRPCClient {
+	return &jinaDiscoverEndpointsRPCClient{cc}
+}
+
+func (c *jinaDiscoverEndpointsRPCClient) EndpointDiscovery(ctx context.Context, in *empty.Empty, opts ...grpc.CallOption) (*EndpointsProto, error) {
+	out := new(EndpointsProto)
+	err := c.cc.Invoke(ctx, "/jina.JinaDiscoverEndpointsRPC/endpoint_discovery", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// JinaDiscoverEndpointsRPCServer is the server API for JinaDiscoverEndpointsRPC service.
+// All implementations must embed UnimplementedJinaDiscoverEndpointsRPCServer
+// for forward compatibility
+type JinaDiscoverEndpointsRPCServer interface {
+	EndpointDiscovery(context.Context, *empty.Empty) (*EndpointsProto, error)
+	mustEmbedUnimplementedJinaDiscoverEndpointsRPCServer()
+}
+
+// UnimplementedJinaDiscoverEndpointsRPCServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaDiscoverEndpointsRPCServer struct {
+}
+
+func (UnimplementedJinaDiscoverEndpointsRPCServer) EndpointDiscovery(context.Context, *empty.Empty) (*EndpointsProto, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method EndpointDiscovery not implemented")
+}
+func (UnimplementedJinaDiscoverEndpointsRPCServer) mustEmbedUnimplementedJinaDiscoverEndpointsRPCServer() {
+}
+
+// UnsafeJinaDiscoverEndpointsRPCServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaDiscoverEndpointsRPCServer will
+// result in compilation errors.
+type UnsafeJinaDiscoverEndpointsRPCServer interface {
+	mustEmbedUnimplementedJinaDiscoverEndpointsRPCServer()
+}
+
+func RegisterJinaDiscoverEndpointsRPCServer(s grpc.ServiceRegistrar, srv JinaDiscoverEndpointsRPCServer) {
+	s.RegisterService(&JinaDiscoverEndpointsRPC_ServiceDesc, srv)
+}
+
+func _JinaDiscoverEndpointsRPC_EndpointDiscovery_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(empty.Empty)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(JinaDiscoverEndpointsRPCServer).EndpointDiscovery(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/jina.JinaDiscoverEndpointsRPC/endpoint_discovery",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(JinaDiscoverEndpointsRPCServer).EndpointDiscovery(ctx, req.(*empty.Empty))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// JinaDiscoverEndpointsRPC_ServiceDesc is the grpc.ServiceDesc for JinaDiscoverEndpointsRPC service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaDiscoverEndpointsRPC_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaDiscoverEndpointsRPC",
+	HandlerType: (*JinaDiscoverEndpointsRPCServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "endpoint_discovery",
+			Handler:    _JinaDiscoverEndpointsRPC_EndpointDiscovery_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "jina.proto",
+}
+
+// JinaGatewayDryRunRPCClient is the client API for JinaGatewayDryRunRPC service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaGatewayDryRunRPCClient interface {
+	DryRun(ctx context.Context, in *empty.Empty, opts ...grpc.CallOption) (*StatusProto, error)
+}
+
+type jinaGatewayDryRunRPCClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaGatewayDryRunRPCClient(cc grpc.ClientConnInterface) JinaGatewayDryRunRPCClient {
+	return &jinaGatewayDryRunRPCClient{cc}
+}
+
+func (c *jinaGatewayDryRunRPCClient) DryRun(ctx context.Context, in *empty.Empty, opts ...grpc.CallOption) (*StatusProto, error) {
+	out := new(StatusProto)
+	err := c.cc.Invoke(ctx, "/jina.JinaGatewayDryRunRPC/dry_run", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// JinaGatewayDryRunRPCServer is the server API for JinaGatewayDryRunRPC service.
+// All implementations must embed UnimplementedJinaGatewayDryRunRPCServer
+// for forward compatibility
+type JinaGatewayDryRunRPCServer interface {
+	DryRun(context.Context, *empty.Empty) (*StatusProto, error)
+	mustEmbedUnimplementedJinaGatewayDryRunRPCServer()
+}
+
+// UnimplementedJinaGatewayDryRunRPCServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaGatewayDryRunRPCServer struct {
+}
+
+func (UnimplementedJinaGatewayDryRunRPCServer) DryRun(context.Context, *empty.Empty) (*StatusProto, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method DryRun not implemented")
+}
+func (UnimplementedJinaGatewayDryRunRPCServer) mustEmbedUnimplementedJinaGatewayDryRunRPCServer() {}
+
+// UnsafeJinaGatewayDryRunRPCServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaGatewayDryRunRPCServer will
+// result in compilation errors.
+type UnsafeJinaGatewayDryRunRPCServer interface {
+	mustEmbedUnimplementedJinaGatewayDryRunRPCServer()
+}
+
+func RegisterJinaGatewayDryRunRPCServer(s grpc.ServiceRegistrar, srv JinaGatewayDryRunRPCServer) {
+	s.RegisterService(&JinaGatewayDryRunRPC_ServiceDesc, srv)
+}
+
+func _JinaGatewayDryRunRPC_DryRun_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(empty.Empty)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(JinaGatewayDryRunRPCServer).DryRun(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/jina.JinaGatewayDryRunRPC/dry_run",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(JinaGatewayDryRunRPCServer).DryRun(ctx, req.(*empty.Empty))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// JinaGatewayDryRunRPC_ServiceDesc is the grpc.ServiceDesc for JinaGatewayDryRunRPC service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaGatewayDryRunRPC_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaGatewayDryRunRPC",
+	HandlerType: (*JinaGatewayDryRunRPCServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "dry_run",
+			Handler:    _JinaGatewayDryRunRPC_DryRun_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "jina.proto",
+}
+
+// JinaInfoRPCClient is the client API for JinaInfoRPC service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaInfoRPCClient interface {
+	XStatus(ctx context.Context, in *empty.Empty, opts ...grpc.CallOption) (*JinaInfoProto, error)
+}
+
+type jinaInfoRPCClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaInfoRPCClient(cc grpc.ClientConnInterface) JinaInfoRPCClient {
+	return &jinaInfoRPCClient{cc}
+}
+
+func (c *jinaInfoRPCClient) XStatus(ctx context.Context, in *empty.Empty, opts ...grpc.CallOption) (*JinaInfoProto, error) {
+	out := new(JinaInfoProto)
+	err := c.cc.Invoke(ctx, "/jina.JinaInfoRPC/_status", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// JinaInfoRPCServer is the server API for JinaInfoRPC service.
+// All implementations must embed UnimplementedJinaInfoRPCServer
+// for forward compatibility
+type JinaInfoRPCServer interface {
+	XStatus(context.Context, *empty.Empty) (*JinaInfoProto, error)
+	mustEmbedUnimplementedJinaInfoRPCServer()
+}
+
+// UnimplementedJinaInfoRPCServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaInfoRPCServer struct {
+}
+
+func (UnimplementedJinaInfoRPCServer) XStatus(context.Context, *empty.Empty) (*JinaInfoProto, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method XStatus not implemented")
+}
+func (UnimplementedJinaInfoRPCServer) mustEmbedUnimplementedJinaInfoRPCServer() {}
+
+// UnsafeJinaInfoRPCServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaInfoRPCServer will
+// result in compilation errors.
+type UnsafeJinaInfoRPCServer interface {
+	mustEmbedUnimplementedJinaInfoRPCServer()
+}
+
+func RegisterJinaInfoRPCServer(s grpc.ServiceRegistrar, srv JinaInfoRPCServer) {
+	s.RegisterService(&JinaInfoRPC_ServiceDesc, srv)
+}
+
+func _JinaInfoRPC_XStatus_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(empty.Empty)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(JinaInfoRPCServer).XStatus(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/jina.JinaInfoRPC/_status",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(JinaInfoRPCServer).XStatus(ctx, req.(*empty.Empty))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// JinaInfoRPC_ServiceDesc is the grpc.ServiceDesc for JinaInfoRPC service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaInfoRPC_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaInfoRPC",
+	HandlerType: (*JinaInfoRPCServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "_status",
+			Handler:    _JinaInfoRPC_XStatus_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "jina.proto",
+}
+
+// JinaExecutorSnapshotClient is the client API for JinaExecutorSnapshot service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaExecutorSnapshotClient interface {
+	Snapshot(ctx context.Context, in *empty.Empty, opts ...grpc.CallOption) (*SnapshotStatusProto, error)
+}
+
+type jinaExecutorSnapshotClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaExecutorSnapshotClient(cc grpc.ClientConnInterface) JinaExecutorSnapshotClient {
+	return &jinaExecutorSnapshotClient{cc}
+}
+
+func (c *jinaExecutorSnapshotClient) Snapshot(ctx context.Context, in *empty.Empty, opts ...grpc.CallOption) (*SnapshotStatusProto, error) {
+	out := new(SnapshotStatusProto)
+	err := c.cc.Invoke(ctx, "/jina.JinaExecutorSnapshot/snapshot", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// JinaExecutorSnapshotServer is the server API for JinaExecutorSnapshot service.
+// All implementations must embed UnimplementedJinaExecutorSnapshotServer
+// for forward compatibility
+type JinaExecutorSnapshotServer interface {
+	Snapshot(context.Context, *empty.Empty) (*SnapshotStatusProto, error)
+	mustEmbedUnimplementedJinaExecutorSnapshotServer()
+}
+
+// UnimplementedJinaExecutorSnapshotServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaExecutorSnapshotServer struct {
+}
+
+func (UnimplementedJinaExecutorSnapshotServer) Snapshot(context.Context, *empty.Empty) (*SnapshotStatusProto, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Snapshot not implemented")
+}
+func (UnimplementedJinaExecutorSnapshotServer) mustEmbedUnimplementedJinaExecutorSnapshotServer() {}
+
+// UnsafeJinaExecutorSnapshotServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaExecutorSnapshotServer will
+// result in compilation errors.
+type UnsafeJinaExecutorSnapshotServer interface {
+	mustEmbedUnimplementedJinaExecutorSnapshotServer()
+}
+
+func RegisterJinaExecutorSnapshotServer(s grpc.ServiceRegistrar, srv JinaExecutorSnapshotServer) {
+	s.RegisterService(&JinaExecutorSnapshot_ServiceDesc, srv)
+}
+
+func _JinaExecutorSnapshot_Snapshot_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(empty.Empty)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(JinaExecutorSnapshotServer).Snapshot(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/jina.JinaExecutorSnapshot/snapshot",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(JinaExecutorSnapshotServer).Snapshot(ctx, req.(*empty.Empty))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// JinaExecutorSnapshot_ServiceDesc is the grpc.ServiceDesc for JinaExecutorSnapshot service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaExecutorSnapshot_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaExecutorSnapshot",
+	HandlerType: (*JinaExecutorSnapshotServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "snapshot",
+			Handler:    _JinaExecutorSnapshot_Snapshot_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "jina.proto",
+}
+
+// JinaExecutorSnapshotProgressClient is the client API for JinaExecutorSnapshotProgress service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaExecutorSnapshotProgressClient interface {
+	SnapshotStatus(ctx context.Context, in *SnapshotId, opts ...grpc.CallOption) (*SnapshotStatusProto, error)
+}
+
+type jinaExecutorSnapshotProgressClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaExecutorSnapshotProgressClient(cc grpc.ClientConnInterface) JinaExecutorSnapshotProgressClient {
+	return &jinaExecutorSnapshotProgressClient{cc}
+}
+
+func (c *jinaExecutorSnapshotProgressClient) SnapshotStatus(ctx context.Context, in *SnapshotId, opts ...grpc.CallOption) (*SnapshotStatusProto, error) {
+	out := new(SnapshotStatusProto)
+	err := c.cc.Invoke(ctx, "/jina.JinaExecutorSnapshotProgress/snapshot_status", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// JinaExecutorSnapshotProgressServer is the server API for JinaExecutorSnapshotProgress service.
+// All implementations must embed UnimplementedJinaExecutorSnapshotProgressServer
+// for forward compatibility
+type JinaExecutorSnapshotProgressServer interface {
+	SnapshotStatus(context.Context, *SnapshotId) (*SnapshotStatusProto, error)
+	mustEmbedUnimplementedJinaExecutorSnapshotProgressServer()
+}
+
+// UnimplementedJinaExecutorSnapshotProgressServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaExecutorSnapshotProgressServer struct {
+}
+
+func (UnimplementedJinaExecutorSnapshotProgressServer) SnapshotStatus(context.Context, *SnapshotId) (*SnapshotStatusProto, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method SnapshotStatus not implemented")
+}
+func (UnimplementedJinaExecutorSnapshotProgressServer) mustEmbedUnimplementedJinaExecutorSnapshotProgressServer() {
+}
+
+// UnsafeJinaExecutorSnapshotProgressServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaExecutorSnapshotProgressServer will
+// result in compilation errors.
+type UnsafeJinaExecutorSnapshotProgressServer interface {
+	mustEmbedUnimplementedJinaExecutorSnapshotProgressServer()
+}
+
+func RegisterJinaExecutorSnapshotProgressServer(s grpc.ServiceRegistrar, srv JinaExecutorSnapshotProgressServer) {
+	s.RegisterService(&JinaExecutorSnapshotProgress_ServiceDesc, srv)
+}
+
+func _JinaExecutorSnapshotProgress_SnapshotStatus_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(SnapshotId)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(JinaExecutorSnapshotProgressServer).SnapshotStatus(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/jina.JinaExecutorSnapshotProgress/snapshot_status",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(JinaExecutorSnapshotProgressServer).SnapshotStatus(ctx, req.(*SnapshotId))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// JinaExecutorSnapshotProgress_ServiceDesc is the grpc.ServiceDesc for JinaExecutorSnapshotProgress service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaExecutorSnapshotProgress_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaExecutorSnapshotProgress",
+	HandlerType: (*JinaExecutorSnapshotProgressServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "snapshot_status",
+			Handler:    _JinaExecutorSnapshotProgress_SnapshotStatus_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "jina.proto",
+}
+
+// JinaExecutorRestoreClient is the client API for JinaExecutorRestore service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaExecutorRestoreClient interface {
+	Restore(ctx context.Context, in *RestoreSnapshotCommand, opts ...grpc.CallOption) (*RestoreSnapshotStatusProto, error)
+}
+
+type jinaExecutorRestoreClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaExecutorRestoreClient(cc grpc.ClientConnInterface) JinaExecutorRestoreClient {
+	return &jinaExecutorRestoreClient{cc}
+}
+
+func (c *jinaExecutorRestoreClient) Restore(ctx context.Context, in *RestoreSnapshotCommand, opts ...grpc.CallOption) (*RestoreSnapshotStatusProto, error) {
+	out := new(RestoreSnapshotStatusProto)
+	err := c.cc.Invoke(ctx, "/jina.JinaExecutorRestore/restore", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// JinaExecutorRestoreServer is the server API for JinaExecutorRestore service.
+// All implementations must embed UnimplementedJinaExecutorRestoreServer
+// for forward compatibility
+type JinaExecutorRestoreServer interface {
+	Restore(context.Context, *RestoreSnapshotCommand) (*RestoreSnapshotStatusProto, error)
+	mustEmbedUnimplementedJinaExecutorRestoreServer()
+}
+
+// UnimplementedJinaExecutorRestoreServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaExecutorRestoreServer struct {
+}
+
+func (UnimplementedJinaExecutorRestoreServer) Restore(context.Context, *RestoreSnapshotCommand) (*RestoreSnapshotStatusProto, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Restore not implemented")
+}
+func (UnimplementedJinaExecutorRestoreServer) mustEmbedUnimplementedJinaExecutorRestoreServer() {}
+
+// UnsafeJinaExecutorRestoreServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaExecutorRestoreServer will
+// result in compilation errors.
+type UnsafeJinaExecutorRestoreServer interface {
+	mustEmbedUnimplementedJinaExecutorRestoreServer()
+}
+
+func RegisterJinaExecutorRestoreServer(s grpc.ServiceRegistrar, srv JinaExecutorRestoreServer) {
+	s.RegisterService(&JinaExecutorRestore_ServiceDesc, srv)
+}
+
+func _JinaExecutorRestore_Restore_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(RestoreSnapshotCommand)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(JinaExecutorRestoreServer).Restore(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/jina.JinaExecutorRestore/restore",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(JinaExecutorRestoreServer).Restore(ctx, req.(*RestoreSnapshotCommand))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// JinaExecutorRestore_ServiceDesc is the grpc.ServiceDesc for JinaExecutorRestore service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaExecutorRestore_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaExecutorRestore",
+	HandlerType: (*JinaExecutorRestoreServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "restore",
+			Handler:    _JinaExecutorRestore_Restore_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "jina.proto",
+}
+
+// JinaExecutorRestoreProgressClient is the client API for JinaExecutorRestoreProgress service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type JinaExecutorRestoreProgressClient interface {
+	RestoreStatus(ctx context.Context, in *RestoreId, opts ...grpc.CallOption) (*RestoreSnapshotStatusProto, error)
+}
+
+type jinaExecutorRestoreProgressClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewJinaExecutorRestoreProgressClient(cc grpc.ClientConnInterface) JinaExecutorRestoreProgressClient {
+	return &jinaExecutorRestoreProgressClient{cc}
+}
+
+func (c *jinaExecutorRestoreProgressClient) RestoreStatus(ctx context.Context, in *RestoreId, opts ...grpc.CallOption) (*RestoreSnapshotStatusProto, error) {
+	out := new(RestoreSnapshotStatusProto)
+	err := c.cc.Invoke(ctx, "/jina.JinaExecutorRestoreProgress/restore_status", in, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// JinaExecutorRestoreProgressServer is the server API for JinaExecutorRestoreProgress service.
+// All implementations must embed UnimplementedJinaExecutorRestoreProgressServer
+// for forward compatibility
+type JinaExecutorRestoreProgressServer interface {
+	RestoreStatus(context.Context, *RestoreId) (*RestoreSnapshotStatusProto, error)
+	mustEmbedUnimplementedJinaExecutorRestoreProgressServer()
+}
+
+// UnimplementedJinaExecutorRestoreProgressServer must be embedded to have forward compatible implementations.
+type UnimplementedJinaExecutorRestoreProgressServer struct {
+}
+
+func (UnimplementedJinaExecutorRestoreProgressServer) RestoreStatus(context.Context, *RestoreId) (*RestoreSnapshotStatusProto, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method RestoreStatus not implemented")
+}
+func (UnimplementedJinaExecutorRestoreProgressServer) mustEmbedUnimplementedJinaExecutorRestoreProgressServer() {
+}
+
+// UnsafeJinaExecutorRestoreProgressServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to JinaExecutorRestoreProgressServer will
+// result in compilation errors.
+type UnsafeJinaExecutorRestoreProgressServer interface {
+	mustEmbedUnimplementedJinaExecutorRestoreProgressServer()
+}
+
+func RegisterJinaExecutorRestoreProgressServer(s grpc.ServiceRegistrar, srv JinaExecutorRestoreProgressServer) {
+	s.RegisterService(&JinaExecutorRestoreProgress_ServiceDesc, srv)
+}
+
+func _JinaExecutorRestoreProgress_RestoreStatus_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(RestoreId)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(JinaExecutorRestoreProgressServer).RestoreStatus(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/jina.JinaExecutorRestoreProgress/restore_status",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(JinaExecutorRestoreProgressServer).RestoreStatus(ctx, req.(*RestoreId))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// JinaExecutorRestoreProgress_ServiceDesc is the grpc.ServiceDesc for JinaExecutorRestoreProgress service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var JinaExecutorRestoreProgress_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "jina.JinaExecutorRestoreProgress",
+	HandlerType: (*JinaExecutorRestoreProgressServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "restore_status",
+			Handler:    _JinaExecutorRestoreProgress_RestoreStatus_Handler,
+		},
+	},
+	Streams:  []grpc.StreamDesc{},
+	Metadata: "jina.proto",
+}
diff --git a/jina/serve/consensus/jina.proto b/jina/serve/consensus/jina.proto
new file mode 100644
index 0000000000000..f1c082e62460a
--- /dev/null
+++ b/jina/serve/consensus/jina.proto
@@ -0,0 +1,306 @@
+//protoc --go_out jina-go-proto --go_opt=paths=source_relative --go_opt=Mjina.proto=jina-go-proto=jraft/jina-go-proto --go-grpc_out jina-go-proto --go-grpc_opt=paths=source_relative --experimental_allow_proto3_optional jina.proto
+syntax = "proto3";
+import "google/protobuf/timestamp.proto";
+import "google/protobuf/struct.proto";
+import "google/protobuf/empty.proto";
+import "docarray.proto";
+
+package jina;
+option go_package = "jraft/jina-go-proto";
+
+
+/**
+ * Represents a the route paths of this message as perceived by the Gateway
+ * start_time is set when the Gateway sends a message to a Pod
+ * end_time is set when the Gateway receives a message from a Pod
+ * thus end_time - start_time includes Executor computation, runtime overhead, serialization and network
+ */
+message RouteProto {
+    string executor = 1; // the name of the BasePod
+    google.protobuf.Timestamp start_time = 2; // time when the Gateway starts sending to the Pod
+    google.protobuf.Timestamp end_time = 3; // time when the Gateway received it from the Pod
+    StatusProto status = 4; // the status of the execution
+}
+
+message JinaInfoProto {
+    map<string, string> jina = 1; // information about the system running and package version information including jina
+    map<string, string> envs = 2; // the environment variable setting
+}
+
+
+/**
+ * Represents a Header.
+ * - The header's content will be defined by the user request.
+ * - It will be copied to the envelope.header
+ * - In-flow operations will modify the envelope.header
+ * - While returning, copy envelope.header back to request.header
+ */
+message HeaderProto {
+    string request_id = 1; // the unique ID of this request. Multiple requests with the same ID will be gathered
+
+    StatusProto status = 2; // status info
+
+    optional string exec_endpoint = 3;  // the endpoint specified by `@requests(on='/abc')`
+
+    optional string target_executor = 4; // if set, the request is targeted to certain executor, regex strings
+
+    optional uint32 timeout = 5; // epoch time in seconds after which the request should be dropped
+}
+
+
+/**
+ * Represents the set of Endpoints exposed by an Executor
+ */
+message EndpointsProto {
+
+    // list of endpoints exposed by an Executor
+    repeated string endpoints = 1;
+    repeated string write_endpoints = 2;
+}
+
+/**
+ * Represents a Status
+ */
+message StatusProto {
+
+    enum StatusCode {
+        SUCCESS = 0; // success
+        ERROR = 1; // error
+    }
+
+    // status code
+    StatusCode code = 1;
+
+    // error description of the very first exception
+    string description = 2;
+
+    message ExceptionProto {
+        // the class name of the exception
+        string name = 1;
+
+        // the list of arguments given to the exception constructor.
+        repeated string args = 2;
+
+        // the exception traceback stacks
+        repeated string stacks = 3;
+
+        // the name of the executor bind to that Executor (if applicable)
+        string executor = 4;
+    }
+
+    // the details of the error
+    ExceptionProto exception = 3;
+}
+
+
+/**
+ * Represents an entity (like an ExecutorRuntime)
+ */
+message RelatedEntity {
+    string id = 1; // unique id of the entity, like the name of a pod
+    string address = 2; // address of the entity, could be an IP address, domain name etc, does not include port
+    uint32 port = 3; // port this entity is listening on
+    optional uint32 shard_id = 4; // the id of the shard it belongs to, if it is a shard
+}
+
+
+/**
+ * Represents a DataRequest
+ */
+message DataRequestProto {
+
+    HeaderProto header = 1; // header contains meta info defined by the user
+
+    google.protobuf.Struct parameters = 2; // extra kwargs that will be used in executor
+
+    repeated RouteProto routes = 3; // status info on every routes
+
+    message DataContentProto {
+        oneof documents {
+            docarray.DocListProto docs = 1; // the docs in this request
+            bytes docs_bytes = 2; // the docs in this request as bytes
+        }
+    }
+
+    DataContentProto data = 4; // container for docs and groundtruths
+}
+
+message DataRequestProtoWoData {
+
+    HeaderProto header = 1; // header contains meta info defined by the user
+
+    google.protobuf.Struct parameters = 2; // extra kwargs that will be used in executor
+
+    repeated RouteProto routes = 3; // status info on every routes
+
+}
+
+
+/**
+ * Represents a list of data requests
+ * This should be replaced by streaming
+ */
+message DataRequestListProto {
+    repeated DataRequestProto requests = 1; // requests in this list
+}
+
+/**
+ * jina gRPC service for DataRequests.
+ */
+service JinaDataRequestRPC {
+    // Used for passing DataRequests to the Executors
+    rpc process_data (DataRequestListProto) returns (DataRequestProto) {
+    }
+}
+
+/**
+ * jina gRPC service for DataRequests.
+ * This is used to send requests to Executors when a list of requests is not needed
+ */
+service JinaSingleDataRequestRPC {
+    // Used for passing DataRequests to the Executors
+    rpc process_single_data (DataRequestProto) returns (DataRequestProto) {
+    }
+}
+
+/**
+ * jina streaming gRPC service.
+ */
+service JinaRPC {
+    // Pass in a Request and a filled Request with matches will be returned.
+    rpc Call (stream DataRequestProto) returns (stream DataRequestProto) {
+    }
+}
+
+/**
+ * jina gRPC service to expose Endpoints from Executors.
+ */
+service JinaDiscoverEndpointsRPC {
+    rpc endpoint_discovery (google.protobuf.Empty) returns (EndpointsProto) {
+    }
+}
+
+
+/**
+ * jina gRPC service to expose Endpoints from Executors.
+ */
+service JinaGatewayDryRunRPC {
+    rpc dry_run (google.protobuf.Empty) returns (StatusProto) {
+    }
+}
+
+/**
+ * jina gRPC service to expose information about running jina version and environment.
+ */
+service JinaInfoRPC {
+    rpc _status (google.protobuf.Empty) returns (JinaInfoProto) {
+    }
+}
+
+/**
+* An ID representation for the Snapshot Job
+*/
+message SnapshotId {
+    string value = 1;
+}
+
+/**
+* An ID representation for the Restore Job
+*/
+message RestoreId {
+    string value = 1;
+}
+
+
+/**
+* Represents the status of a Snapshot.
+*/
+message SnapshotStatusProto {
+
+    SnapshotId id = 1; // An ID for the Snapshot Job.
+
+    enum Status {
+        STATE_UNSPECIFIED = 0;
+
+        QUEUED = 1; // Snapshot Job is admitted (validated and persisted) and waiting for resources.
+
+        SCHEDULED = 2; // Snapshot Job is scheduled to run as soon as resource allocation is ready.
+
+        RUNNING = 3; // Resource allocation has been successful. Snapshot Job is RUNNING.
+
+        SUCCEEDED = 4; // Snapshot Job has finished successfully.
+
+        FAILED = 5; // Snapshot Job has failed.
+
+        NOT_FOUND = 6; // If the specific snapshot id is unknown or not found
+    }
+
+    // status code
+    Status status = 2;
+
+    // snapshot file
+    string snapshot_file = 3;
+}
+
+/**
+* jina gRPC service to trigger a snapshot at the Executor Runtime.
+*/
+service JinaExecutorSnapshot {
+    rpc snapshot (google.protobuf.Empty) returns (SnapshotStatusProto) {
+    }
+}
+
+/**
+* jina gRPC service to trigger a snapshot at the Executor Runtime.
+*/
+service JinaExecutorSnapshotProgress {
+    rpc snapshot_status (SnapshotId) returns (SnapshotStatusProto) {
+    }
+}
+
+/**
+* Represents the status of a Restore.
+*/
+message RestoreSnapshotStatusProto {
+
+    RestoreId id = 1; // An ID for the Restore Job.
+
+    enum Status {
+        STATE_UNSPECIFIED = 0;
+
+        RUNNING = 1; // Resource allocation has been successful. Restore Job is RUNNING.
+
+        SUCCEEDED = 2; // Resource Job has finished successfully.
+
+        FAILED = 3; // Resource Job has failed.
+
+        NOT_FOUND = 6; // If the specific restore id is unknown or not found
+    }
+
+    // status code
+    Status status = 2;
+}
+
+/**
+* Commands to restore an Executor from a snapshot file
+*/
+message RestoreSnapshotCommand {
+    // snapshot file
+    string snapshot_file = 1;
+}
+
+/**
+* jina gRPC service to trigger a restore at the Executor Runtime.
+*/
+service JinaExecutorRestore {
+    rpc restore (RestoreSnapshotCommand) returns (RestoreSnapshotStatusProto) {
+    }
+}
+
+/**
+* jina gRPC service to trigger a snapshot at the Executor Runtime.
+*/
+service JinaExecutorRestoreProgress {
+    rpc restore_status (RestoreId) returns (RestoreSnapshotStatusProto) {
+    }
+}
diff --git a/jina/serve/consensus/jina_raft/executor_connection.go b/jina/serve/consensus/jina_raft/executor_connection.go
new file mode 100644
index 0000000000000..7d3fed88abd38
--- /dev/null
+++ b/jina/serve/consensus/jina_raft/executor_connection.go
@@ -0,0 +1,29 @@
+package server
+
+import (
+    "google.golang.org/grpc/credentials/insecure"
+    "google.golang.org/grpc"
+    hclog "github.com/hashicorp/go-hclog"
+)
+
+func defaultExecutorDialOptions() []grpc.DialOption {
+    return []grpc.DialOption{
+        grpc.WithTransportCredentials(insecure.NewCredentials()),
+        grpc.WithDefaultCallOptions(grpc.WaitForReady(true)),
+    }
+}
+
+type executor struct {
+    target             string
+    connection_options []grpc.DialOption
+    Logger             hclog.Logger
+}
+
+func (executor *executor) newConnection() (*grpc.ClientConn, error) {
+    conn, err := grpc.Dial(executor.target, executor.connection_options...)
+    if err != nil {
+        executor.Logger.Error("Dialing failed", "error", err)
+        return nil, err
+    }
+    return conn, err
+}
diff --git a/jina/serve/consensus/jina_raft/fsm.go b/jina/serve/consensus/jina_raft/fsm.go
new file mode 100644
index 0000000000000..9907cb18401d1
--- /dev/null
+++ b/jina/serve/consensus/jina_raft/fsm.go
@@ -0,0 +1,321 @@
+package server
+
+import (
+    "context"
+    "fmt"
+    "os"
+    "io"
+    "io/ioutil"
+    "log"
+    "sync"
+    "time"
+    "errors"
+
+    "google.golang.org/protobuf/proto"
+    "google.golang.org/protobuf/types/known/emptypb"
+    healthpb "google.golang.org/grpc/health/grpc_health_v1"
+    empty "github.com/golang/protobuf/ptypes/empty"
+
+    "github.com/hashicorp/raft"
+    pb "jraft/jina-go-proto"
+    hclog "github.com/hashicorp/go-hclog"
+)
+
+
+type executorFSM struct {
+    executor *executor
+    mtx      sync.RWMutex
+    snapshot *snapshot
+    write_endpoints []string
+    RaftID   string
+    logger   hclog.Logger
+}
+
+
+func NewExecutorFSM(target string, LogLevel string, raftID string) *executorFSM {
+    fsm_logger := hclog.New(&hclog.LoggerOptions{
+                    Name:   "executorFSM-" + raftID,
+                    Level:  hclog.LevelFromString(LogLevel),
+                })
+    executor := &executor{
+                target:             target,
+                connection_options: defaultExecutorDialOptions(),
+                Logger: fsm_logger,
+                }
+
+    conn, _ := executor.newConnection()
+    defer conn.Close()
+    client := pb.NewJinaDiscoverEndpointsRPCClient(conn)
+    response, err := client.EndpointDiscovery(context.Background(), &emptypb.Empty{})
+    if err != nil {
+        fsm_logger.Error("Error getting endpoints discovery", "error", err)
+    }
+    write_endpoints := response.WriteEndpoints
+    fsm_logger.Debug("List of endpoints that should trigger Raft Apply:", "endpoints", write_endpoints)
+    return &executorFSM{
+        executor: executor,
+        write_endpoints: write_endpoints,
+        logger: fsm_logger,
+        RaftID: raftID,
+    }
+}
+
+
+func DummyExecutorFSM() *executorFSM {
+    fsm_logger := hclog.New(&hclog.LoggerOptions{
+                    Name:   "executorFSM-dummy",
+                    Level:  hclog.LevelFromString("INFO"),
+                })
+    executor := &executor{
+                target:             "0.0.0.0:54321", //TODO: randomize
+                connection_options: defaultExecutorDialOptions(),
+                Logger: fsm_logger,
+            }
+
+    return &executorFSM{
+        executor: executor,
+        write_endpoints: []string{},
+        logger: fsm_logger,
+        RaftID: "dummy",
+
+    }
+}
+
+func (executor *executorFSM) isSnapshotInProgress() bool {
+    if executor.snapshot != nil &&
+        *executor.snapshot.status == pb.SnapshotStatusProto_RUNNING {
+        return true
+    }
+    return false
+}
+
+// triggered once the followers have committed the log
+func (fsm *executorFSM) Apply(l *raft.Log) interface{} {
+    fsm.mtx.Lock()
+    defer fsm.mtx.Unlock()
+    fsm.logger.Debug("Apply new log entry")
+    for {
+        if !fsm.isSnapshotInProgress() {
+            // we need not to return error but make it slow, wait until not anymore in progress
+            break
+        }
+        fsm.logger.Error("cannot execute Apply because a snapshot is in progress")
+        time.Sleep(1 * time.Second)
+    }
+    if fsm.isSnapshotInProgress() {
+        // we need not to return error but make it slow, wait until not anymore in progress
+        fsm.logger.Error("Cannot accept new requests when snap shotting is in progress.")
+        return fmt.Errorf("Cannot accept new requests when snap shotting is in progress.")
+    }
+    conn, err := fsm.executor.newConnection()
+    if err != nil {
+        return err
+    }
+    defer conn.Close()
+    client := pb.NewJinaSingleDataRequestRPCClient(conn)
+    dataRequestProto := &pb.DataRequestProto{}
+    err = proto.Unmarshal(l.Data, dataRequestProto)
+    if err != nil {
+        fsm.logger.Error("Error while unmarshalling log into DataRequestProto", "error", err)
+        return err
+    }
+
+    response, err := client.ProcessSingleData(context.Background(), dataRequestProto)
+    if err != nil {
+        fsm.logger.Error("Error when calling Executor", "error", err)
+        return err
+    }
+    fsm.logger.Debug("Return Apply Log Response")
+    return response
+}
+
+func (fsm *executorFSM) Snapshot() (raft.FSMSnapshot, error) {
+    // Make sure that any future calls to f.Apply() don't change the snapshot.
+    fsm.mtx.Lock()
+    defer fsm.mtx.Unlock()
+    fsm.logger.Debug("Snapshot FSM state")
+    conn, err := fsm.executor.newConnection()
+    if err != nil {
+        fsm.logger.Error("Error setting a new connection with Executor", "error", err)
+        return nil, err
+    }
+    defer conn.Close()
+    client := pb.NewJinaExecutorSnapshotClient(conn)
+    response, err := client.Snapshot(context.Background(), &emptypb.Empty{})
+    if err != nil {
+        fsm.logger.Error("Error triggering a snapshot", "error", err)
+        return nil, err
+    }
+    snapshot := &snapshot{
+        executor:          fsm.executor,
+        id:                response.Id,
+        status:            &response.Status,
+        snapshotFile:      response.SnapshotFile,
+        Logger:            fsm.logger,
+    }
+    fsm.snapshot = snapshot
+    fsm.logger.Debug("Snapshot successfully attached to FSM")
+    return snapshot, nil
+}
+
+func (fsm *executorFSM) Restore(r io.ReadCloser) error {
+    // I think restore here is not well set
+    fsm.logger.Debug("Restore FSM state")
+    bytes, err := io.ReadAll(r)
+    // write bytes to temporary file, and pass it in the request
+    if err != nil {
+        fsm.logger.Error("Error reading bytes from the snapshot file", "error", err)
+        return err
+    }
+    tempDir := os.TempDir()
+    file, err := ioutil.TempFile(tempDir, "temp")
+    if err != nil {
+        log.Print(err)
+    }
+    defer os.Remove(file.Name()) // remove the file when done
+
+     fsm.logger.Debug("Temporary file name where to write state", "filename", file.Name())
+
+    // Write some data to the file
+    if _, err := file.Write(bytes); err != nil {
+        fsm.logger.Error("Error writing snapshot bytes to temporary file", "error", err)
+        return err
+    }
+
+    // Close the file
+    if err := file.Close(); err != nil {
+        fsm.logger.Error("Error closing file", "error", err)
+        return err
+    }
+    fsm.logger.Debug("Calling Executor to request restore")
+    conn, err := fsm.executor.newConnection()
+    if err != nil {
+        fsm.logger.Error("Error setting a new connection with Executor", "error", err)
+        return err
+    }
+    defer conn.Close()
+    client := pb.NewJinaExecutorRestoreClient(conn)
+    restoreCommandProto := &pb.RestoreSnapshotCommand{}
+    restoreCommandProto.SnapshotFile = file.Name()
+    restoreResponse, err := client.Restore(context.Background(), restoreCommandProto)
+    if err != nil {
+        fsm.logger.Error("Restore command to Executor failed", "error", err)
+        return err
+    }
+    fsm.logger.Debug("Start Checking status of Restore")
+    ticker := time.NewTicker(1 * time.Second)
+    done := make(chan bool)
+    defer close(done)
+    timeout := time.NewTimer(500 * time.Second)
+
+    go func(funcTicker *time.Ticker) {
+        for {
+            select {
+            case t := <-funcTicker.C:
+                fsm.logger.Debug("Checking restore status at", "time", t)
+                conn, err = fsm.executor.newConnection()
+                if err == nil {
+                    defer conn.Close()
+                    client := pb.NewJinaExecutorRestoreProgressClient(conn)
+                    response, err := client.RestoreStatus(context.Background(), restoreResponse.Id)
+                    if err != nil {
+                        fsm.logger.Error("Error fetching restore status for", "ID", restoreResponse.Id, "error", err)
+                    } else {
+                        fsm.logger.Debug("Restore", "status", response.Status, "at time", t)
+                        if response.Status == pb.RestoreSnapshotStatusProto_FAILED ||
+                            response.Status == pb.RestoreSnapshotStatusProto_SUCCEEDED {
+                            if response.Status == pb.RestoreSnapshotStatusProto_FAILED {
+                                 err = errors.New("Restoring Executor failed")
+                            }
+                            timeout.Stop()
+                            done <- true
+                            return
+                        }
+                    }
+                }
+            case <-timeout.C:
+                fsm.logger.Error("Timed out waiting for restore status.")
+                timeout.Stop()
+                done <- true
+                return
+            }
+        }
+    }(ticker)
+    <-done
+    ticker.Stop()
+    return err
+}
+
+func (fsm *executorFSM) Read(ctx context.Context, dataRequestProto *pb.DataRequestProto) (*pb.DataRequestProto, error) {
+    fsm.logger.Debug("Call Read Endpoint")
+    conn, err := fsm.executor.newConnection()
+    if err != nil {
+        fsm.logger.Error("Error setting a new connection with Executor", "error", err)
+        return nil, err
+    }
+    defer conn.Close()
+    client := pb.NewJinaSingleDataRequestRPCClient(conn)
+    response, err := client.ProcessSingleData(ctx, dataRequestProto)
+    if err != nil {
+        fsm.logger.Error("Error calling Read endpoint", "error", err)
+        return nil, err
+    }
+    fsm.logger.Debug("Return Read Endpoint Response")
+    return response, err
+}
+
+func (fsm *executorFSM) EndpointDiscovery(ctx context.Context, empty *empty.Empty) (*pb.EndpointsProto, error) {
+    fsm.logger.Debug("Call EndpointDiscovery")
+    conn, err := fsm.executor.newConnection()
+    if err != nil {
+        fsm.logger.Error("Error setting a new connection with Executor", "error", err)
+        return nil, err
+    }
+    defer conn.Close()
+    client := pb.NewJinaDiscoverEndpointsRPCClient(conn)
+    response, err := client.EndpointDiscovery(ctx, empty)
+    if err != nil {
+        fsm.logger.Error("Error calling EndpointDiscovery endpoint", "error", err)
+        return nil, err
+    }
+    fsm.logger.Debug("Return EndpointDiscovery Response")
+    return response, err
+}
+
+
+func (fsm *executorFSM) XStatus(ctx context.Context, empty *empty.Empty) (*pb.JinaInfoProto, error) {
+    fsm.logger.Debug("Call XStatus")
+    conn, err := fsm.executor.newConnection()
+    if err != nil {
+        fsm.logger.Error("Error setting a new connection with Executor", "error", err)
+        return nil, err
+    }
+    defer conn.Close()
+    client := pb.NewJinaInfoRPCClient(conn)
+    response, err := client.XStatus(ctx, empty)
+    if err != nil {
+        fsm.logger.Error("Error calling Status endpoint", "error", err)
+        return nil, err
+    }
+    fsm.logger.Debug("Return XStatus Response")
+    return response, err
+}
+
+
+func (fsm *executorFSM) Check(ctx context.Context, req *healthpb.HealthCheckRequest) (*healthpb.HealthCheckResponse, error) {
+    fsm.logger.Debug("Call Check")
+    conn, err := fsm.executor.newConnection()
+    if err != nil {
+        fsm.logger.Error("Error setting a new connection with Executor", "error", err)
+        return nil, err
+    }
+    defer conn.Close()
+    resp, err := healthpb.NewHealthClient(conn).Check(ctx, req)
+    if err != nil {
+        fsm.logger.Error("Error calling Check endpoint", "error", err)
+        return nil, err
+    }
+    fsm.logger.Debug("Return Check Response")
+    return resp, err
+}
+
diff --git a/jina/serve/consensus/jina_raft/jinaraft.go b/jina/serve/consensus/jina_raft/jinaraft.go
new file mode 100644
index 0000000000000..12e3aac94dd93
--- /dev/null
+++ b/jina/serve/consensus/jina_raft/jinaraft.go
@@ -0,0 +1,77 @@
+package server
+
+import (
+    "fmt"
+    "os"
+    "github.com/hashicorp/raft"
+    hclog "github.com/hashicorp/go-hclog"
+)
+
+// Jina re-implementation of raft.GetConfiguration. Returns the persisted configuration of the Raft cluster
+// without starting a Raft instance or connecting to the cluster. This function
+// has identical behavior to Raft.GetConfiguration.
+func JinaGetConfiguration(conf *raft.Config, fsm raft.FSM, logs raft.LogStore, stable raft.StableStore,
+    snaps raft.SnapshotStore, trans raft.Transport) (raft.Configuration, error) {
+    logLevel := os.Getenv("JINA_LOG_LEVEL")
+    if logLevel == "" {
+        logLevel = "INFO"
+    }
+    logger := hclog.New(&hclog.LoggerOptions{
+                    Name:   "GetConfiguration",
+                    Level:  hclog.LevelFromString(logLevel),
+                })
+    logger.Debug("Getting the configuration from the RAFT node")
+    r, err := DummyNewRaft(conf, fsm, logs, stable, snaps, trans)
+    if err != nil {
+        logger.Error("Error creating a New Dummy Raft", "error", err)
+        return raft.Configuration{}, err
+    }
+    future := r.GetConfiguration()
+    if err = future.Error(); err != nil {
+        logger.Error("Error getting configuration: returning an empty configuration", "error", err)
+        return raft.Configuration{}, err
+    }
+    logger.Debug("Obtained the configuration from the RAFT node")
+    return future.Configuration(), nil
+}
+
+// Jina Re-implementation of NewRaft so that restoring is not triggered. DummyNewRaft is used to get a dummy new Raft node.
+// It takes a configuration, as well as implementations of various interfaces that are required. If we have any
+// old state, such as snapshots, logs, peers, etc, all those will ** not ** be restored
+// when creating the Raft node.
+func DummyNewRaft(conf *raft.Config, fsm raft.FSM, logs raft.LogStore, stable raft.StableStore, snaps raft.SnapshotStore, trans raft.Transport) (*raft.Raft, error) {
+    // Validate the configuration.
+    logLevel := os.Getenv("JINA_LOG_LEVEL")
+    if logLevel == "" {
+        logLevel = "INFO"
+    }
+    logger := hclog.New(&hclog.LoggerOptions{
+                    Name:   "DummyNewRaft",
+                    Level:  hclog.LevelFromString(logLevel),
+                })
+
+    if err := raft.ValidateConfig(conf); err != nil {
+        logger.Error("Error validating the Configuration", "error", err)
+        return nil, err
+    }
+
+    // Read the index of the last log entry.
+    lastIndex, err := logs.LastIndex()
+    if err != nil {
+        logger.Error("Error getting the last index: %v", err)
+        return nil, fmt.Errorf("Error getting the last index", "error", err)
+    }
+
+    // Get the last log entry.
+    var lastLog raft.Log
+    if lastIndex > 0 {
+        if err = logs.GetLog(lastIndex, &lastLog); err != nil {
+            logger.Error("Error getting the last index", "error", err)
+            return nil, fmt.Errorf("Error getting the last log at index %d: %v", lastIndex, err)
+        }
+    }
+
+    // Create Raft struct.
+    r := &raft.Raft{}
+    return r, nil
+}
\ No newline at end of file
diff --git a/jina/serve/consensus/jina_raft/rpc.go b/jina/serve/consensus/jina_raft/rpc.go
new file mode 100644
index 0000000000000..c3116db3db69c
--- /dev/null
+++ b/jina/serve/consensus/jina_raft/rpc.go
@@ -0,0 +1,143 @@
+package server
+
+import (
+    "context"
+    "time"
+    "sync/atomic"
+    "errors"
+    "io"
+
+    "github.com/Jille/raft-grpc-leader-rpc/rafterrors"
+    "github.com/golang/protobuf/proto"
+    empty "github.com/golang/protobuf/ptypes/empty"
+
+    "github.com/hashicorp/raft"
+    pb "jraft/jina-go-proto"
+    healthpb "google.golang.org/grpc/health/grpc_health_v1"
+    hclog "github.com/hashicorp/go-hclog"
+)
+
+type RpcInterface struct {
+    Executor *executorFSM
+    Raft     *raft.Raft
+    Logger   hclog.Logger
+    pb.UnimplementedJinaSingleDataRequestRPCServer
+    pb.UnimplementedJinaDiscoverEndpointsRPCServer
+    pb.UnimplementedJinaInfoRPCServer
+    pb.UnimplementedJinaRPCServer
+}
+
+
+func (rpc *RpcInterface) getRaftState() raft.RaftState {
+    stateAddr := (uint32)(rpc.Raft.State())
+    return raft.RaftState(atomic.LoadUint32(&stateAddr))
+}
+
+
+func (rpc *RpcInterface) Call(stream pb.JinaRPC_CallServer) error {
+  ctx := context.Background()
+  for {
+    req, err := stream.Recv()
+    if err == io.EOF {
+      rpc.Logger.Debug("Streaming received", "EOF", err)
+      return nil
+    }
+    if err != nil {
+      rpc.Logger.Error("Error receiving request in streaming", "error", err)
+      return err
+    }
+    rpc.Logger.Debug("Received request in streaming")
+    // process the input message and generate the response message
+    resp, err := rpc.ProcessSingleData(ctx, req)
+    if err != nil {
+        rpc.Logger.Error("Error processing single data", "error", err)
+        return err
+    }
+    // send the response message back to the client
+    if err := stream.Send(resp); err != nil {
+      rpc.Logger.Error("Error streaming response back", "error", err)
+      return err
+    }
+  }
+}
+
+
+/**
+ * jina gRPC func for DataRequests.
+ * This is used to send requests to Executors when a list of requests is not needed
+ */
+func (rpc *RpcInterface) ProcessSingleData(
+    ctx context.Context,
+    dataRequestProto *pb.DataRequestProto) (*pb.DataRequestProto, error) {
+    rpc.Logger.Debug("Calling ProcessSingleData")
+    endpoint := dataRequestProto.Header.ExecEndpoint
+    found := false
+
+    // Loop through the list and check if the search string is in the list
+    for _, s := range rpc.Executor.write_endpoints {
+        if s == *endpoint {
+            found = true
+            break
+        }
+    }
+
+    if found {
+        rpc.Logger.Debug("Calling a Write Endpoint:", "endpoint", *endpoint)
+        if rpc.getRaftState() == raft.Leader && rpc.Executor.isSnapshotInProgress() {
+            err := errors.New("Leader cannot process write request while Snapshotting")
+            rpc.Logger.Error("Leader cannot process write request while Snapshotting")
+            return nil, err
+        }
+        bytes, err := proto.Marshal(dataRequestProto)
+        if err != nil {
+            rpc.Logger.Error("Error marshalling DataRequestProto into bytes:", "error", err)
+            return nil, err
+        }
+        // replicate logs to the followers and then to itself
+        rpc.Logger.Debug("Call raft.Apply")
+        // here we should read the `on=` from dat
+        future := rpc.Raft.Apply(bytes, time.Second)
+        if err := future.Error(); err != nil {
+            rpc.Logger.Error("Error from calling RAFT apply:", "error", err)
+            return nil, rafterrors.MarkRetriable(err)
+        }
+        response, test := future.Response().(*pb.DataRequestProto)
+        if test {
+            rpc.Logger.Debug("Return from RAFT Apply:", "Response", response.String())
+            return response, nil
+        } else {
+            err := future.Response().(error)
+            return nil, err
+        }
+    } else {
+        rpc.Logger.Debug("Calling a Read Endpoint:", "endpoint", *endpoint)
+        return rpc.Executor.Read(ctx, dataRequestProto)
+    }
+}
+
+func (rpc *RpcInterface) EndpointDiscovery(ctx context.Context, empty *empty.Empty) (*pb.EndpointsProto, error) {
+    rpc.Logger.Debug("Get an Endpoint Discovery Request")
+    return rpc.Executor.EndpointDiscovery(ctx, empty)
+}
+
+func (rpc *RpcInterface) XStatus(ctx context.Context, empty *empty.Empty) (*pb.JinaInfoProto, error) {
+   rpc.Logger.Debug("Get an XStatus Request")
+   return rpc.Executor.XStatus(ctx, empty)
+}
+
+func (rpc *RpcInterface) Check(ctx context.Context, req *healthpb.HealthCheckRequest) (*healthpb.HealthCheckResponse, error) {
+    rpc.Logger.Debug("Get a Check Request")
+    return rpc.Executor.Check(ctx, req)
+}
+
+func (rpc *RpcInterface) Watch(req *healthpb.HealthCheckRequest, stream healthpb.Health_WatchServer) error {
+    rpc.Logger.Debug("Get a Watch Request")
+    healthCheckStatus := &healthpb.HealthCheckResponse{}
+    healthCheckStatus.Status = 1
+    err := stream.Send(healthCheckStatus)
+    if err != nil {
+        rpc.Logger.Error("Error sending back health Check Status", "error", err)
+        return err
+    }
+    return nil
+}
diff --git a/jina/serve/consensus/jina_raft/snapshot.go b/jina/serve/consensus/jina_raft/snapshot.go
new file mode 100644
index 0000000000000..c454803c253c6
--- /dev/null
+++ b/jina/serve/consensus/jina_raft/snapshot.go
@@ -0,0 +1,120 @@
+package server
+
+import (
+    "context"
+    "fmt"
+    "sync"
+    "time"
+    "os"
+    "io"
+
+    "github.com/hashicorp/raft"
+    pb "jraft/jina-go-proto"
+    hclog "github.com/hashicorp/go-hclog"
+)
+
+type snapshot struct {
+    executor          *executor
+    id                *pb.SnapshotId
+    mu                sync.RWMutex
+    status            *pb.SnapshotStatusProto_Status
+    snapshotFile      string
+    Logger            hclog.Logger
+}
+
+func (s *snapshot) Release() {
+}
+
+func (s *snapshot) store(data *pb.SnapshotStatusProto_Status) {
+    s.mu.Lock()
+    defer s.mu.Unlock()
+
+    s.status = data
+}
+
+func (s *snapshot) get() *pb.SnapshotStatusProto_Status {
+    s.mu.RLock()
+    defer s.mu.RUnlock()
+
+    return s.status
+}
+
+func (s *snapshot) Persist(sink raft.SnapshotSink) error {
+    s.Logger.Debug("Start persist operation")
+    var err error
+    ticker := time.NewTicker(1 * time.Second)
+    done := make(chan bool)
+    defer close(done)
+    timeout := time.NewTimer(500 * time.Second)
+    var status *pb.SnapshotStatusProto_Status
+
+    defer func() {
+        if err != nil || (status != nil && *status != pb.SnapshotStatusProto_SUCCEEDED) {
+            failed_snapshot := pb.SnapshotStatusProto_FAILED
+            s.store(&failed_snapshot)
+            sink.Cancel()
+        } else {
+            err = sink.Close()
+        }
+    }()
+
+    go func(funcTicker *time.Ticker) {
+        for {
+            select {
+            case t := <-funcTicker.C:
+                s.Logger.Debug("Checking snapshot status at", "time", t)
+                conn, err := s.executor.newConnection()
+                if err == nil {
+                    defer conn.Close()
+                    client := pb.NewJinaExecutorSnapshotProgressClient(conn)
+                    response, err := client.SnapshotStatus(context.Background(), s.id)
+                    if err != nil {
+                        s.Logger.Error("Error fetching snapshot status for", "ID", response.Id, "error", err)
+                    } else {
+                        s.store(&response.Status)
+                        s.Logger.Debug("Snapshot", "status", response.Status, "at time", t)
+                        if response.Status == pb.SnapshotStatusProto_FAILED ||
+                            response.Status == pb.SnapshotStatusProto_SUCCEEDED {
+                            timeout.Stop()
+                            done <- true
+                            return
+                        }
+                    }
+                }
+            case <-timeout.C:
+                s.Logger.Error("Timed out waiting for snapshot status.")
+                timeout.Stop()
+                done <- true
+                return
+            }
+        }
+    }(ticker)
+
+    <-done
+    ticker.Stop()
+
+    status = s.get()
+    if status != nil && *status != pb.SnapshotStatusProto_SUCCEEDED {
+        msg := fmt.Sprintf("persist job %s failed with status %s", s.id.Value, status)
+        s.Logger.Error(msg)
+        err = fmt.Errorf(msg)
+        return err
+    }
+    source, err := os.Open(s.snapshotFile)
+    if err != nil {
+       s.Logger.Error("Error opening a file where Executor created snapshot", "error", err)
+       return err
+    }
+    defer source.Close()
+
+    _, err = io.Copy(sink, source)
+    if err != nil {
+       s.Logger.Error("Error copying temporary Executor snapshot", "error", err)
+       return err
+    }
+    err = os.Remove(s.snapshotFile)
+    if err != nil {
+       s.Logger.Error("Error removing Executor shanpshot File", "error", err)
+    }
+    return err
+}
diff --git a/jina/serve/consensus/jraft.go b/jina/serve/consensus/jraft.go
new file mode 100644
index 0000000000000..b025c802d0d01
--- /dev/null
+++ b/jina/serve/consensus/jraft.go
@@ -0,0 +1,70 @@
+package main
+
+/*
+
+#cgo pkg-config: python3
+#include <Python.h>
+#include <stdbool.h>
+
+// Workaround missing variadic function support
+// https://github.com/golang/go/issues/975
+int PyArg_ParseTuple_run(PyObject * args, PyObject * kwargs, char **myAddr, char **raftId, char **raftDir, char **executorTarget, int *HeartbeatTimeout, int *ElectionTimeout, int *CommitTimeout, int *MaxAppendEntries, bool *BatchApplyCh, bool *ShutdownOnRemove, uint64_t *TrailingLogs, int *snapshotInterval, uint64_t *SnapshotThreshold, int *LeaderLeaseTimeout, char **LogLevel, bool *NoSnapshotRestoreOnStart) {
+    static char *kwlist[] = {"myAddr", "raftId", "raftDir", "executorTarget", "HeartbeatTimeout", "ElectionTimeout", "CommitTimeout", "MaxAppendEntries", "BatchApplyCh", "ShutdownOnRemove", "TrailingLogs", "SnapshotInterval", "SnapshotThreshold", "LeaderLeaseTimeout", "LogLevel", "NoSnapshotRestoreOnStart", NULL};
+    return PyArg_ParseTupleAndKeywords(args, kwargs, "ssss|llllppklklsp", kwlist, myAddr, raftId, raftDir, executorTarget, HeartbeatTimeout, ElectionTimeout, CommitTimeout, MaxAppendEntries, BatchApplyCh, ShutdownOnRemove, TrailingLogs, snapshotInterval, SnapshotThreshold, LeaderLeaseTimeout, LogLevel, NoSnapshotRestoreOnStart);
+}
+
+int PyArg_ParseTuple_add_voter(PyObject * args, char **a, char **b, char **c) {
+    return PyArg_ParseTuple(args, "sss", a, b, c);
+}
+
+int PyArg_ParseTuple_get_configuration(PyObject * args, char **a, char **b) {
+    return PyArg_ParseTuple(args, "ss", a, b);
+}
+
+PyObject * run(PyObject* , PyObject*, PyObject*);
+
+PyObject * add_voter(PyObject* , PyObject*);
+PyObject * get_configuration(PyObject* , PyObject*);
+
+static PyObject *AddVoterError = NULL;
+
+static PyMethodDef methods[] = {
+    {"run", (PyCFunction)run, METH_VARARGS | METH_KEYWORDS, "Run the raft Node server"},
+    {"add_voter", (PyCFunction)add_voter, METH_VARARGS, "Client to add voter"},
+    {"get_configuration", (PyCFunction)get_configuration, METH_VARARGS, "Get configuration"},
+    {NULL, NULL, 0, NULL}
+};
+
+static struct PyModuleDef jraftmodule = {
+   PyModuleDef_HEAD_INIT, "jraft", NULL, -1, methods
+};
+
+PyMODINIT_FUNC
+PyInit_jraft(void)
+{
+    PyObject *m;
+    m = PyModule_Create(&jraftmodule);
+    if (m == NULL)
+        return NULL;
+
+    AddVoterError = PyErr_NewException("jraft.AddVoterError", PyExc_RuntimeError, NULL);
+    Py_XINCREF(AddVoterError);
+
+    if (PyModule_AddObject(m, "error", AddVoterError) < 0) {
+        Py_XDECREF(AddVoterError);
+        Py_CLEAR(AddVoterError);
+        Py_DECREF(m);
+        return NULL;
+    }
+
+    return m;
+}
+
+void raise_exception(char *msg) {
+    PyErr_SetString(PyExc_ValueError, msg);
+}
+
+*/
+import "C"
+
+
diff --git a/jina/serve/consensus/jraft.h b/jina/serve/consensus/jraft.h
new file mode 100644
index 0000000000000..06753e107f9af
--- /dev/null
+++ b/jina/serve/consensus/jraft.h
@@ -0,0 +1,86 @@
+/* Code generated by cmd/cgo; DO NOT EDIT. */
+
+/* package command-line-arguments */
+
+
+#line 1 "cgo-builtin-export-prolog"
+
+#include <stddef.h> /* for ptrdiff_t below */
+
+#ifndef GO_CGO_EXPORT_PROLOGUE_H
+#define GO_CGO_EXPORT_PROLOGUE_H
+
+#ifndef GO_CGO_GOSTRING_TYPEDEF
+typedef struct { const char *p; ptrdiff_t n; } _GoString_;
+#endif
+
+#endif
+
+/* Start of preamble from import "C" comments.  */
+
+
+#line 3 "run.go"
+ #include <Python.h>
+ #include <stdbool.h>
+ int PyArg_ParseTuple_run(PyObject * args, PyObject * kwargs, char **myAddr, char **raftId, char **raftDir, char **executorTarget, int *HeartbeatTimeout, int *ElectionTimeout, int *CommitTimeout, int *MaxAppendEntries, bool *BatchApplyCh, bool *ShutdownOnRemove, uint64_t *TrailingLogs, int *snapshotInterval, uint64_t *SnapshotThreshold, int *LeaderLeaseTimeout, char **LogLevel, bool *NoSnapshotRestoreOnStart);
+ int PyArg_ParseTuple_add_voter(PyObject * args, char **a, char **b, char **c);
+ int PyArg_ParseTuple_get_configuration(PyObject * args, char **a, char **b);
+ void raise_exception(char *msg);
+
+#line 1 "cgo-generated-wrapper"
+
+
+/* End of preamble from import "C" comments.  */
+
+
+/* Start of boilerplate cgo prologue.  */
+#line 1 "cgo-gcc-export-header-prolog"
+
+#ifndef GO_CGO_PROLOGUE_H
+#define GO_CGO_PROLOGUE_H
+
+typedef signed char GoInt8;
+typedef unsigned char GoUint8;
+typedef short GoInt16;
+typedef unsigned short GoUint16;
+typedef int GoInt32;
+typedef unsigned int GoUint32;
+typedef long long GoInt64;
+typedef unsigned long long GoUint64;
+typedef GoInt64 GoInt;
+typedef GoUint64 GoUint;
+typedef __SIZE_TYPE__ GoUintptr;
+typedef float GoFloat32;
+typedef double GoFloat64;
+typedef float _Complex GoComplex64;
+typedef double _Complex GoComplex128;
+
+/*
+  static assertion to make sure the file is being used on architecture
+  at least with matching size of GoInt.
+*/
+typedef char _check_for_64_bit_pointer_matching_GoInt[sizeof(void*)==64/8 ? 1:-1];
+
+#ifndef GO_CGO_GOSTRING_TYPEDEF
+typedef _GoString_ GoString;
+#endif
+typedef void *GoMap;
+typedef void *GoChan;
+typedef struct { void *t; void *v; } GoInterface;
+typedef struct { void *data; GoInt len; GoInt cap; } GoSlice;
+
+#endif
+
+/* End of boilerplate cgo prologue.  */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+extern PyObject* run(PyObject* self, PyObject* args, PyObject* kwargs);
+extern PyObject* add_voter(PyObject* self, PyObject* args);
+extern PyObject* get_configuration(PyObject* self, PyObject* args);
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/jina/serve/consensus/run.go b/jina/serve/consensus/run.go
new file mode 100644
index 0000000000000..4576125c9dd4b
--- /dev/null
+++ b/jina/serve/consensus/run.go
@@ -0,0 +1,460 @@
+package main
+
+// #include <Python.h>
+// #include <stdbool.h>
+// int PyArg_ParseTuple_run(PyObject * args, PyObject * kwargs, char **myAddr, char **raftId, char **raftDir, char **executorTarget, int *HeartbeatTimeout, int *ElectionTimeout, int *CommitTimeout, int *MaxAppendEntries, bool *BatchApplyCh, bool *ShutdownOnRemove, uint64_t *TrailingLogs, int *snapshotInterval, uint64_t *SnapshotThreshold, int *LeaderLeaseTimeout, char **LogLevel, bool *NoSnapshotRestoreOnStart);
+// int PyArg_ParseTuple_add_voter(PyObject * args, char **a, char **b, char **c);
+// int PyArg_ParseTuple_get_configuration(PyObject * args, char **a, char **b);
+// void raise_exception(char *msg);
+import "C"
+
+
+import (
+    "context"
+    "flag"
+    "fmt"
+    "log"
+    "net"
+    "os"
+    "os/signal"
+    "syscall"
+    "path/filepath"
+    "time"
+    "unsafe"
+    transport "github.com/Jille/raft-grpc-transport"
+    "github.com/Jille/raftadmin"
+    "github.com/hashicorp/raft"
+    boltdb "github.com/hashicorp/raft-boltdb"
+    jinaraft "jraft/jina_raft"
+    pb "jraft/jina-go-proto"
+    "google.golang.org/grpc"
+    "google.golang.org/grpc/credentials/insecure"
+    "google.golang.org/grpc/reflection"
+    healthpb "google.golang.org/grpc/health/grpc_health_v1"
+    hclog "github.com/hashicorp/go-hclog"
+)
+
+func NewRaft(ctx context.Context,
+            myID string,
+            myAddress string,
+            raftDir string,
+            HeartbeatTimeout int,
+            ElectionTimeout int,
+            CommitTimeout int,
+            MaxAppendEntries int,
+            BatchApplyCh bool,
+            ShutdownOnRemove bool,
+            TrailingLogs uint64,
+            SnapshotInterval int,
+            SnapshotThreshold uint64,
+            LeaderLeaseTimeout int,
+            LogLevel string,
+            NoSnapshotRestoreOnStart bool,
+            fsm raft.FSM) (*raft.Raft, *transport.Manager, error) {
+    config := raft.DefaultConfig()
+    config.LocalID = raft.ServerID(myID)
+    config.HeartbeatTimeout         = time.Duration(HeartbeatTimeout) * time.Millisecond
+    config.ElectionTimeout          = time.Duration(ElectionTimeout) * time.Millisecond
+    config.CommitTimeout            = time.Duration(CommitTimeout) * time.Millisecond
+    config.MaxAppendEntries         = MaxAppendEntries
+    config.BatchApplyCh             = BatchApplyCh
+    config.ShutdownOnRemove         = ShutdownOnRemove
+    config.TrailingLogs             = TrailingLogs
+    config.SnapshotInterval         = time.Duration(SnapshotInterval) * time.Second
+    config.SnapshotThreshold        = SnapshotThreshold
+    config.LeaderLeaseTimeout       = time.Duration(LeaderLeaseTimeout) * time.Millisecond
+    config.LogLevel                 = LogLevel
+    config.NoSnapshotRestoreOnStart = NoSnapshotRestoreOnStart
+    config.Logger = hclog.New(&hclog.LoggerOptions{
+                    Name:   "raft-" + myID,
+                    Level:  hclog.LevelFromString(LogLevel),
+                })
+
+    baseDir := filepath.Join(raftDir, myID)
+    err := os.MkdirAll(baseDir, os.ModePerm)
+    if err != nil {
+        fmt.Printf("Error creating baseDir (%v) folder: %v\n", baseDir, err)
+    }
+
+    logs_db, err := boltdb.NewBoltStore(filepath.Join(baseDir, "logs.dat"))
+    if err != nil {
+        return nil, nil, fmt.Errorf(`boltdb.NewBoltStore(%q): %v`, filepath.Join(baseDir, "logs.dat"), err)
+    }
+
+    stable_db, err := boltdb.NewBoltStore(filepath.Join(baseDir, "stable.dat"))
+    if err != nil {
+        return nil, nil, fmt.Errorf(`boltdb.NewBoltStore(%q): %v`, filepath.Join(baseDir, "stable.dat"), err)
+    }
+
+    file_snapshot, err := raft.NewFileSnapshotStore(baseDir, 3, os.Stderr)
+    if err != nil {
+        return nil, nil, fmt.Errorf(`raft.NewFileSnapshotStore(%q, ...): %v`, baseDir, err)
+    }
+
+    tm := transport.New(raft.ServerAddress(myAddress), []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())})
+
+    r, err := raft.NewRaft(config, fsm, logs_db, stable_db, file_snapshot, tm.Transport())
+
+    if err != nil {
+        return nil, nil, fmt.Errorf("raft.NewRaft: %v", err)
+    }
+
+    cfg := raft.Configuration{
+        Servers: []raft.Server{
+            {
+                Suffrage: raft.Voter,
+                ID:       raft.ServerID(myID),
+                Address:  raft.ServerAddress(myAddress),
+            },
+        },
+    }
+    f := r.BootstrapCluster(cfg)
+    // raft bootstrap error can be ignored safely https://github.com/hashicorp/raft/blob/44124c28758b8cfb675e90c75a204a08a84f8d4f/api.go#L220
+    if err := f.Error(); err != nil {
+        return r, tm, nil
+    }
+
+    return r, tm, nil
+}
+
+func Run(myAddr string,
+         raftId string,
+         raftDir string,
+         executorTarget string,
+         HeartbeatTimeout int,
+         ElectionTimeout int,
+         CommitTimeout int,
+         MaxAppendEntries int,
+         BatchApplyCh bool,
+         ShutdownOnRemove bool,
+         TrailingLogs uint64,
+         SnapshotInterval int,
+         SnapshotThreshold uint64,
+         LeaderLeaseTimeout int,
+         LogLevel string,
+         NoSnapshotRestoreOnStart bool) {
+    run_logger := hclog.New(&hclog.LoggerOptions{
+                    Name:   "run_raft-" + raftId,
+                    Level:  hclog.LevelFromString(LogLevel),
+                })
+    if raftId == "" {
+        log.Fatalf("flag --raft_id is required")
+    }
+    run_logger.Info("Running RAFT node in", "address", myAddr, "with the ID", raftId, "in directory", raftDir, "and connecting to Executor", executorTarget)
+    ctx := context.Background()
+    _, port, err := net.SplitHostPort(myAddr)
+    if err != nil {
+        run_logger.Error("failed to parse", "local address", myAddr, "with error", err)
+        log.Fatalf("failed to parse local address (%q): %v", myAddr, err)
+    }
+    run_logger.Debug("starting to listen on", "port", port)
+    sock, err := net.Listen("tcp", fmt.Sprintf(":%s", port))
+    if err != nil {
+        run_logger.Error("failed to listen", "error", err)
+        log.Fatalf("failed to listen: %v", err)
+    }
+    defer sock.Close()
+
+    executorFSM := jinaraft.NewExecutorFSM(executorTarget, LogLevel, raftId)
+
+    r, tm, err := NewRaft(ctx,
+                        raftId,
+                        myAddr,
+                        raftDir,
+                        HeartbeatTimeout,
+                        ElectionTimeout,
+                        CommitTimeout,
+                        MaxAppendEntries,
+                        BatchApplyCh,
+                        ShutdownOnRemove,
+                        TrailingLogs,
+                        SnapshotInterval,
+                        SnapshotThreshold,
+                        LeaderLeaseTimeout,
+                        LogLevel,
+                        NoSnapshotRestoreOnStart,
+                        executorFSM)
+    if err != nil {
+        run_logger.Error("Failed to start RAFT node", "error", err)
+        log.Fatalf("Failed to start RAFT node: %v", err)
+    }
+    grpcServer := grpc.NewServer()
+    rpc_logger := hclog.New(&hclog.LoggerOptions{
+                    Name:   "rpc-" + raftId,
+                    Level:  hclog.LevelFromString(LogLevel),
+                })
+
+    pb.RegisterJinaSingleDataRequestRPCServer(grpcServer, &jinaraft.RpcInterface{
+        Executor: executorFSM,
+        Raft:     r,
+        Logger:   rpc_logger,
+    })
+    pb.RegisterJinaDiscoverEndpointsRPCServer(grpcServer, &jinaraft.RpcInterface{
+        Executor: executorFSM,
+        Raft:     r,
+        Logger:   rpc_logger,
+    })
+    pb.RegisterJinaInfoRPCServer(grpcServer, &jinaraft.RpcInterface{
+        Executor: executorFSM,
+        Raft:     r,
+        Logger:   rpc_logger,
+    })
+    pb.RegisterJinaRPCServer(grpcServer, &jinaraft.RpcInterface{
+                                    Executor: executorFSM,
+                                    Raft:     r,
+                                    Logger:   rpc_logger,
+                              })
+    tm.Register(grpcServer)
+
+    healthpb.RegisterHealthServer(grpcServer, &jinaraft.RpcInterface{
+        Executor: executorFSM,
+        Raft:     r,
+        Logger:   rpc_logger,
+    })
+
+    raftadmin.Register(grpcServer, r)
+    reflection.Register(grpcServer)
+    sigchnl := make(chan os.Signal, 1)
+    signal.Notify(sigchnl, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT, os.Interrupt)
+    go func(){
+        sig := <-sigchnl
+        run_logger.Info("Received", "signal", sig)
+        run_logger.Info("gRPCServer stopping")
+        grpcServer.GracefulStop()
+        sock.Close()
+        shutdownResult := r.Shutdown()
+        err := shutdownResult.Error()
+        if err != nil {
+            run_logger.Error("Error returned while shutting RAFT down", "error", err)
+            log.Fatalf("Error returned while shutting RAFT down: %v", err)
+        }
+        run_logger.Info("RAFT shutdown whithout error")
+    }()
+    if err := grpcServer.Serve(sock); err != nil {
+        run_logger.Error("failed to serve", "error", err)
+        log.Fatalf("failed to serve: %v", err)
+    }
+}
+
+func findServerByID(servers []raft.Server, id raft.ServerID) *raft.Server {
+    for _, server := range servers {
+        if server.ID == id {
+            return &server
+        }
+    }
+    return nil
+}
+
+func main() {
+    raftDefaultConfig := raft.DefaultConfig()
+
+    myAddr                   := flag.String("address", "localhost:50051", "TCP host+port for this node")
+    raftId                   := flag.String("raft_id", "", "Node id used by Raft")
+    raftDir                  := flag.String("raft_data_dir", "data/", "Raft data dir")
+    executorTarget           := flag.String("executor_target", "localhost:54321", "underlying executor host+port")
+    HeartbeatTimeout         := flag.Int("heartbeat_timeout", int(raftDefaultConfig.HeartbeatTimeout / time.Millisecond), "HeartbeatTimeout for the RAFT node")
+    ElectionTimeout          := flag.Int("election_timeout", int(raftDefaultConfig.ElectionTimeout / time.Millisecond), "ElectionTimeout for the RAFT node")
+    CommitTimeout            := flag.Int("commit_timeout", int(raftDefaultConfig.CommitTimeout / time.Millisecond), "CommitTimeout for the RAFT node")
+    MaxAppendEntries         := flag.Int("max_append_entries", raftDefaultConfig.MaxAppendEntries, "MaxAppendEntries for the RAFT node")
+    BatchApplyCh             := flag.Bool("batch_applych", raftDefaultConfig.BatchApplyCh, "BatchApplyCh for the RAFT node")
+    ShutdownOnRemove         := flag.Bool("shutdown_on_remove", raftDefaultConfig.ShutdownOnRemove, "ShutdownOnRemove for the RAFT node")
+    TrailingLogs             := flag.Uint64("trailing_logs", raftDefaultConfig.TrailingLogs, "TrailingLogs for the RAFT node")
+    SnapshotInterval         := flag.Int("snapshot_interval", int(raftDefaultConfig.SnapshotInterval / time.Second), "SnapshotInterval for the RAFT node")
+    SnapshotThreshold        := flag.Uint64("snapshot_threshold", raftDefaultConfig.SnapshotThreshold, "SnapshotThreshold for the RAFT node")
+    LeaderLeaseTimeout       := flag.Int("leader_lease_timeout", int(raftDefaultConfig.LeaderLeaseTimeout / time.Millisecond), "LeaderLeaseTimeout for the RAFT node")
+    LogLevel                 := flag.String("log_level", raftDefaultConfig.LogLevel, "LogLevel for the RAFT node")
+    NoSnapshotRestoreOnStart := flag.Bool("no_snapshot_restore_on_start", raftDefaultConfig.NoSnapshotRestoreOnStart, "NoSnapshotRestoreOnStart for the RAFT node")
+
+    Run(*myAddr,
+        *raftId,
+        *raftDir,
+        *executorTarget,
+        *HeartbeatTimeout,
+        *ElectionTimeout,
+        *CommitTimeout,
+        *MaxAppendEntries,
+        *BatchApplyCh,
+        *ShutdownOnRemove,
+        *TrailingLogs,
+        *SnapshotInterval,
+        *SnapshotThreshold,
+        *LeaderLeaseTimeout,
+        *LogLevel,
+        *NoSnapshotRestoreOnStart)
+}
+
+
+//export run
+func run(self *C.PyObject, args *C.PyObject, kwargs *C.PyObject) *C.PyObject {
+    var myAddr *C.char
+    var raftId *C.char
+    var raftDir *C.char
+    var executorTarget *C.char
+    var HeartbeatTimeout C.int
+    var ElectionTimeout C.int
+    var CommitTimeout C.int
+    var MaxAppendEntries C.int
+    var BatchApplyCh C.bool
+    var ShutdownOnRemove C.bool
+    var TrailingLogs C.uint64_t
+    var SnapshotInterval C.int
+    var SnapshotThreshold C.uint64_t
+    var LeaderLeaseTimeout C.int
+    var LogLevel *C.char
+    var NoSnapshotRestoreOnStart C.bool
+
+    raftDefaultConfig := raft.DefaultConfig()
+    HeartbeatTimeout         = C.int(int64(raftDefaultConfig.HeartbeatTimeout / time.Millisecond))
+    ElectionTimeout          = C.int(int64(raftDefaultConfig.ElectionTimeout / time.Millisecond))
+    CommitTimeout            = C.int(int64(raftDefaultConfig.CommitTimeout / time.Millisecond))
+    MaxAppendEntries         = C.int(raftDefaultConfig.MaxAppendEntries)
+    BatchApplyCh             = C.bool(raftDefaultConfig.BatchApplyCh)
+    ShutdownOnRemove         = C.bool(raftDefaultConfig.ShutdownOnRemove)
+    TrailingLogs             = C.uint64_t(raftDefaultConfig.TrailingLogs)
+    SnapshotInterval         = C.int(raftDefaultConfig.SnapshotInterval / time.Second)
+    SnapshotThreshold        = C.uint64_t(raftDefaultConfig.SnapshotThreshold)
+    LeaderLeaseTimeout       = C.int(raftDefaultConfig.LeaderLeaseTimeout / time.Millisecond)
+    LogLevel                 = C.CString(raftDefaultConfig.LogLevel)
+    NoSnapshotRestoreOnStart = C.bool(raftDefaultConfig.NoSnapshotRestoreOnStart)
+
+    if C.PyArg_ParseTuple_run(args,
+                             kwargs,
+                             &myAddr,
+                             &raftId,
+                             &raftDir,
+                             &executorTarget,
+                             &HeartbeatTimeout,
+                             &ElectionTimeout,
+                             &CommitTimeout,
+                             &MaxAppendEntries,
+                             &BatchApplyCh,
+                             &ShutdownOnRemove,
+                             &TrailingLogs,
+                             &SnapshotInterval,
+                             &SnapshotThreshold,
+                             &LeaderLeaseTimeout,
+                             &LogLevel,
+                             &NoSnapshotRestoreOnStart) != 0 {
+        Run(C.GoString(myAddr),
+            C.GoString(raftId),
+            C.GoString(raftDir),
+            C.GoString(executorTarget),
+            int(HeartbeatTimeout),
+            int(ElectionTimeout),
+            int(CommitTimeout),
+            int(MaxAppendEntries),
+            BatchApplyCh != false,
+            ShutdownOnRemove != false,
+            uint64(TrailingLogs),
+            int(SnapshotInterval),
+            uint64(SnapshotThreshold),
+            int(LeaderLeaseTimeout),
+            C.GoString(LogLevel),
+            NoSnapshotRestoreOnStart != false)
+    }
+    C.Py_IncRef(C.Py_None);
+    return C.Py_None;
+}
+
+//export add_voter
+func add_voter(self *C.PyObject, args *C.PyObject) *C.PyObject {
+    //TODO: Instantiate new logger based on JINA_LOG_LEVEL
+    logLevel := os.Getenv("JINA_LOG_LEVEL")
+    if logLevel == "" {
+        logLevel = "INFO"
+    }
+    logger := hclog.New(&hclog.LoggerOptions{
+                    Name:   "add_voter",
+                    Level:  hclog.LevelFromString(logLevel),
+                })
+    var target *C.char
+    var raftId *C.char
+    var voterAddress *C.char
+    if C.PyArg_ParseTuple_add_voter(args, &target, &raftId, &voterAddress) != 0 {
+        err := AddVoter(C.GoString(target), C.GoString(raftId), C.GoString(voterAddress))
+        if err != nil {
+            logger.Error("Error received calling AddVoter", "error", err)
+            C.raise_exception(C.CString("Error from AddVoter"))
+            return nil
+        }
+    }
+    C.Py_IncRef(C.Py_None);
+    return C.Py_None;
+}
+
+//export get_configuration
+func get_configuration(self *C.PyObject, args *C.PyObject) *C.PyObject {
+    logLevel := os.Getenv("JINA_LOG_LEVEL")
+    if logLevel == "" {
+        logLevel = "INFO"
+    }
+    logger := hclog.New(&hclog.LoggerOptions{
+                    Name:   "get_configuration",
+                    Level:  hclog.LevelFromString(logLevel),
+                })
+
+    var raftId *C.char
+    var raftDir *C.char
+
+    if C.PyArg_ParseTuple_get_configuration(args, &raftId, &raftDir) != 0 {
+        config := raft.DefaultConfig()
+        config.LocalID = raft.ServerID(C.GoString(raftId))
+
+        baseDir := filepath.Join(C.GoString(raftDir), C.GoString(raftId))
+
+        logs_db, err := boltdb.NewBoltStore(filepath.Join(baseDir, "logs.dat"))
+        if logs_db != nil {
+            defer logs_db.Close()
+        }
+
+        if err != nil {
+            C.Py_IncRef(C.Py_None);
+            return C.Py_None;
+        }
+
+        stable_db, err := boltdb.NewBoltStore(filepath.Join(baseDir, "stable.dat"))
+        if stable_db != nil {
+            defer stable_db.Close()
+        }
+
+        if err != nil {
+            C.Py_IncRef(C.Py_None);
+            return C.Py_None;
+        }
+
+        file_snapshot, err := raft.NewFileSnapshotStore(baseDir, 3, os.Stderr)
+        if err != nil {
+            C.Py_IncRef(C.Py_None);
+            return C.Py_None;
+        }
+
+        // dummy transport
+        tm := transport.New(raft.ServerAddress("0.0.0.0:54321"), []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())})
+
+        // this will create a raft object, this is not really needed and we should find a work around so
+        // we can get the configuration wihtout initializing anything else
+        executorFSM := jinaraft.DummyExecutorFSM()
+
+        conf, err := jinaraft.JinaGetConfiguration(config, executorFSM, logs_db, stable_db, file_snapshot, tm.Transport())
+        if err != nil {
+            C.Py_IncRef(C.Py_None);
+            return C.Py_None;
+        } else {
+            logger.Debug("configuration already present in the node:", "configuration", conf, "with number of servers", len(conf.Servers))
+        }
+
+        if len(conf.Servers) == 0 {
+            C.Py_IncRef(C.Py_None);
+            return C.Py_None;
+        }
+
+        server := findServerByID(conf.Servers, raft.ServerID(C.GoString(raftId)))
+        cstr := C.CString(string(server.Address))
+        defer C.free(unsafe.Pointer(cstr))
+        pyStr := C.PyUnicode_FromString(cstr)
+        return pyStr
+
+    }
+    C.raise_exception(C.CString("Error from get_configuration, wrong parameters passed"))
+    return nil
+}
diff --git a/jina/serve/consensus/scripts/fetch-protos.sh b/jina/serve/consensus/scripts/fetch-protos.sh
new file mode 100755
index 0000000000000..e7bd721c1d243
--- /dev/null
+++ b/jina/serve/consensus/scripts/fetch-protos.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+
+set -e
+
+# This script fetches the latest protobuf files from the Docarray repository
+# and copies them to the `jina-core/proto` directory.
+
+# The script is meant to be run from the root of the repository.
+
+DOCARRAY_VERSION=$1
+
+if [ -z "$DOCARRAY_VERSION" ]; then
+    echo "Please provide a Docarray version as the second argument."
+    exit 1
+fi
+
+if [[ $DOCARRAY_VERSION != "v*" ]]; then
+    DOCARRAY_VERSION="v$DOCARRAY_VERSION"
+fi
+
+echo "Fetching protos for Docarray version $DOCARRAY_VERSION"
+wget https://raw.githubusercontent.com/jina-ai/docarray/$DOCARRAY_VERSION/docarray/proto/docarray.proto -O jina/proto/docarray.proto
\ No newline at end of file
diff --git a/jina/serve/consensus/scripts/protogen.sh b/jina/serve/consensus/scripts/protogen.sh
new file mode 100755
index 0000000000000..24e0d286206f7
--- /dev/null
+++ b/jina/serve/consensus/scripts/protogen.sh
@@ -0,0 +1,41 @@
+#/bin/sh
+
+set -e
+
+# This script is used to generate gRPC client stubs from the proto files.
+
+GO_MODULE="jraft"
+DOCARRAY_PROTO="docarray.proto"
+DOCARRAY_DIR="../../docarray"
+DOCARRAY_PACKAGE="$GO_MODULE/docarray"
+
+JINA_PROTO="jina.proto"
+JINA_DIR="../../jina"
+JINA_PACKAGE="$GO_MODULE/jina-go-proto"
+
+
+cd jina/proto
+if ! $(grep -q '^option go_package = ' docarray.proto);then
+       awk '/package docarray;/{print; print "option go_package = \"'${DOCARRAY_PACKAGE}'\";";next}1' docarray.proto > temp.proto
+       mv temp.proto docarray.proto
+fi
+protoc --go_out=${DOCARRAY_DIR} \
+       --go_opt=paths=source_relative \
+       --go_opt=M${DOCARRAY_PROTO}=${DOCARRAY_PACKAGE} \
+       --go-grpc_out=${DOCARRAY_DIR} \
+       --go-grpc_opt=paths=source_relative \
+       --experimental_allow_proto3_optional \
+       ${DOCARRAY_PROTO} 
+
+if ! $(grep -q '^option go_package = ' jina.proto);then
+       awk '/package jina;/{print; print "option go_package = \"'${JINA_PACKAGE}'\";";next}1' jina.proto > temp.proto
+       mv temp.proto jina.proto
+fi
+protoc --go_out=${JINA_DIR} \
+       --go_opt=paths=source_relative \
+       --go_opt=M${JINA_PROTO}=${JINA_PACKAGE} \
+       --go-grpc_out=${JINA_DIR} \
+       --go-grpc_opt=paths=source_relative \
+       --experimental_allow_proto3_optional \
+       ${JINA_PROTO} 
+cd -
diff --git a/jina/serve/executors/__init__.py b/jina/serve/executors/__init__.py
index 77aa02803dab5..fcdebb015ccb2 100644
--- a/jina/serve/executors/__init__.py
+++ b/jina/serve/executors/__init__.py
@@ -7,6 +7,7 @@
 import inspect
 import multiprocessing
 import os
+import threading
 import warnings
 from types import SimpleNamespace
 from typing import (
@@ -41,12 +42,14 @@
     avoid_concurrent_lock_cls,
 )
 from jina.serve.executors.metas import get_executor_taboo
-from jina.serve.helper import store_init_kwargs, wrap_func
+from jina.serve.helper import (
+    _get_workspace_from_name_and_shards,
+    store_init_kwargs,
+    wrap_func,
+)
 from jina.serve.instrumentation import MetricsTimer
 
 if TYPE_CHECKING:  # pragma: no cover
-    import threading
-
     from opentelemetry.context.context import Context
 
 __dry_run_endpoint__ = '_jina_dry_run_'
@@ -249,6 +252,10 @@ def __init__(
         except RuntimeError:
             pass
 
+        self._write_lock = (
+            threading.Lock()
+        )  # watch because this makes it no serializable
+
     def _get_endpoint_models_dict(self):
         from jina._docarray import docarray_v2
 
@@ -354,6 +361,22 @@ def requests(self):
             self._requests = copy.copy(self.requests_by_class[self.__class__.__name__])
             return self._requests
 
+    @property
+    def write_endpoints(self):
+        """
+        Get the list of endpoints bound to write methods
+
+        :return: Returns the list of endpoints bound to write methods
+        """
+        if hasattr(self, '_write_methods'):
+            endpoints = []
+            for endpoint, fn in self.requests.items():
+                if fn.fn.__name__ in self._write_methods:
+                    endpoints.append(endpoint)
+            return endpoints
+        else:
+            return []
+
     def _add_requests(self, _requests: Optional[Dict]):
         if _requests:
             func_names = {f.fn.__name__: e for e, f in self.requests.items()}
@@ -552,17 +575,14 @@ def workspace(self) -> Optional[str]:
             or __cache_path__
         )
         if workspace:
-            complete_workspace = os.path.join(workspace, self.metas.name)
             shard_id = getattr(
                 self.runtime_args,
                 'shard_id',
                 None,
             )
-            if shard_id is not None and shard_id != -1:
-                complete_workspace = os.path.join(complete_workspace, str(shard_id))
-            if not os.path.exists(complete_workspace):
-                os.makedirs(complete_workspace)
-            return os.path.abspath(complete_workspace)
+            return _get_workspace_from_name_and_shards(
+                workspace=workspace, shard_id=shard_id, name=self.metas.name
+            )
 
     def __enter__(self):
         return self
@@ -673,6 +693,7 @@ def serve(
         py_modules: Optional[List[str]] = None,
         quiet: Optional[bool] = False,
         quiet_error: Optional[bool] = False,
+        raft_configuration: Optional[dict] = None,
         reload: Optional[bool] = False,
         replicas: Optional[int] = 1,
         retries: Optional[int] = -1,
@@ -680,6 +701,7 @@ def serve(
         shards: Optional[int] = 1,
         ssl_certfile: Optional[str] = None,
         ssl_keyfile: Optional[str] = None,
+        stateful: Optional[bool] = False,
         timeout_ctrl: Optional[int] = 60,
         timeout_ready: Optional[int] = 600000,
         timeout_send: Optional[int] = None,
@@ -776,6 +798,7 @@ def serve(
           `Executor cookbook <https://docs.jina.ai/concepts/executor/executor-files/>`__
         :param quiet: If set, then no log will be emitted from this object.
         :param quiet_error: If set, then exception stack information will not be added to the log
+        :param raft_configuration: Dictionary of kwargs arguments that will be passed to the RAFT node as configuration options when starting the RAFT node.
         :param reload: If set, the Executor will restart while serving if YAML configuration source or Executor modules are changed. If YAML configuration is changed, the whole deployment is reloaded and new processes will be restarted. If only Python modules of the Executor have changed, they will be reloaded to the interpreter without restarting process.
         :param replicas: The number of replicas in the deployment
         :param retries: Number of retries per gRPC call. If <0 it defaults to max(3, num_replicas)
@@ -783,6 +806,7 @@ def serve(
         :param shards: The number of shards in the deployment running at the same time. For more details check https://docs.jina.ai/concepts/flow/create-flow/#complex-flow-topologies
         :param ssl_certfile: the path to the certificate file
         :param ssl_keyfile: the path to the key file
+        :param stateful: If set, start consensus module to make sure write operations are properly replicated between all the replicas
         :param timeout_ctrl: The timeout in milliseconds of the control request, -1 for waiting forever
         :param timeout_ready: The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever
         :param timeout_send: The timeout in milliseconds used when sending data requests to Executors, -1 means no timeout, disabled by default
@@ -1024,3 +1048,40 @@ def monitor(
             )
 
         return contextlib.nullcontext()
+
+    def snapshot(self, snapshot_file: str):
+        """
+        Interface to take a snapshot from the Executor. Implement it to enable periodic snapshots
+        :param snapshot_file: The file path where to store the binary representation of the Executor snapshot
+        """
+        raise Exception('Raising an Exception. Snapshot is not enabled by default')
+
+    def restore(self, snapshot_file: str):
+        """
+        Interface to restore the state of the Executor from a snapshot that has been taken by the snapshot method.
+        :param snapshot_file: The file path from where to reconstruct the Executor
+        """
+        pass
+
+    def _run_snapshot(self, snapshot_file: str, did_raise_exception):
+        try:
+            from pathlib import Path
+
+            p = Path(snapshot_file)
+            p.parent.mkdir(parents=True, exist_ok=True)
+            p.touch()
+            with self._write_lock:
+                self.snapshot(snapshot_file)
+        except:
+            did_raise_exception.set()
+            raise
+
+    def _run_restore(self, snapshot_file: str, did_raise_exception):
+        try:
+            with self._write_lock:
+                self.restore(snapshot_file)
+        except:
+            did_raise_exception.set()
+            raise
+        finally:
+            os.remove(snapshot_file)
diff --git a/jina/serve/executors/decorators.py b/jina/serve/executors/decorators.py
index 65ee94a66495d..fd21018d70812 100644
--- a/jina/serve/executors/decorators.py
+++ b/jina/serve/executors/decorators.py
@@ -42,9 +42,9 @@ def arg_wrapper(self, *args, **kwargs):
 
             if self.__class__ == cls:
                 with ImportExtensions(
-                    required=False,
-                    help_text=f'FileLock is needed to guarantee non-concurrent initialization of replicas in the '
-                    f'same machine.',
+                        required=False,
+                        help_text=f'FileLock is needed to guarantee non-concurrent initialization of replicas in the '
+                                  f'same machine.',
                 ):
                     import filelock
 
@@ -90,6 +90,122 @@ def _inherit_from_parent_class_inner(cls_):
         _inherit_from_parent_class_inner(cls)
 
 
+def write(
+        func: Optional[
+            Callable[
+                [
+                    'DocumentArray',
+                    Dict,
+                    'DocumentArray',
+                    List['DocumentArray'],
+                    List['DocumentArray'],
+                ],
+                Optional[Union['DocumentArray', Dict]],
+            ]
+        ] = None
+):
+    """
+    `@write` is a decorator indicating that the function decorated will change the Executor finite state machine
+
+    Calls to methods decorated with `write` will be handled by `RAFT` consensus algorithm to guarantee the consensus of the Executor between replicas when used as a `StatefulDeployment`
+
+    EXAMPLE USAGE
+
+    .. code-block:: python
+
+        from jina import Executor, requests, Flow, write
+        from docarray import Document
+
+
+        # define Executor with custom `@requests` endpoints
+        class MyExecutor(Executor):
+
+            @requests(on='/index')
+            @write
+            def index(self, docs, **kwargs):
+                print(docs)  # index docs here
+
+            @requests(on=['/search', '/query'])
+            def search(self, docs, **kwargs):
+                print(docs)  # perform search here
+
+            @requests  # default/fallback endpoint
+            def foo(self, docs, **kwargs):
+                print(docs)  # process docs here
+
+
+        f = Flow().add(uses=MyExecutor, stateful=True, replicas=3)  # add your Executor to a Flow
+        with f:
+            f.post(
+                on='/index', inputs=Document(text='I am here!')
+            )  # send doc to `index` method which will be replicated using RAFT
+            f.post(
+                on='/search', inputs=Document(text='Who is there?')
+            )  # send doc to `search` method, that will bypass the RAFT apply
+            f.post(
+                on='/query', inputs=Document(text='Who is there?')
+            )  # send doc to `search` method
+            f.post(on='/bar', inputs=Document(text='Who is there?'))  # send doc to `foo` method
+
+    :param func: the method to decorate
+    :return: decorated function
+    """
+
+    class WriteMethodDecorator:
+        def __init__(self, fn):
+            self._requests_decorator = None
+            fn = self._unwrap_requests_decorator(fn)
+            if iscoroutinefunction(fn):
+
+                @functools.wraps(fn)
+                async def arg_wrapper(
+                        executor_instance, *args, **kwargs
+                ):  # we need to get the summary from the executor, so we need to access the self
+                    with executor_instance._write_lock:
+                        return await fn(executor_instance, *args, **kwargs)
+
+                self.fn = arg_wrapper
+            else:
+
+                @functools.wraps(fn)
+                def arg_wrapper(
+                        executor_instance, *args, **kwargs
+                ):  # we need to get the summary from the executor, so we need to access the self
+                    with executor_instance._write_lock:
+                        return fn(executor_instance, *args, **kwargs)
+
+                self.fn = arg_wrapper
+
+        def _unwrap_requests_decorator(self, fn):
+            if type(fn).__name__ == 'FunctionMapper':
+                self._requests_decorator = fn
+                return fn.fn
+            else:
+                return fn
+
+        def _inject_owner_attrs(self, owner, name):
+            if not hasattr(owner, '_write_methods'):
+                owner._write_methods = []
+
+            owner._write_methods.append(self.fn.__name__)
+
+        def __set_name__(self, owner, name):
+            if self._requests_decorator:
+                self._requests_decorator._inject_owner_attrs(owner, name, None, None)
+            self._inject_owner_attrs(owner, name)
+
+            setattr(owner, name, self.fn)
+
+        def __call__(self, *args, **kwargs):
+            # this is needed to make this decorator work in combination with `@requests`
+            return self.fn(*args, **kwargs)
+
+    if func:
+        return WriteMethodDecorator(func)
+    else:
+        return WriteMethodDecorator
+
+
 def requests(
     func: Optional[
         Callable[
@@ -163,10 +279,12 @@ def foo(self, docs, **kwargs):
     class FunctionMapper:
         def __init__(self, fn):
             self._batching_decorator = None
+            self._write_decorator = None
             fn = self._unwrap_batching_decorator(fn)
+            fn = self._unwrap_write_decorator(fn)
             arg_spec = inspect.getfullargspec(fn)
             if not arg_spec.varkw and not __args_executor_func__.issubset(
-                arg_spec.args
+                    arg_spec.args
             ):
                 raise TypeError(
                     f'{fn} accepts only {arg_spec.args} which is fewer than expected, '
@@ -177,7 +295,7 @@ def __init__(self, fn):
 
                 @functools.wraps(fn)
                 async def arg_wrapper(
-                    executor_instance, *args, **kwargs
+                        executor_instance, *args, **kwargs
                 ):  # we need to get the summary from the executor, so we need to access the self
                     return await fn(executor_instance, *args, **kwargs)
 
@@ -186,7 +304,7 @@ async def arg_wrapper(
 
                 @functools.wraps(fn)
                 def arg_wrapper(
-                    executor_instance, *args, **kwargs
+                        executor_instance, *args, **kwargs
                 ):  # we need to get the summary from the executor, so we need to access the self
                     return fn(executor_instance, *args, **kwargs)
 
@@ -199,6 +317,13 @@ def _unwrap_batching_decorator(self, fn):
             else:
                 return fn
 
+        def _unwrap_write_decorator(self, fn):
+            if type(fn).__name__ == 'WriteMethodDecorator':
+                self._write_decorator = fn
+                return fn.fn
+            else:
+                return fn
+
         def _inject_owner_attrs(
             self, owner, name, request_schema_arg, response_schema_arg
         ):
@@ -238,6 +363,8 @@ def __set_name__(self, owner, name):
             _init_requests_by_class(owner)
             if self._batching_decorator:
                 self._batching_decorator._inject_owner_attrs(owner, name)
+            if self._write_decorator:
+                self._write_decorator._inject_owner_attrs(owner, name)
             self.fn.class_name = owner.__name__
             self._inject_owner_attrs(owner, name, request_schema, response_schema)
 
@@ -252,19 +379,19 @@ def __call__(self, *args, **kwargs):
 
 
 def dynamic_batching(
-    func: Callable[
-        [
-            'DocumentArray',
-            Dict,
-            'DocumentArray',
-            List['DocumentArray'],
-            List['DocumentArray'],
-        ],
-        Optional[Union['DocumentArray', Dict]],
-    ] = None,
-    *,
-    preferred_batch_size: Optional[int] = None,
-    timeout: Optional[float] = 10_000,
+        func: Callable[
+            [
+                'DocumentArray',
+                Dict,
+                'DocumentArray',
+                List['DocumentArray'],
+                List['DocumentArray'],
+            ],
+            Optional[Union['DocumentArray', Dict]],
+        ] = None,
+        *,
+        preferred_batch_size: Optional[int] = None,
+        timeout: Optional[float] = 10_000,
 ):
     """
     `@dynamic_batching` defines the dynamic batching behavior of an Executor.
@@ -291,7 +418,7 @@ def __init__(self, fn):
 
                 @functools.wraps(fn)
                 async def arg_wrapper(
-                    executor_instance, *args, **kwargs
+                        executor_instance, *args, **kwargs
                 ):  # we need to get the summary from the executor, so we need to access the self
                     return await fn(executor_instance, *args, **kwargs)
 
@@ -300,7 +427,7 @@ async def arg_wrapper(
 
                 @functools.wraps(fn)
                 def arg_wrapper(
-                    executor_instance, *args, **kwargs
+                        executor_instance, *args, **kwargs
                 ):  # we need to get the summary from the executor, so we need to access the self
                     return fn(executor_instance, *args, **kwargs)
 
@@ -345,9 +472,9 @@ def __call__(self, *args, **kwargs):
 
 
 def monitor(
-    *,
-    name: Optional[str] = None,
-    documentation: Optional[str] = None,
+        *,
+        name: Optional[str] = None,
+        documentation: Optional[str] = None,
 ):
     """
     Decorator and context manager that allows monitoring of an Executor.
diff --git a/jina/serve/executors/run.py b/jina/serve/executors/run.py
new file mode 100644
index 0000000000000..356b73170d4fd
--- /dev/null
+++ b/jina/serve/executors/run.py
@@ -0,0 +1,230 @@
+import argparse
+import multiprocessing
+import os
+import copy
+from typing import Dict, Optional, Type, Union, TYPE_CHECKING
+
+from jina.logging.logger import JinaLogger
+from jina.serve.runtimes.asyncio import AsyncNewLoopRuntime
+from jina.serve.helper import _get_workspace_from_name_and_shards
+from jina.constants import RAFT_TO_EXECUTOR_PORT
+
+if TYPE_CHECKING:
+    import threading
+
+
+def run_raft(
+        args: 'argparse.Namespace',
+        is_ready: Union['multiprocessing.Event', 'threading.Event'],
+):
+    """Method to run the RAFT
+
+    This method is the target for the Pod's `thread` or `process`
+
+
+    :param args: namespace args from the Pod
+    :param is_ready: concurrency event to communicate Executor runtime is ready to receive messages
+    """
+
+    import jraft
+
+    def pascal_case_dict(d):
+        new_d = {}
+        for key, value in d.items():
+            new_key = key
+            if '_' in key:
+                new_key = ''.join(word.capitalize() for word in key.split('_'))
+            new_d[new_key] = value
+        return new_d
+
+    raft_id = str(args.replica_id)
+    shard_id = args.shard_id if args.shards > 1 else -1
+
+    raft_dir = _get_workspace_from_name_and_shards(
+        workspace=args.workspace, name='raft', shard_id=shard_id
+    )
+
+    port = args.port[0] if isinstance(args.port, list) else args.port
+    address = f'{args.host}:{port}'
+    executor_target = f'{args.host}:{port + RAFT_TO_EXECUTOR_PORT}'
+
+    # if the Executor was already persisted, retrieve its port and host configuration
+    logger = JinaLogger('run_raft', **vars(args))
+    persisted_address = jraft.get_configuration(raft_id, raft_dir)
+    if persisted_address:
+        logger.debug(f'Configuration found on the node: Address {persisted_address}')
+        address = persisted_address
+        executor_host, port = persisted_address.split(':')
+        executor_target = f'{executor_host}:{int(port) + 1}'
+
+    raft_configuration = pascal_case_dict(args.raft_configuration or {})
+    log_level = raft_configuration.get('LogLevel', os.getenv('JINA_LOG_LEVEL', 'INFO'))
+    raft_configuration['LogLevel'] = log_level
+    is_ready.wait()
+    logger.debug(f'Will run the RAFT node with RAFT configuration {raft_configuration}')
+    jraft.run(
+        address,
+        raft_id,
+        raft_dir,
+        executor_target,
+        **raft_configuration,
+    )
+
+
+def run(
+        args: 'argparse.Namespace',
+        name: str,
+        runtime_cls: Type[AsyncNewLoopRuntime],
+        envs: Dict[str, str],
+        is_started: Union['multiprocessing.Event', 'threading.Event'],
+        is_shutdown: Union['multiprocessing.Event', 'threading.Event'],
+        is_ready: Union['multiprocessing.Event', 'threading.Event'],
+        jaml_classes: Optional[Dict] = None,
+):
+    """Method representing the :class:`BaseRuntime` activity.
+
+    This method is the target for the Pod's `thread` or `process`
+
+    .. note::
+        :meth:`run` is running in subprocess/thread, the exception can not be propagated to the main process.
+        Hence, please do not raise any exception here.
+
+    .. note::
+        Please note that env variables are process-specific. Subprocess inherits envs from
+        the main process. But Subprocess's envs do NOT affect the main process. It does NOT
+        mess up user local system envs.
+
+    .. warning::
+        If you are using ``thread`` as backend, envs setting will likely be overidden by others
+
+    .. note::
+        `jaml_classes` contains all the :class:`JAMLCompatible` classes registered in the main process.
+        When using `spawn` as the multiprocessing start method, passing this argument to `run` method re-imports
+        & re-registers all `JAMLCompatible` classes.
+
+    :param args: namespace args from the Pod
+    :param name: name of the Pod to have proper logging
+    :param runtime_cls: the runtime class to instantiate
+    :param envs: a dictionary of environment variables to be set in the new Process
+    :param is_started: concurrency event to communicate runtime is properly started. Used for better logging
+    :param is_shutdown: concurrency event to communicate runtime is terminated
+    :param is_ready: concurrency event to communicate runtime is ready to receive messages
+    :param jaml_classes: all the `JAMLCompatible` classes imported in main process
+    """
+    req_handler_cls = None
+    if runtime_cls == 'GatewayRuntime':
+        from jina.serve.runtimes.gateway.request_handling import GatewayRequestHandler
+        req_handler_cls = GatewayRequestHandler
+    elif runtime_cls == 'WorkerRuntime':
+        from jina.serve.runtimes.worker.request_handling import WorkerRequestHandler
+        req_handler_cls = WorkerRequestHandler
+    elif runtime_cls == 'HeadRuntime':
+        from jina.serve.runtimes.head.request_handling import HeaderRequestHandler
+        req_handler_cls = HeaderRequestHandler
+
+    logger = JinaLogger(name, **vars(args))
+
+    def _unset_envs():
+        if envs:
+            for k in envs.keys():
+                os.environ.pop(k, None)
+
+    def _set_envs():
+        if args.env:
+            os.environ.update({k: str(v) for k, v in envs.items()})
+
+    try:
+        _set_envs()
+
+        runtime = AsyncNewLoopRuntime(
+            args=args,
+            req_handler_cls=req_handler_cls,
+            gateway_load_balancer=getattr(args, 'gateway_load_balancer', False)
+        )
+    except Exception as ex:
+        logger.error(
+            f'{ex!r} during {runtime_cls!r} initialization'
+            + f'\n add "--quiet-error" to suppress the exception details'
+            if not args.quiet_error
+            else '',
+            exc_info=not args.quiet_error,
+        )
+    else:
+        if not is_shutdown.is_set():
+            is_started.set()
+            with runtime:
+                # here the ready event is being set
+                is_ready.set()
+                runtime.run_forever()
+    finally:
+        _unset_envs()
+        is_shutdown.set()
+        logger.debug(f'process terminated')
+
+
+def run_stateful(args: 'argparse.Namespace',
+                 name: str,
+                 runtime_cls: Type[AsyncNewLoopRuntime],
+                 envs: Dict[str, str]):
+    """
+    Method to be called in Docker containers when Stateful Executor is required. This will start
+    2 processes in the Docker container.
+    """
+    import signal
+    from jina.jaml import JAML
+    is_ready = multiprocessing.Event()
+    is_shutdown = multiprocessing.Event()
+    is_started = multiprocessing.Event()
+    raft_worker = multiprocessing.Process(
+        target=run_raft,
+        kwargs={
+            'args': args,
+            'is_ready': is_ready,
+        },
+        name=name,
+        daemon=True,
+    )
+    cargs = copy.deepcopy(args)
+
+    if isinstance(cargs.port, int):
+        cargs.port += 1
+    elif isinstance(cargs.port, list):
+        cargs.port = [port + 1 for port in cargs.port]
+    worker = multiprocessing.Process(
+        target=run,
+        kwargs={
+            'args': cargs,
+            'name': name,
+            'envs': envs,
+            'is_started': is_started,
+            'is_shutdown': is_shutdown,
+            'is_ready': is_ready,
+            'runtime_cls': runtime_cls,
+            'jaml_classes': JAML.registered_classes(),
+        },
+        name=name,
+        daemon=False,
+    )
+
+    try:
+        HANDLED_SIGNALS = (
+            signal.SIGINT,  # Unix signal 2. Sent by Ctrl+C.
+            signal.SIGTERM,  # Unix signal 15. Sent by `kill <pid>`.
+            signal.SIGSEGV,
+        )
+
+        def signal_handler(*args, **kwargs):
+            worker.terminate()
+            raft_worker.terminate()
+
+        for sig in HANDLED_SIGNALS:
+            signal.signal(sig, signal_handler)
+
+        worker.start()
+        raft_worker.start()
+
+        worker.join()
+        raft_worker.join()
+    except:
+        worker.terminate()
+        raft_worker.terminate()
diff --git a/jina/serve/helper.py b/jina/serve/helper.py
index e2bd575c1b412..96688ac2a8dc3 100644
--- a/jina/serve/helper.py
+++ b/jina/serve/helper.py
@@ -1,5 +1,6 @@
 import functools
 import inspect
+import os
 import typing
 from typing import Any, Dict, List, Optional, Tuple, Union
 
@@ -76,10 +77,10 @@ def arg_wrapper(self, *args, **kwargs):
 
 
 def extract_trailing_metadata(error: grpc.aio.AioRpcError) -> Optional[str]:
-    '''Return formatted string of the trailing metadata if exists otherwise return None
+    """Return formatted string of the trailing metadata if exists otherwise return None
     :param error: AioRpcError
     :return: string of Metadata or None
-    '''
+    """
     if type(error) == grpc.aio.AioRpcError:
         trailing_metadata = error.trailing_metadata()
         if trailing_metadata and len(trailing_metadata):
@@ -89,10 +90,10 @@ def extract_trailing_metadata(error: grpc.aio.AioRpcError) -> Optional[str]:
 
 
 def format_grpc_error(error: grpc.aio.AioRpcError) -> str:
-    '''Adds grpc context trainling metadata if available
+    """Adds grpc context trainling metadata if available
     :param error: AioRpcError
     :return: formatted error
-    '''
+    """
     default_string = str(error)
     trailing_metadata = extract_trailing_metadata(error)
     if trailing_metadata:
@@ -101,6 +102,16 @@ def format_grpc_error(error: grpc.aio.AioRpcError) -> str:
     return default_string
 
 
+def _get_workspace_from_name_and_shards(workspace, name, shard_id):
+    if workspace:
+        complete_workspace = os.path.join(workspace, name)
+        if shard_id is not None and shard_id != -1:
+            complete_workspace = os.path.join(complete_workspace, str(shard_id))
+        if not os.path.exists(complete_workspace):
+            os.makedirs(complete_workspace, exist_ok=True)
+        return os.path.abspath(complete_workspace)
+
+
 def get_default_grpc_options() -> List[Tuple[str, Any]]:
     """
     Returns a list of default options used for creating grpc channels.
diff --git a/jina/serve/networking/__init__.py b/jina/serve/networking/__init__.py
index 203f6327767f0..13b1ddf6dc375 100644
--- a/jina/serve/networking/__init__.py
+++ b/jina/serve/networking/__init__.py
@@ -50,15 +50,15 @@ class GrpcConnectionPool:
     K8S_PORT_MONITORING = 9090
 
     def __init__(
-        self,
-        runtime_name,
-        logger: Optional[JinaLogger] = None,
-        compression: Optional[str] = None,
-        metrics_registry: Optional['CollectorRegistry'] = None,
-        meter: Optional['Meter'] = None,
-        aio_tracing_client_interceptors: Optional[Sequence['ClientInterceptor']] = None,
-        tracing_client_interceptor: Optional['OpenTelemetryClientInterceptor'] = None,
-        channel_options: Optional[list] = None,
+            self,
+            runtime_name,
+            logger: Optional[JinaLogger] = None,
+            compression: Optional[str] = None,
+            metrics_registry: Optional['CollectorRegistry'] = None,
+            meter: Optional['Meter'] = None,
+            aio_tracing_client_interceptors: Optional[Sequence['ClientInterceptor']] = None,
+            tracing_client_interceptor: Optional['OpenTelemetryClientInterceptor'] = None,
+            channel_options: Optional[list] = None,
     ):
         self._logger = logger or JinaLogger(self.__class__.__name__)
         self.channel_options = channel_options
@@ -71,8 +71,8 @@ def __init__(
 
         if metrics_registry:
             with ImportExtensions(
-                required=True,
-                help_text='You need to install the `prometheus_client` to use the montitoring functionality of jina',
+                    required=True,
+                    help_text='You need to install the `prometheus_client` to use the montitoring functionality of jina',
             ):
                 from prometheus_client import Summary
 
@@ -146,16 +146,16 @@ def __init__(
         self._deployment_address_map = {}
 
     def send_requests(
-        self,
-        requests: List[Request],
-        deployment: str,
-        head: bool = False,
-        shard_id: Optional[int] = None,
-        polling_type: PollingType = PollingType.ANY,
-        endpoint: Optional[str] = None,
-        metadata: Optional[Dict[str, str]] = None,
-        timeout: Optional[float] = None,
-        retries: Optional[int] = -1,
+            self,
+            requests: List[Request],
+            deployment: str,
+            head: bool = False,
+            shard_id: Optional[int] = None,
+            polling_type: PollingType = PollingType.ANY,
+            endpoint: Optional[str] = None,
+            metadata: Optional[Dict[str, str]] = None,
+            timeout: Optional[float] = None,
+            retries: Optional[int] = -1,
     ) -> List[asyncio.Task]:
         """Send a request to target via one or all of the pooled connections, depending on polling_type
 
@@ -197,12 +197,12 @@ def send_requests(
         return results
 
     def send_discover_endpoint(
-        self,
-        deployment: str,
-        head: bool = True,
-        shard_id: Optional[int] = None,
-        timeout: Optional[float] = None,
-        retries: Optional[int] = -1,
+            self,
+            deployment: str,
+            head: bool = True,
+            shard_id: Optional[int] = None,
+            timeout: Optional[float] = None,
+            retries: Optional[int] = -1,
     ) -> Optional[asyncio.Task]:
         """Sends a discover Endpoint call to target.
 
@@ -227,15 +227,15 @@ def send_discover_endpoint(
             return None
 
     def send_requests_once(
-        self,
-        requests: List[Request],
-        deployment: str,
-        metadata: Optional[Dict[str, str]] = None,
-        head: bool = False,
-        shard_id: Optional[int] = None,
-        endpoint: Optional[str] = None,
-        timeout: Optional[float] = None,
-        retries: Optional[int] = -1,
+            self,
+            requests: List[Request],
+            deployment: str,
+            metadata: Optional[Dict[str, str]] = None,
+            head: bool = False,
+            shard_id: Optional[int] = None,
+            endpoint: Optional[str] = None,
+            timeout: Optional[float] = None,
+            retries: Optional[int] = -1,
     ) -> Optional[asyncio.Task]:
         """Send a request to target via only one of the pooled connections
 
@@ -267,11 +267,11 @@ def send_requests_once(
             return None
 
     def add_connection(
-        self,
-        deployment: str,
-        address: str,
-        head: bool = False,
-        shard_id: Optional[int] = None,
+            self,
+            deployment: str,
+            address: str,
+            head: bool = False,
+            shard_id: Optional[int] = None,
     ):
         """
         Adds a connection for a deployment to this connection pool
@@ -290,11 +290,11 @@ def add_connection(
         self._deployment_address_map[deployment] = address
 
     async def remove_connection(
-        self,
-        deployment: str,
-        address: str,
-        head: bool = False,
-        shard_id: Optional[int] = None,
+            self,
+            deployment: str,
+            address: str,
+            head: bool = False,
+            shard_id: Optional[int] = None,
     ):
         """
         Removes a connection to a deployment
@@ -319,17 +319,17 @@ async def close(self):
         await self._connections.close()
 
     async def _handle_aiorpcerror(
-        self,
-        error: AioRpcError,
-        retry_i: int = 0,
-        request_id: str = '',
-        tried_addresses: Set[str] = {
-            ''
-        },  # same deployment can have multiple addresses (replicas)
-        total_num_tries: int = 1,  # number of retries + 1
-        current_address: str = '',  # the specific address that was contacted during this attempt
-        current_deployment: str = '',  # the specific deployment that was contacted during this attempt
-        connection_list: Optional[_ReplicaList] = None,
+            self,
+            error: AioRpcError,
+            retry_i: int = 0,
+            request_id: str = '',
+            tried_addresses: Set[str] = {
+                ''
+            },  # same deployment can have multiple addresses (replicas)
+            total_num_tries: int = 1,  # number of retries + 1
+            current_address: str = '',  # the specific address that was contacted during this attempt
+            current_deployment: str = '',  # the specific deployment that was contacted during this attempt
+            connection_list: Optional[_ReplicaList] = None,
     ) -> 'Optional[Union[AioRpcError, InternalNetworkError]]':
         # connection failures, cancelled requests, and timed out requests should be retried
         # all other cases should not be retried and will be raised immediately
@@ -339,9 +339,12 @@ async def _handle_aiorpcerror(
         # if an Executor is down behind an API gateway, grpc.StatusCode.NOT_FOUND is returned
         # requests usually gets cancelled when the server shuts down
         # retries for cancelled requests will hit another replica in K8s
-        self._logger.debug(
-            f'GRPC call to {current_deployment} errored, with error {format_grpc_error(error)} and for the {retry_i + 1}th time.'
-        )
+        if error.code() == grpc.StatusCode.UNAVAILABLE and 'not the leader' in error.details():
+            self._logger.debug(f'RAFT node of {current_deployment} is not the leader. Trying next replica, if available.')
+        else:
+            self._logger.debug(
+                f'gRPC call to {current_deployment} errored, with error {format_grpc_error(error)} and for the {retry_i + 1}th time.'
+            )
         errors_to_retry = [
             grpc.StatusCode.UNAVAILABLE,
             grpc.StatusCode.DEADLINE_EXCEEDED,
@@ -357,7 +360,7 @@ async def _handle_aiorpcerror(
             return error
         elif error.code() in errors_to_retry and retry_i >= total_num_tries - 1:
             self._logger.debug(
-                f'GRPC call for {current_deployment} failed, retries exhausted'
+                f'gRPC call for {current_deployment} failed, retries exhausted'
             )
             from jina.excepts import InternalNetworkError
 
@@ -375,20 +378,23 @@ async def _handle_aiorpcerror(
                 details=error.details(),
             )
         else:
-            self._logger.debug(
-                f'GRPC call to deployment {current_deployment} failed with error {format_grpc_error(error)}, for retry attempt {retry_i + 1}/{total_num_tries - 1}.'
-                f' Trying next replica, if available.'
-            )
+            if error.code() == grpc.StatusCode.UNAVAILABLE and 'not the leader' in error.details():
+                self._logger.debug(f'RAFT node of {current_deployment} is not the leader. Trying next replica, if available.')
+            else:
+                self._logger.debug(
+                    f'gRPC call to deployment {current_deployment} failed with error {format_grpc_error(error)}, for retry attempt {retry_i + 1}/{total_num_tries - 1}.'
+                    f' Trying next replica, if available.'
+                )
             return None
 
     def _send_requests(
-        self,
-        requests: List[Request],
-        connections: _ReplicaList,
-        endpoint: Optional[str] = None,
-        metadata: Optional[Dict[str, str]] = None,
-        timeout: Optional[float] = None,
-        retries: Optional[int] = -1,
+            self,
+            requests: List[Request],
+            connections: _ReplicaList,
+            endpoint: Optional[str] = None,
+            metadata: Optional[Dict[str, str]] = None,
+            timeout: Optional[float] = None,
+            retries: Optional[int] = -1,
     ) -> 'asyncio.Task[Union[Tuple, AioRpcError, InternalNetworkError]]':
         # this wraps the awaitable object from grpc as a coroutine so it can be used as a task
         # the grpc call function is not a coroutine but some _AioCall
@@ -402,17 +408,27 @@ def _send_requests(
 
         async def task_wrapper():
             tried_addresses = set()
+            num_replicas = len(connections.get_all_connections())
             if retries is None or retries < 0:
                 total_num_tries = (
-                    max(DEFAULT_MINIMUM_RETRIES, len(connections.get_all_connections()))
-                    + 1
+                        max(DEFAULT_MINIMUM_RETRIES, len(connections.get_all_connections()))
+                        + 1
                 )
             else:
                 total_num_tries = 1 + retries  # try once, then do all the retries
             for i in range(total_num_tries):
-                current_connection = await connections.get_next_connection(
-                    num_retries=total_num_tries
-                )
+                current_connection = None
+                while (
+                        current_connection is None
+                        or current_connection.address in tried_addresses
+                ):
+                    current_connection = await connections.get_next_connection(
+                        num_retries=total_num_tries
+                    )
+                    # if you request to retry more than the amount of replicas, we just skip, we could balance the
+                    # retries in the future
+                    if len(tried_addresses) >= num_replicas:
+                        break
                 tried_addresses.add(current_connection.address)
                 try:
                     return await current_connection.send_requests(
@@ -440,10 +456,10 @@ async def task_wrapper():
         return asyncio.create_task(task_wrapper())
 
     def _send_discover_endpoint(
-        self,
-        connection_list: _ReplicaList,
-        timeout: Optional[float] = None,
-        retries: Optional[int] = -1,
+            self,
+            connection_list: _ReplicaList,
+            timeout: Optional[float] = None,
+            retries: Optional[int] = -1,
     ) -> asyncio.Task:
         # this wraps the awaitable object from grpc as a coroutine so it can be used as a task
         # the grpc call function is not a coroutine but some _AioCall
@@ -452,11 +468,11 @@ async def task_wrapper():
             tried_addresses = set()
             if retries is None or retries < 0:
                 total_num_tries = (
-                    max(
-                        DEFAULT_MINIMUM_RETRIES,
-                        len(connection_list.get_all_connections()),
-                    )
-                    + 1
+                        max(
+                            DEFAULT_MINIMUM_RETRIES,
+                            len(connection_list.get_all_connections()),
+                        )
+                        + 1
                 )
             else:
                 total_num_tries = 1 + retries  # try once, then do all the retries
@@ -487,9 +503,9 @@ async def task_wrapper():
         return asyncio.create_task(task_wrapper())
 
     async def warmup(
-        self,
-        deployment: str,
-        stop_event: 'threading.Event',
+            self,
+            deployment: str,
+            stop_event: 'threading.Event',
     ):
         """Executes JinaInfoRPC against the provided deployment. A single task is created for each replica connection.
         :param deployment: deployment name and the replicas that needs to be warmed up.
diff --git a/jina/serve/networking/connection_stub.py b/jina/serve/networking/connection_stub.py
index 0bdc7080f997b..b27cd9fe8db37 100644
--- a/jina/serve/networking/connection_stub.py
+++ b/jina/serve/networking/connection_stub.py
@@ -57,7 +57,8 @@ async def _init_stubs(self):
         available_services = await get_available_services(self.channel)
         stubs = defaultdict(lambda: None)
         for service in available_services:
-            stubs[service] = self.STUB_MAPPING[service](self.channel)
+            if service in self.STUB_MAPPING:
+                stubs[service] = self.STUB_MAPPING[service](self.channel)
         self.data_list_stub = stubs['jina.JinaDataRequestRPC']
         self.single_data_stub = stubs['jina.JinaSingleDataRequestRPC']
         self.stream_stub = stubs['jina.JinaRPC']
diff --git a/jina/serve/networking/utils.py b/jina/serve/networking/utils.py
index 5490cb7665211..f408938125559 100644
--- a/jina/serve/networking/utils.py
+++ b/jina/serve/networking/utils.py
@@ -144,10 +144,12 @@ async def send_health_check_async(
 ) -> health_pb2.HealthCheckResponse:
     """
     Sends a request asynchronously to the target via grpc
+
     :param target: where to send the request to, like 126.0.0.1:8080
     :param timeout: timeout for the send
     :param tls: if True, use tls encryption for the grpc channel
     :param root_certificates: the path to the root certificates for tls, only used if tls is True
+
     :returns: the response health check
     """
 
diff --git a/jina/serve/runtimes/asyncio.py b/jina/serve/runtimes/asyncio.py
index 2e9d34200ac97..1fb4aaa3a724c 100644
--- a/jina/serve/runtimes/asyncio.py
+++ b/jina/serve/runtimes/asyncio.py
@@ -11,6 +11,7 @@
 from jina.logging.logger import JinaLogger
 from jina.parsers import set_gateway_parser
 from jina.parsers.helper import _set_gateway_uses
+from jina.serve.networking.utils import send_health_check_async
 
 # Keep these imports even if not used, since YAML parser needs to find them in imported modules
 from jina.serve.runtimes.gateway.composite import CompositeGateway
@@ -253,7 +254,6 @@ async def async_setup(self):
             raise PortAlreadyUsed(f'port:{self.args.port}')
 
         self.server = self._get_server()
-
         await self.server.setup_server()
 
     async def async_teardown(self):
diff --git a/jina/serve/runtimes/head/request_handling.py b/jina/serve/runtimes/head/request_handling.py
index 1baabc7b7a28e..ad92a44821940 100644
--- a/jina/serve/runtimes/head/request_handling.py
+++ b/jina/serve/runtimes/head/request_handling.py
@@ -16,6 +16,7 @@
 from jina.serve.runtimes.monitoring import MonitoringRequestMixin
 from jina.serve.runtimes.worker.request_handling import WorkerRequestHandler
 from jina.types.request.data import DataRequest, Response
+from jina._docarray import docarray_v2
 
 if TYPE_CHECKING:  # pragma: no cover
     from prometheus_client import CollectorRegistry
@@ -247,7 +248,6 @@ async def _handle_data_request(
             polling_type=polling_type,
             retries=retries,
         )
-
         if len(worker_results) == 0:
             if exceptions:
                 # raise the underlying error first
@@ -260,27 +260,34 @@ async def _handle_data_request(
         worker_results, metadata = zip(*worker_results)
 
         response_request = worker_results[0]
+        for i, worker_result in enumerate(worker_results):
+            if worker_result.header.status.code == jina_pb2.StatusProto.SUCCESS:
+                response_request = worker_result
+                break
+
         uses_after_metadata = None
-        if uses_after_address:
-            result = await connection_pool.send_requests_once(
-                worker_results,
-                deployment='uses_after',
-                timeout=timeout_send,
-                retries=retries,
-            )
-            if issubclass(type(result), BaseException):
-                self._update_end_failed_requests_metrics()
-                raise result
-            else:
-                response_request, uses_after_metadata = result
-        elif len(worker_results) > 1 and reduce:
-            response_request = WorkerRequestHandler.reduce_requests(worker_results)
-        elif len(worker_results) > 1 and not reduce:
-            # worker returned multiple responses, but the head is configured to skip reduction
-            # just concatenate the docs in this case
-            response_request.data.docs = WorkerRequestHandler.get_docs_from_request(
-                requests
-            )
+        if not docarray_v2:
+            # for docaaray v2, we get the first successfull
+            if uses_after_address:
+                result = await connection_pool.send_requests_once(
+                    worker_results,
+                    deployment='uses_after',
+                    timeout=timeout_send,
+                    retries=retries,
+                )
+                if issubclass(type(result), BaseException):
+                    self._update_end_failed_requests_metrics()
+                    raise result
+                else:
+                    response_request, uses_after_metadata = result
+            elif len(worker_results) > 1 and reduce:
+                response_request = WorkerRequestHandler.reduce_requests(worker_results)
+            elif len(worker_results) > 1 and not reduce:
+                # worker returned multiple responses, but the head is configured to skip reduction
+                # just concatenate the docs in this case
+                response_request.data.docs = WorkerRequestHandler.get_docs_from_request(
+                    requests
+                )
 
         merged_metadata = self._merge_metadata(
             metadata,
diff --git a/jina/serve/runtimes/servers/grpc.py b/jina/serve/runtimes/servers/grpc.py
index 2bb998695c112..03ac4946e0e41 100644
--- a/jina/serve/runtimes/servers/grpc.py
+++ b/jina/serve/runtimes/servers/grpc.py
@@ -63,9 +63,26 @@ async def setup_server(self):
             jina_pb2_grpc.add_JinaGatewayDryRunRPCServicer_to_server(
                 self._request_handler, self.server
             )
+        if hasattr(self._request_handler, 'snapshot'):
+            jina_pb2_grpc.add_JinaExecutorSnapshotServicer_to_server(
+                self._request_handler, self.server
+            )
+        if hasattr(self._request_handler, 'snapshot_status'):
+            jina_pb2_grpc.add_JinaExecutorSnapshotProgressServicer_to_server(
+                self._request_handler, self.server
+            )
+        if hasattr(self._request_handler, 'restore'):
+            jina_pb2_grpc.add_JinaExecutorRestoreServicer_to_server(
+                self._request_handler, self.server
+            )
+        if hasattr(self._request_handler, 'restore_status'):
+            jina_pb2_grpc.add_JinaExecutorRestoreProgressServicer_to_server(
+                self._request_handler, self.server
+            )
+
         jina_pb2_grpc.add_JinaInfoRPCServicer_to_server(
-            self._request_handler, self.server
-        )
+                self._request_handler, self.server
+            )
 
         service_names = (
             jina_pb2.DESCRIPTOR.services_by_name['JinaRPC'].full_name,
diff --git a/jina/serve/runtimes/worker/request_handling.py b/jina/serve/runtimes/worker/request_handling.py
index e65d0b97d6ef1..9ee4f529c472c 100644
--- a/jina/serve/runtimes/worker/request_handling.py
+++ b/jina/serve/runtimes/worker/request_handling.py
@@ -1,25 +1,27 @@
 import functools
 import json
 import os
-from typing import Dict, Tuple
+import argparse
+import asyncio
+import grpc
+import uuid
+import tempfile
+
+from typing import TYPE_CHECKING, AsyncIterator, List, Optional, Dict, Tuple
+import threading
 
 from jina._docarray import DocumentArray
 from jina.constants import __default_endpoint__
 from jina.excepts import BadConfigSource
 from jina.serve.executors import BaseExecutor
 from jina.serve.runtimes.worker.batch_queue import BatchQueue
-import argparse
-import asyncio
-from typing import TYPE_CHECKING, AsyncIterator, List, Optional
-
-import grpc
-
 from jina.excepts import RuntimeTerminated
 from jina.helper import get_full_version
 from jina.importer import ImportExtensions
 from jina.proto import jina_pb2
 from jina.serve.instrumentation import MetricsTimer
 from jina.types.request.data import DataRequest
+from jina._docarray import docarray_v2
 
 if TYPE_CHECKING:  # pragma: no cover
     from opentelemetry.propagate import Context
@@ -28,9 +30,11 @@
 
     from opentelemetry import metrics, trace
     from opentelemetry.context.context import Context
+    from opentelemetry.propagate import Context
     from prometheus_client import CollectorRegistry
 
     from jina.logging.logger import JinaLogger
+    from jina.types.request import Request
 
 
 class WorkerRequestHandler:
@@ -142,6 +146,13 @@ def __init__(
         # the below is of "shape" exec_endpoint_name -> parameters_key -> batch_queue
         self._batchqueue_instances: Dict[str, Dict[str, BatchQueue]] = {}
         self._init_batchqueue_dict()
+        self._snapshot = None
+        self._did_snapshot_raise_exception = None
+        self._restore = None
+        self._did_restore_raise_exception = None
+        self._snapshot_thread = None
+        self._restore_thread = None
+        self._snapshot_parent_directory = tempfile.mkdtemp()
         self._hot_reload_task = None
         if self.args.reload:
             self._hot_reload_task = asyncio.create_task(self._hot_reload())
@@ -707,8 +718,12 @@ def reduce(docs_matrix: List['DocumentArray']) -> Optional['DocumentArray']:
         :return: the resulting DocumentArray
         """
         if docs_matrix:
-            da = docs_matrix[0]
-            da.reduce_all(docs_matrix[1:])
+            if not docarray_v2:
+                da = docs_matrix[0]
+                da.reduce_all(docs_matrix[1:])
+            else:
+                from docarray.utils.reduce import reduce_all
+                da = reduce_all(docs_matrix)
             return da
 
     @staticmethod
@@ -724,16 +739,21 @@ def reduce_requests(requests: List['DataRequest']) -> 'DataRequest':
         :param requests: List of DataRequest objects
         :return: the resulting DataRequest
         """
+        req = requests[0]
+        for i, worker_result in enumerate(requests):
+            if worker_result.header.status.code == jina_pb2.StatusProto.SUCCESS:
+                req = worker_result
+                break
         docs_matrix, _ = WorkerRequestHandler._get_docs_matrix_from_request(requests)
 
         # Reduction is applied in-place to the first DocumentArray in the matrix
         da = WorkerRequestHandler.reduce(docs_matrix)
-        WorkerRequestHandler.replace_docs(requests[0], da)
+        WorkerRequestHandler.replace_docs(req, da)
 
         params = WorkerRequestHandler.get_parameters_dict_from_request(requests)
-        WorkerRequestHandler.replace_parameters(requests[0], params)
+        WorkerRequestHandler.replace_parameters(req, params)
 
-        return requests[0]
+        return req
 
     # serving part
     async def process_single_data(self, request: DataRequest, context) -> DataRequest:
@@ -759,7 +779,9 @@ async def endpoint_discovery(self, empty, context) -> jina_pb2.EndpointsProto:
         endpoints_proto.endpoints.extend(
             list(self._executor.requests.keys())
         )
-
+        endpoints_proto.write_endpoints.extend(
+            list(self._executor.write_endpoints)
+        )
         return endpoints_proto
 
     def _extract_tracing_context(
@@ -866,3 +888,156 @@ async def stream(
             yield await self.process_data([request], context)
 
     Call = stream
+
+    def _create_snapshot_status(
+            self,
+            snapshot_directory: str,
+    ) -> 'jina_pb2.SnapshotStatusProto':
+        _id = str(uuid.uuid4())
+        self.logger.debug(f'Generated snapshot id: {_id}')
+        return jina_pb2.SnapshotStatusProto(
+            id=jina_pb2.SnapshotId(value=_id),
+            status=jina_pb2.SnapshotStatusProto.Status.RUNNING,
+            snapshot_file=os.path.join(os.path.join(snapshot_directory, _id), 'state.bin'),
+        )
+
+    def _create_restore_status(
+            self,
+    ) -> 'jina_pb2.SnapshotStatusProto':
+        _id = str(uuid.uuid4())
+        self.logger.debug(f'Generated restore id: {_id}')
+        return jina_pb2.RestoreSnapshotStatusProto(
+            id=jina_pb2.RestoreId(value=_id),
+            status=jina_pb2.RestoreSnapshotStatusProto.Status.RUNNING,
+        )
+
+    async def snapshot(self, request, context) -> 'jina_pb2.SnapshotStatusProto':
+        """
+        method to start a snapshot process of the Executor
+        :param request: the empty request
+        :param context: grpc context
+
+        :return: the status of the snapshot
+        """
+        self.logger.debug(f' Calling snapshot')
+        if (
+                self._snapshot
+                and self._snapshot_thread
+                and self._snapshot_thread.is_alive()
+        ):
+            raise RuntimeError(
+                f'A snapshot with id {self._snapshot.id.value} is currently in progress. Cannot start another.'
+            )
+        else:
+            self._snapshot = self._create_snapshot_status(
+                self._snapshot_parent_directory,
+            )
+            self._did_snapshot_raise_exception = threading.Event()
+            self._snapshot_thread = threading.Thread(
+                target=self._executor._run_snapshot,
+                args=(self._snapshot.snapshot_file, self._did_snapshot_raise_exception),
+            )
+            self._snapshot_thread.start()
+            return self._snapshot
+
+    async def snapshot_status(
+            self, request: 'jina_pb2.SnapshotId', context
+    ) -> 'jina_pb2.SnapshotStatusProto':
+        """
+        method to start a snapshot process of the Executor
+        :param request: the snapshot Id to get the status from
+        :param context: grpc context
+
+        :return: the status of the snapshot
+        """
+        self.logger.debug(
+            f'Checking status of snapshot with ID of request {request.value} and current snapshot {self._snapshot.id.value if self._snapshot else "DOES NOT EXIST"}')
+        if not self._snapshot or (self._snapshot.id.value != request.value):
+            return jina_pb2.SnapshotStatusProto(
+                id=jina_pb2.SnapshotId(value=request.value),
+                status=jina_pb2.SnapshotStatusProto.Status.NOT_FOUND,
+            )
+        elif self._snapshot_thread and self._snapshot_thread.is_alive():
+            return jina_pb2.SnapshotStatusProto(
+                id=jina_pb2.SnapshotId(value=request.value),
+                status=jina_pb2.SnapshotStatusProto.Status.RUNNING,
+            )
+        elif self._snapshot_thread and not self._snapshot_thread.is_alive():
+            status = jina_pb2.SnapshotStatusProto.Status.SUCCEEDED
+            if self._did_snapshot_raise_exception.is_set():
+                status = jina_pb2.SnapshotStatusProto.Status.FAILED
+            self._did_snapshot_raise_exception = None
+            return jina_pb2.SnapshotStatusProto(
+                id=jina_pb2.SnapshotId(value=request.value),
+                status=status,
+            )
+
+        return jina_pb2.SnapshotStatusProto(
+            id=jina_pb2.SnapshotId(value=request.value),
+            status=jina_pb2.SnapshotStatusProto.Status.NOT_FOUND,
+        )
+
+    async def restore(self, request: 'jina_pb2.RestoreSnapshotCommand', context):
+        """
+        method to start a restore process of the Executor
+        :param request: the command request with the path from where to restore the Executor
+        :param context: grpc context
+
+        :return: the status of the snapshot
+        """
+        self.logger.debug(f' Calling restore')
+        if (
+                self._restore
+                and self._restore_thread
+                and self._restore_thread.is_alive()
+        ):
+            raise RuntimeError(
+                f'A restore with id {self._restore.id.value} is currently in progress. Cannot start another.'
+            )
+        else:
+            self._restore = self._create_restore_status()
+            self._did_restore_raise_exception = threading.Event()
+            self._restore_thread = threading.Thread(
+                target=self._executor._run_restore,
+                args=(request.snapshot_file, self._did_restore_raise_exception),
+            )
+            self._restore_thread.start()
+        return self._restore
+
+    async def restore_status(
+            self, request, context
+    ) -> 'jina_pb2.RestoreSnapshotStatusProto':
+        """
+        method to start a snapshot process of the Executor
+        :param request: the request with the Restore ID from which to get status
+        :param context: grpc context
+
+        :return: the status of the snapshot
+        """
+        self.logger.debug(
+            f'Checking status of restore with ID of request {request.value} and current restore {self._restore.id.value if self._restore else "DOES NOT EXIST"}')
+        if not self._restore or (self._restore.id.value != request.value):
+            return jina_pb2.RestoreSnapshotStatusProto(
+                id=jina_pb2.RestoreId(value=request.value),
+                status=jina_pb2.RestoreSnapshotStatusProto.Status.NOT_FOUND,
+            )
+        elif self._restore_thread and self._restore_thread.is_alive():
+            return jina_pb2.RestoreSnapshotStatusProto(
+                id=jina_pb2.RestoreId(value=request.value),
+                status=jina_pb2.RestoreSnapshotStatusProto.Status.RUNNING,
+            )
+        elif self._restore_thread and not self._restore_thread.is_alive():
+            status = jina_pb2.RestoreSnapshotStatusProto.Status.SUCCEEDED
+            if self._did_restore_raise_exception.is_set():
+                status = jina_pb2.RestoreSnapshotStatusProto.Status.FAILED
+            self._did_restore_raise_exception = None
+            return jina_pb2.RestoreSnapshotStatusProto(
+                id=jina_pb2.RestoreId(value=request.value),
+                status=status,
+            )
+
+        return jina_pb2.RestoreSnapshotStatusProto(
+            id=jina_pb2.RestoreId(value=request.value),
+            status=jina_pb2.RestoreSnapshotStatusProto.Status.NOT_FOUND,
+        )
+
diff --git a/jina_cli/api.py b/jina_cli/api.py
index c5e5d437070c6..65f6546c06cb6 100644
--- a/jina_cli/api.py
+++ b/jina_cli/api.py
@@ -43,26 +43,25 @@ def executor_native(args: 'Namespace'):
 
     :param args: arguments coming from the CLI.
     """
-    from jina.serve.runtimes.asyncio import AsyncNewLoopRuntime
-    if args.runtime_cls == 'WorkerRuntime':
-        from jina.serve.runtimes.worker.request_handling import WorkerRequestHandler
-        req_handler_cls = WorkerRequestHandler
-    elif args.runtime_cls == 'HeadRuntime':
-        from jina.serve.runtimes.head.request_handling import HeaderRequestHandler
-        req_handler_cls = HeaderRequestHandler
+    from jina.serve.executors.run import run, run_stateful
+    import multiprocessing
+    from jina.jaml import JAML
+    envs = {}
+    envs.update(args.env or {})
+    if not args.stateful:
+        run(name=args.name,
+            args=args,
+            runtime_cls=args.runtime_cls,
+            envs=envs,
+            is_started=multiprocessing.Event(),
+            is_shutdown=multiprocessing.Event(),
+            is_ready=multiprocessing.Event(),
+            jaml_classes=JAML.registered_classes())
     else:
-        raise RuntimeError(
-            f' runtime_cls {args.runtime_cls} is not supported with `--native` argument. `WorkerRuntime` is supported'
-        )
-
-    with AsyncNewLoopRuntime(args, req_handler_cls=req_handler_cls) as rt:
-        name = (
-            rt.server._request_handler._executor.metas.name
-            if hasattr(rt.server, '_request_handler') and hasattr(rt.server._request_handler, '_executor')
-            else args.runtime_cls
-        )
-        rt.logger.info(f'Executor {name} started')
-        rt.run_forever()
+        run_stateful(name=args.name,
+                     args=args,
+                     runtime_cls=args.runtime_cls,
+                     envs=envs)
 
 
 def executor(args: 'Namespace'):
@@ -178,7 +177,7 @@ def new(args: 'Namespace'):
     import shutil
 
     from jina.constants import __resources_path__
-    
+
     if args.type == 'deployment':
         shutil.copytree(
             os.path.join(__resources_path__, 'project-template', 'deployment'), os.path.abspath(args.name)
diff --git a/jina_cli/autocomplete.py b/jina_cli/autocomplete.py
index 26d095bd9a63c..ae1a913cdf611 100644
--- a/jina_cli/autocomplete.py
+++ b/jina_cli/autocomplete.py
@@ -45,6 +45,7 @@
             '--disable-reduce',
             '--allow-concurrent',
             '--grpc-server-options',
+            '--raft-configuration',
             '--grpc-channel-options',
             '--entrypoint',
             '--docker-kwargs',
@@ -62,6 +63,7 @@
             '--pod-role',
             '--noblock-on-start',
             '--floating',
+            '--replica-id',
             '--reload',
             '--install-requirements',
             '--port',
@@ -77,6 +79,8 @@
             '--metrics',
             '--metrics-exporter-host',
             '--metrics-exporter-port',
+            '--stateful',
+            '--peer-ports',
             '--force-update',
             '--force',
             '--prefer-platform',
@@ -164,6 +168,7 @@
             '--pod-role',
             '--noblock-on-start',
             '--floating',
+            '--replica-id',
             '--reload',
             '--port',
             '--ports',
@@ -180,6 +185,8 @@
             '--metrics',
             '--metrics-exporter-host',
             '--metrics-exporter-port',
+            '--stateful',
+            '--peer-ports',
         ],
         'auth login': ['--help', '--force'],
         'auth logout': ['--help'],
@@ -288,6 +295,7 @@
             '--disable-reduce',
             '--allow-concurrent',
             '--grpc-server-options',
+            '--raft-configuration',
             '--grpc-channel-options',
             '--entrypoint',
             '--docker-kwargs',
@@ -305,6 +313,7 @@
             '--pod-role',
             '--noblock-on-start',
             '--floating',
+            '--replica-id',
             '--reload',
             '--install-requirements',
             '--port',
@@ -320,6 +329,8 @@
             '--metrics',
             '--metrics-exporter-host',
             '--metrics-exporter-port',
+            '--stateful',
+            '--peer-ports',
             '--force-update',
             '--force',
             '--prefer-platform',
@@ -356,6 +367,7 @@
             '--disable-reduce',
             '--allow-concurrent',
             '--grpc-server-options',
+            '--raft-configuration',
             '--grpc-channel-options',
             '--entrypoint',
             '--docker-kwargs',
@@ -373,6 +385,7 @@
             '--pod-role',
             '--noblock-on-start',
             '--floating',
+            '--replica-id',
             '--reload',
             '--install-requirements',
             '--port',
@@ -388,6 +401,8 @@
             '--metrics',
             '--metrics-exporter-host',
             '--metrics-exporter-port',
+            '--stateful',
+            '--peer-ports',
             '--force-update',
             '--force',
             '--prefer-platform',
diff --git a/scripts/update-version.sh b/scripts/update-version.sh
new file mode 100755
index 0000000000000..8b22af2c0f898
--- /dev/null
+++ b/scripts/update-version.sh
@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+
+# Requirements
+# brew install hub
+# npm install -g git-release-notes
+# pip install twine wheel
+
+set -ex
+
+INIT_FILE='jina/__init__.py'
+VER_TAG='__version__ = '
+
+function escape_slashes {
+    sed 's/\//\\\//g'
+}
+
+function update_ver_line {
+    local OLD_LINE_PATTERN=$1
+    local NEW_LINE=$2
+    local FILE=$3
+
+    local NEW=$(echo "${NEW_LINE}" | escape_slashes)
+    if [ "$(uname)" == "Darwin" ]; then
+      sed -i '' '/'"${OLD_LINE_PATTERN}"'/s/.*/'"${NEW}"'/' "${FILE}";
+    else
+      sed -i '/'"${OLD_LINE_PATTERN}"'/s/.*/'"${NEW}"'/' "${FILE}";
+    fi
+    head -n10 ${FILE}
+}
+
+
+BRANCH=$(git rev-parse --abbrev-ref HEAD)
+
+#if [[ "$BRANCH" != "main" ]]; then
+#  printf "You are not at main branch, exit\n";
+#  exit 1;
+#fi
+
+LAST_UPDATE=`git show --no-notes --format=format:"%H" $BRANCH | head -n 1`
+LAST_COMMIT=`git show --no-notes --format=format:"%H" origin/$BRANCH | head -n 1`
+
+#if [ $LAST_COMMIT != $LAST_UPDATE ]; then
+#    printf "Your local $BRANCH is behind the remote master, exit\n"
+#    exit 1;
+#fi
+
+# update the current version
+export RELEASE_VER=$(sed -n '/^__version__/p' $INIT_FILE | cut -d \' -f2)
+LAST_VER=$(git tag -l | sort -V | tail -n1)
+printf "last version: \e[1;32m$LAST_VER\e[0m\n"
+
+if [[ $1 == "final" ]]; then
+  printf "this will be a final release: \e[1;33m$RELEASE_VER\e[0m\n"
+
+  NEXT_VER=$(echo $RELEASE_VER | awk -F. -v OFS=. 'NF==1{print ++$NF}; NF>1{$NF=sprintf("%0*d", length($NF), ($NF+1)); print}')
+  printf "bump master version to: \e[1;32m$NEXT_VER\e[0m\n"
+
+  VER_TAG_NEXT=$VER_TAG\'${NEXT_VER}\'
+  update_ver_line "$VER_TAG" "$VER_TAG_NEXT" "$INIT_FILE"
+
+elif [[ $1 == 'rc' ]]; then
+  printf "this will be a release candidate: \e[1;33m$RELEASE_VER\e[0m\n"
+  DOT_RELEASE_VER=$(echo $RELEASE_VER | sed "s/rc/\./")
+  NEXT_VER=$(echo $DOT_RELEASE_VER | awk -F. -v OFS=. 'NF==1{print ++$NF}; NF>1{$NF=sprintf("%0*d", length($NF), ($NF+1)); print}')
+  NEXT_VER=$(echo $NEXT_VER | sed "s/\.\([^.]*\)$/rc\1/")
+  printf "bump master version to: \e[1;32m$NEXT_VER\e[0m, this will be the next version\n"
+
+  VER_TAG_NEXT=$VER_TAG\'${NEXT_VER}\'
+  update_ver_line "$VER_TAG" "$VER_TAG_NEXT" "$INIT_FILE"
+
+elif [[ $1 == 'stateful' ]]; then
+  # as a prerelease, pypi update only, no back commit etc.
+  COMMITS_SINCE_LAST_VER=$(git rev-list $LAST_VER..HEAD --count)
+  NEXT_VER=$RELEASE_VER"a"$COMMITS_SINCE_LAST_VER
+  printf "this will be a stateful executor release: \e[1;33m$NEXT_VER\e[0m\n"
+
+  VER_TAG_NEXT=$VER_TAG\'${NEXT_VER}\'
+  update_ver_line "$VER_TAG" "$VER_TAG_NEXT" "$INIT_FILE"
+
+else
+  # as a prerelease, pypi update only, no back commit etc.
+  COMMITS_SINCE_LAST_VER=$(git rev-list $LAST_VER..HEAD --count)
+  NEXT_VER=$RELEASE_VER".dev"$COMMITS_SINCE_LAST_VER
+  printf "this will be a developmental release: \e[1;33m$NEXT_VER\e[0m\n"
+
+  VER_TAG_NEXT=$VER_TAG\'${NEXT_VER}\'
+  update_ver_line "$VER_TAG" "$VER_TAG_NEXT" "$INIT_FILE"
+
+fi
\ No newline at end of file
diff --git a/setup.py b/setup.py
index e394c46297b21..5885007058472 100644
--- a/setup.py
+++ b/setup.py
@@ -1,8 +1,9 @@
 import os
+import subprocess
 import sys
 from os import path
 
-from setuptools import find_packages, setup
+from setuptools import Extension, find_packages, setup
 from setuptools.command.develop import develop
 from setuptools.command.egg_info import egg_info
 from setuptools.command.install import install
@@ -160,6 +161,30 @@ def get_extra_requires(path, add_all=True):
     final_deps.add('grpcio-health-checking>=1.49.0')
     final_deps.add('grpcio-reflection>=1.49.0')
 
+
+extra_golang_kw = {}
+
+ret_code = -1
+
+try:
+    ret_code = subprocess.run(['go', 'version']).returncode
+except Exception:
+    pass
+
+if ret_code == 0:
+    extra_golang_kw = {
+        'build_golang': {'root': 'jraft', 'strip': False},
+        'ext_modules': [
+            Extension(
+                'jraft',
+                ['jina/serve/consensus/run.go'],
+                py_limited_api=True,
+                define_macros=[('Py_LIMITED_API', None)],
+            )
+        ],
+        'setup_requires': ['setuptools-golang'],
+    }
+
 setup(
     name=pkg_name,
     packages=find_packages(),
@@ -218,4 +243,5 @@ def get_extra_requires(path, add_all=True):
     },
     keywords='jina cloud-native cross-modal multimodal neural-search query search index elastic neural-network encoding '
     'embedding serving docker container image video audio deep-learning mlops',
+    **extra_golang_kw,
 )
diff --git a/tests/conftest.py b/tests/conftest.py
index c4b978b2c7425..3249101ad6e18 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -64,7 +64,6 @@ def test_grpc_fork_support_false(monkeypatch):
     monkeypatch.setenv('GRPC_ENABLE_FORK_SUPPORT', 'true')
 
 
-
 @pytest.fixture(autouse=True)
 def test_timeout_ctrl_time(monkeypatch):
     monkeypatch.setenv('JINA_DEFAULT_TIMEOUT_CTRL', '500')
diff --git a/tests/integration/container_runtime_args/test_container_get_args.py b/tests/integration/container_runtime_args/test_container_get_args.py
index 10d1156fabceb..8408272d2c286 100644
--- a/tests/integration/container_runtime_args/test_container_get_args.py
+++ b/tests/integration/container_runtime_args/test_container_get_args.py
@@ -1,6 +1,3 @@
-import os
-import time
-
 import pytest
 
 from jina import Client, Document, DocumentArray, Flow
diff --git a/tests/integration/external_deployment/test_external_deployment.py b/tests/integration/external_deployment/test_external_deployment.py
index 1eb1a05780bc1..e49d1a5ee2ea2 100644
--- a/tests/integration/external_deployment/test_external_deployment.py
+++ b/tests/integration/external_deployment/test_external_deployment.py
@@ -75,7 +75,6 @@ def test_flow_with_external_deployment(
         del external_args['name']
         del external_args['external']
         del external_args['deployment_role']
-        print(external_args)
         flow = Flow().add(
             **external_args,
             name='external_fake',
diff --git a/tests/integration/issues/github_3124/test_cli_executor.py b/tests/integration/issues/github_3124/test_cli_executor.py
index 1f1087b3cbd40..7df4fa5224695 100644
--- a/tests/integration/issues/github_3124/test_cli_executor.py
+++ b/tests/integration/issues/github_3124/test_cli_executor.py
@@ -29,12 +29,3 @@ def test_executor_cli_docker(docker_image):
     process.terminate()
     assert poll is None
 
-
-def test_zed_runtime_cli_docker(docker_image):
-    process = subprocess.Popen(
-        ['jina', 'executor', '--native', '--uses', 'docker://clitest:latest']
-    )
-    time.sleep(5)
-    poll = process.poll()
-    process.terminate()
-    assert poll == 1  # failed
diff --git a/tests/integration/stateful/__init__.py b/tests/integration/stateful/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/tests/integration/stateful/stateful_no_snapshot_exec/Dockerfile b/tests/integration/stateful/stateful_no_snapshot_exec/Dockerfile
new file mode 100644
index 0000000000000..45b1d8b11f3b7
--- /dev/null
+++ b/tests/integration/stateful/stateful_no_snapshot_exec/Dockerfile
@@ -0,0 +1,7 @@
+FROM jinaai/jina:test-pip
+
+# setup the workspace
+COPY . /workspace
+WORKDIR /workspace
+
+ENTRYPOINT ["jina", "executor", "--uses", "config.yml"]
diff --git a/tests/integration/stateful/stateful_no_snapshot_exec/__init__.py b/tests/integration/stateful/stateful_no_snapshot_exec/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/tests/integration/stateful/stateful_no_snapshot_exec/config.yml b/tests/integration/stateful/stateful_no_snapshot_exec/config.yml
new file mode 100644
index 0000000000000..f5c0e8e6026ed
--- /dev/null
+++ b/tests/integration/stateful/stateful_no_snapshot_exec/config.yml
@@ -0,0 +1,3 @@
+!MyStateExecutorNoSnapshot
+py_modules: executor.py
+
diff --git a/tests/integration/stateful/stateful_no_snapshot_exec/executor.py b/tests/integration/stateful/stateful_no_snapshot_exec/executor.py
new file mode 100644
index 0000000000000..1aae10cea9802
--- /dev/null
+++ b/tests/integration/stateful/stateful_no_snapshot_exec/executor.py
@@ -0,0 +1,48 @@
+from jina import DocumentArray, Executor, requests
+from jina.serve.executors.decorators import write
+import random
+import os
+
+from typing import Dict, List
+
+from docarray.documents import TextDoc
+
+
+class TextDocWithId(TextDoc):
+    id: str
+    tags: Dict[str, str] = {}
+    l: List[str] = []
+
+
+random_num = random.randint(0, 50000)
+
+
+class MyStateExecutorNoSnapshot(Executor):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._docs = DocumentArray[TextDocWithId]()
+        self._docs_dict = {}
+        self.logger.warning(f'RANDOM NUM {random_num}')
+
+    @requests(on=['/index'])
+    @write
+    def index(self, docs: DocumentArray[TextDocWithId], **kwargs):
+        for doc in docs:
+            self.logger.debug(f'Indexing doc {doc.text} with ID {doc.id}')
+            self._docs.append(doc)
+            self._docs_dict[doc.id] = doc
+
+    @requests(on=['/search'])
+    def search(self, docs: DocumentArray[TextDocWithId], **kwargs):
+        for doc in docs:
+            self.logger.debug(f'Searching against {len(self._docs)} documents')
+            doc.text = self._docs_dict[doc.id].text
+            doc.tags['pid'] = os.getpid()
+            doc.tags['num'] = random_num
+
+    @requests(on=['/similarity'])
+    def search_similarity(self, docs: DocumentArray[TextDocWithId], **kwargs):
+        for doc in docs:
+            self.logger.debug(f'Searching similarity against {len(self._docs)} documents')
+            doc.text = 'similarity'
+            doc.l = [doc.id for doc in self._docs]
diff --git a/tests/integration/stateful/stateful_snapshot_exec/Dockerfile b/tests/integration/stateful/stateful_snapshot_exec/Dockerfile
new file mode 100644
index 0000000000000..45b1d8b11f3b7
--- /dev/null
+++ b/tests/integration/stateful/stateful_snapshot_exec/Dockerfile
@@ -0,0 +1,7 @@
+FROM jinaai/jina:test-pip
+
+# setup the workspace
+COPY . /workspace
+WORKDIR /workspace
+
+ENTRYPOINT ["jina", "executor", "--uses", "config.yml"]
diff --git a/tests/integration/stateful/stateful_snapshot_exec/__init__.py b/tests/integration/stateful/stateful_snapshot_exec/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/tests/integration/stateful/stateful_snapshot_exec/config.yml b/tests/integration/stateful/stateful_snapshot_exec/config.yml
new file mode 100644
index 0000000000000..331f4c4974b22
--- /dev/null
+++ b/tests/integration/stateful/stateful_snapshot_exec/config.yml
@@ -0,0 +1,3 @@
+!MyStateExecutor
+py_modules: executor.py
+
diff --git a/tests/integration/stateful/stateful_snapshot_exec/executor.py b/tests/integration/stateful/stateful_snapshot_exec/executor.py
new file mode 100644
index 0000000000000..ec2154385bcd2
--- /dev/null
+++ b/tests/integration/stateful/stateful_snapshot_exec/executor.py
@@ -0,0 +1,64 @@
+from jina import DocumentArray, Executor, requests
+from jina.serve.executors.decorators import write
+import random
+from typing import Dict, List
+import os
+
+from docarray.documents import TextDoc
+
+
+class TextDocWithId(TextDoc):
+    id: str
+    tags: Dict[str, str] = {}
+    l: List[str] = []
+
+
+random_num = random.randint(0, 50000)
+
+
+class MyStateExecutor(Executor):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._docs = DocumentArray[TextDocWithId]()
+        self._docs_dict = {}
+        self.logger.warning(f'RANDOM NUM {random_num}')
+
+    @requests(on=['/index'])
+    @write
+    def index(self, docs: DocumentArray[TextDocWithId], **kwargs):
+        for doc in docs:
+            self.logger.debug(f'Indexing doc {doc.text} with ID {doc.id}')
+            self._docs.append(doc)
+            self._docs_dict[doc.id] = doc
+
+    @requests(on=['/search'])
+    def search(self, docs: DocumentArray[TextDocWithId], **kwargs):
+        for doc in docs:
+            self.logger.debug(f'Searching against {len(self._docs)} documents')
+            doc.text = self._docs_dict[doc.id].text
+            doc.tags['pid'] = os.getpid()
+            doc.tags['num'] = random_num
+
+    @requests(on=['/similarity'])
+    def search_similarity(self, docs: DocumentArray[TextDocWithId], **kwargs):
+        for doc in docs:
+            self.logger.debug(f'Searching similarity against {len(self._docs)} documents')
+            doc.text = 'similarity'
+            doc.l = [doc.id for doc in self._docs]
+
+    def snapshot(self, snapshot_file: str):
+        self.logger.warning(
+            f'Snapshotting to {snapshot_file} with {len(self._docs)} documents'
+        )
+        self.logger.warning(f'Snapshotting with order {[d.text for d in self._docs]}')
+        with open(snapshot_file, 'wb') as f:
+            self._docs.save_binary(f)
+
+    def restore(self, snapshot_file: str):
+        self._docs = DocumentArray[TextDocWithId].load_binary(snapshot_file)
+        for doc in self._docs:
+            self._docs_dict[doc.id] = doc
+        self.logger.warning(
+            f'Restoring from {snapshot_file} with {len(self._docs)} documents'
+        )
+        self.logger.warning(f'Restoring with order {[d.text for d in self._docs]}')
diff --git a/tests/integration/stateful/test_stateful.py b/tests/integration/stateful/test_stateful.py
new file mode 100644
index 0000000000000..d3d83f4e1ddbd
--- /dev/null
+++ b/tests/integration/stateful/test_stateful.py
@@ -0,0 +1,260 @@
+import time
+
+import pytest
+import os
+
+from jina import Client, Document, DocumentArray, Flow, Deployment
+from docarray.documents import TextDoc
+from typing import Dict, List
+
+from jina.helper import random_port
+
+from tests.integration.stateful.stateful_no_snapshot_exec.executor import MyStateExecutorNoSnapshot
+from tests.integration.stateful.stateful_snapshot_exec.executor import MyStateExecutor
+
+cur_dir = os.path.dirname(os.path.abspath(__file__))
+
+
+class TextDocWithId(TextDoc):
+    id: str
+    tags: Dict[str, str] = {}
+    l: List[str] = []
+
+
+@pytest.fixture(scope='module')
+def stateful_exec_docker_image_built():
+    import docker
+
+    client = docker.from_env()
+    client.images.build(
+        path=os.path.join(cur_dir, 'stateful_snapshot_exec/'), tag='stateful-exec'
+    )
+    client.close()
+    yield
+    time.sleep(2)
+    client = docker.from_env()
+    client.containers.prune()
+
+
+def assert_is_indexed(client, search_da):
+    docs = client.search(inputs=search_da, request_size=1, return_type=DocumentArray[TextDocWithId])
+    for doc in docs:
+        assert doc.text == f'ID {doc.id}'
+
+
+def assert_all_replicas_indexed(client, search_da, num_replicas=3, key='pid'):
+    for query in search_da:
+        pids = set()
+        for _ in range(10):
+            for resp in client.search(inputs=query, request_size=1, return_type=DocumentArray[TextDocWithId]):
+                pids.add(resp.tags[key])
+                assert resp.text == f'ID {query.id}'
+            if len(pids) == num_replicas:
+                break
+        assert len(pids) == num_replicas
+
+
+@pytest.mark.parametrize('executor_cls', [MyStateExecutor, MyStateExecutorNoSnapshot])
+@pytest.mark.parametrize('shards', [2, 1])
+def test_stateful_index_search(executor_cls, shards, tmpdir, stateful_exec_docker_image_built):
+    replicas = 3
+    if shards > 1:
+        peer_ports = {}
+        for shard in range(shards):
+            peer_ports[shard] = [random_port() for _ in range(replicas)]
+    else:
+        peer_ports = [random_port() for _ in range(replicas)]
+    dep = Deployment(
+        uses=executor_cls,
+        replicas=replicas,
+        workspace=tmpdir,
+        stateful=True,
+        raft_configuration={
+            'snapshot_interval': 10,
+            'snapshot_threshold': 5,
+            'trailing_logs': 10,
+            'LogLevel': 'INFO',
+        },
+        shards=shards,
+        volumes=[str(tmpdir) + ':' + '/workspace'],
+        peer_ports=peer_ports,
+        polling={'/index': 'ANY', '/search': 'ALL'}
+    )
+    with dep:
+        index_da = DocumentArray[TextDocWithId](
+            [TextDocWithId(id=f'{i}', text=f'ID {i}') for i in range(100)]
+        )
+        search_da = DocumentArray[TextDocWithId]([TextDocWithId(id=f'{i}') for i in range(1)])
+        dep.index(inputs=index_da, request_size=1, return_type=DocumentArray[TextDocWithId])
+
+        # allowing some time for the state to be replicated
+        time.sleep(20)
+        # checking against the main read replica
+        assert_is_indexed(dep, search_da)
+        assert_all_replicas_indexed(dep, search_da)
+        docs = dep.post(on='/similarity', inputs=search_da, request_size=1,
+                        return_type=DocumentArray[TextDocWithId])
+        for doc in docs:
+            assert doc.text == 'similarity'
+            assert len(doc.l) == len(index_da)  # good merging of results
+
+
+@pytest.mark.parametrize('executor_cls', [MyStateExecutor])
+@pytest.mark.parametrize('shards', [1])
+def test_stateful_index_search_restore(executor_cls, shards, tmpdir, stateful_exec_docker_image_built):
+    replicas = 3
+    peer_ports = {}
+    for shard in range(shards):
+        peer_ports[shard] = [random_port() for _ in range(replicas)]
+    dep = Deployment(
+        uses=executor_cls,
+        replicas=replicas,
+        workspace=tmpdir,
+        stateful=True,
+        raft_configuration={
+            'snapshot_interval': 10,
+            'snapshot_threshold': 5,
+            'trailing_logs': 10,
+            'LogLevel': 'INFO',
+        },
+        shards=shards,
+        volumes=[str(tmpdir) + ':' + '/workspace'],
+        peer_ports=peer_ports,
+        polling={'/index': 'ANY', '/search': 'ALL'}
+    )
+    with dep:
+        index_da = DocumentArray[TextDocWithId](
+            [TextDocWithId(id=f'{i}', text=f'ID {i}') for i in range(100)]
+        )
+        search_da = DocumentArray[TextDocWithId]([TextDocWithId(id=f'{i}') for i in range(1)])
+        dep.index(inputs=index_da, request_size=1, return_type=DocumentArray[TextDocWithId])
+
+        # allowing some time for the state to be replicated
+        time.sleep(20)
+        # checking against the main read replica
+        assert_is_indexed(dep, search_da)
+        assert_all_replicas_indexed(dep, search_da)
+
+    # test restoring
+    with dep:
+        index_da = DocumentArray[TextDocWithId](
+            [TextDocWithId(id=f'{i}', text=f'ID {i}') for i in range(100, 200)]
+        )
+        dep.index(inputs=index_da, request_size=1, return_type=DocumentArray[TextDocWithId])
+        time.sleep(20)
+        search_da = DocumentArray[TextDocWithId]([TextDocWithId(id=f'{i}') for i in range(200)])
+        assert_all_replicas_indexed(dep, search_da)
+
+
+@pytest.mark.skip('Not sure how containerization will work with docarray v2')
+@pytest.mark.parametrize('shards', [1, 2])
+def test_stateful_index_search_container(shards, tmpdir, stateful_exec_docker_image_built):
+    replicas = 3
+    peer_ports = {}
+    for shard in range(shards):
+        peer_ports[shard] = [random_port() for _ in range(replicas)]
+
+    dep = Deployment(
+        uses='docker://stateful-exec',
+        replicas=replicas,
+        stateful=True,
+        raft_configuration={
+            'snapshot_interval': 10,
+            'snapshot_threshold': 5,
+            'trailing_logs': 10,
+            'LogLevel': 'INFO',
+        },
+        shards=shards,
+        workspace='/workspace/tmp',
+        volumes=[str(tmpdir) + ':' + '/workspace/tmp'],
+        peer_ports=peer_ports,
+        polling={'/index': 'ANY', '/search': 'ALL'}
+    )
+    with dep:
+        index_da = DocumentArray[TextDocWithId](
+            [TextDocWithId(id=f'{i}', text=f'ID {i}') for i in range(100)]
+        )
+        search_da = DocumentArray[TextDocWithId]([TextDocWithId(id=f'{i}') for i in range(100)])
+        dep.index(inputs=index_da, request_size=1, return_type=DocumentArray[TextDocWithId])
+
+        # allowing some time for the state to be replicated
+        time.sleep(10)
+        # checking against the main read replica
+        assert_is_indexed(dep, search_da)
+        assert_all_replicas_indexed(dep, search_da, key='random_num')
+
+    # test restoring
+    with dep:
+        index_da = DocumentArray[TextDocWithId](
+            [Document(id=f'{i}', text=f'ID {i}') for i in range(100, 200)]
+        )
+        dep.index(inputs=index_da, request_size=1, return_type=DocumentArray[TextDocWithId])
+        time.sleep(10)
+        search_da = DocumentArray[TextDocWithId]([TextDocWithId(id=f'{i}') for i in range(200)])
+        assert_all_replicas_indexed(dep, search_da, key='random_num')
+
+
+@pytest.mark.skip()
+@pytest.mark.parametrize('executor_cls', [MyStateExecutor, MyStateExecutorNoSnapshot])
+def test_add_new_replica(executor_cls, tmpdir):
+    from jina.parsers import set_pod_parser
+    from jina.orchestrate.pods.factory import PodFactory
+    gateway_port = random_port()
+
+    ctx_mngr = Flow(port=gateway_port).add(
+        uses=executor_cls,
+        replicas=3,
+        workspace=tmpdir,
+        stateful=True,
+        raft_configuration={
+            'snapshot_interval': 10,
+            'snapshot_threshold': 5,
+            'trailing_logs': 10,
+            'LogLevel': 'INFO',
+        },
+    )
+    with ctx_mngr:
+        index_da = DocumentArray[TextDocWithId](
+            [Document(id=f'{i}', text=f'ID {i}') for i in range(100)]
+        )
+        ctx_mngr.index(inputs=index_da, request_size=1)
+        # allowing sometime for snapshots
+        time.sleep(30)
+
+        new_replica_port = random_port()
+        args = set_pod_parser().parse_args([])
+        args.host = args.host[0]
+        args.port = [new_replica_port]
+        args.stateful = True
+        args.workspace = str(tmpdir)
+        args.uses = executor_cls.__name__
+        args.replica_id = '4'
+        with PodFactory.build_pod(args) as p:
+            import psutil
+            current_pid = os.getpid()
+            ports = set()
+            for proc in psutil.process_iter(['pid', 'ppid', 'name']):
+                if proc.info['ppid'] == current_pid and proc.info['pid'] != current_pid:
+                    for conn in proc.connections():
+                        if conn.status == 'LISTEN':
+                            ports.add(conn.laddr.port)
+            for port in ports:
+                try:
+                    leader_address = f'0.0.0.0:{port}'  # detect the Pods addresses of the original Flow
+                    voter_address = f'0.0.0.0:{new_replica_port}'
+                    import jraft
+                    jraft.add_voter(
+                        leader_address, '4', voter_address
+                    )
+                    break
+                except:
+                    pass
+            time.sleep(10)
+            index_da = DocumentArray[TextDocWithId](
+                [TextDocWithId(id=f'{i}', text=f'ID {i}') for i in range(100, 200)]
+            )
+            ctx_mngr.index(inputs=index_da, request_size=1, return_type=DocumentArray[TextDocWithId])
+            time.sleep(10)
+            search_da = DocumentArray[TextDocWithId]([TextDocWithId(id=f'{i}') for i in range(200)])
+            client = Client(port=new_replica_port)
+            assert_is_indexed(client, search_da=search_da)
diff --git a/tests/k8s/test_k8s_graceful_request_handling.py b/tests/k8s/test_k8s_graceful_request_handling.py
index f1d533b821e33..7dfbf3e96cadb 100644
--- a/tests/k8s/test_k8s_graceful_request_handling.py
+++ b/tests/k8s/test_k8s_graceful_request_handling.py
@@ -302,7 +302,7 @@ async def test_no_message_lost_during_kill(logger, docker_images, tmpdir):
             daemon=True,
         )
         process.start()
-        time.sleep(1.0)
+        time.sleep(3.0)
         logger.debug('Kill 2 replicas')
 
         pods = core_client.list_namespaced_pod(
@@ -386,7 +386,7 @@ async def test_linear_processing_time_scaling(docker_images, logger, tmpdir):
     with portforward.forward(
         namespace, gateway_pod_name, flow.port, flow.port, config_path
     ):
-        time.sleep(0.1)
+        time.sleep(3.0)
         client_kwargs = dict(
             host='localhost',
             port=flow.port,
diff --git a/tests/unit/serve/executors/test_executor.py b/tests/unit/serve/executors/test_executor.py
index 5df1fec040ae4..33841fcb696f8 100644
--- a/tests/unit/serve/executors/test_executor.py
+++ b/tests/unit/serve/executors/test_executor.py
@@ -15,6 +15,7 @@
 from jina.constants import __cache_path__
 from jina.excepts import RuntimeFailToStart
 from jina.helper import random_port
+from jina.serve.executors.decorators import write
 from jina.serve.executors.metas import get_default_metas
 from jina.serve.networking.utils import send_request_async
 from jina.serve.runtimes.asyncio import AsyncNewLoopRuntime
@@ -663,3 +664,27 @@ def foo(self, docs, **kwargs):
 
     exec = MyExecutor2()
     assert exec.dynamic_batching['foo'] == expected_values
+
+
+def test_write_decorator():
+    class WriteExecutor(Executor):
+        @requests(on='/index')
+        @write()
+        def index(self, **kwargs):
+            pass
+
+        @write()
+        @requests(on='/update')
+        def update(self, **kwargs):
+            pass
+
+        @requests(on='/search')
+        def search(self, **kwargs):
+            pass
+
+        @requests
+        def foo(self, **kwargs):
+            pass
+
+    exec = WriteExecutor()
+    assert set(exec.write_endpoints) == {'/index', '/update'}
diff --git a/tests/unit/serve/runtimes/head/test_head_runtime.py b/tests/unit/serve/runtimes/head/test_head_runtime.py
index e18029fbcd556..4680520babcc5 100644
--- a/tests/unit/serve/runtimes/head/test_head_runtime.py
+++ b/tests/unit/serve/runtimes/head/test_head_runtime.py
@@ -103,46 +103,6 @@ def test_uses_before_uses_after():
     _destroy_runtime(args, cancel_event, runtime_thread)
 
 
-def test_decompress(monkeypatch):
-    call_counts = multiprocessing.Manager().Queue()
-
-    def decompress(self):
-        call_counts.put_nowait('called')
-        from jina.proto import jina_pb2
-
-        self._pb_body = jina_pb2.DataRequestProto()
-        self._pb_body.ParseFromString(self.buffer)
-        self.buffer = None
-
-    monkeypatch.setattr(
-        DataRequest,
-        '_decompress',
-        decompress,
-    )
-
-    args = _generate_pod_args()
-    args.polling = PollingType.ANY
-    connection_list_dict = {0: [f'fake_ip:8080']}
-    args.connection_list = json.dumps(connection_list_dict)
-    cancel_event, handle_queue, runtime_thread = _create_runtime(args)
-
-    with grpc.insecure_channel(
-        f'{args.host}:{args.port[0]}',
-        options=get_default_grpc_options(),
-    ) as channel:
-        stub = jina_pb2_grpc.JinaSingleDataRequestRPCStub(channel)
-        response, call = stub.process_single_data.with_call(_create_test_data_message())
-
-    assert response
-    assert 'is-error' in dict(call.trailing_metadata())
-    assert _queue_length_copy(call_counts) == 0
-    assert len(response.docs) == 1
-    assert _queue_length_copy(call_counts) == 1
-    assert not handle_queue.empty()
-
-    _destroy_runtime(args, cancel_event, runtime_thread)
-
-
 @pytest.mark.parametrize('polling', ['any', 'all'])
 def test_dynamic_polling(polling):
     args = _generate_pod_args(
diff --git a/toy_stateful.py b/toy_stateful.py
new file mode 100644
index 0000000000000..2e523184049b5
--- /dev/null
+++ b/toy_stateful.py
@@ -0,0 +1,195 @@
+import os
+import time
+
+from jina import Executor, Flow, requests, DocumentArray, Document, Client
+from jina.serve.executors.decorators import write
+from jina.parsers import set_pod_parser
+from jina.orchestrate.pods.factory import PodFactory
+
+os.environ['JINA_LOG_LEVEL'] = 'DEBUG'
+
+
+class MyStateExecutor(Executor):
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._docs = DocumentArray()
+
+    @requests(on=['/index'])
+    @write
+    def index(self, docs, **kwargs):
+        time.sleep(0.2)
+        for doc in docs:
+            self.logger.debug(f'Indexing doc {doc.text}')
+            self._docs.append(doc)
+
+    @requests(on=['/search'])
+    def search(self, docs, **kwargs):
+        time.sleep(0.2)
+        for doc in docs:
+            self.logger.debug(f'Searching doc {doc.text} against {len(self._docs)} indexed documents')
+
+    def snapshot(self, snapshot_file: str):
+        self.logger.warning(f'Snapshotting to {snapshot_file} with {len(self._docs)} documents')
+        self.logger.warning(f'Snapshotting with order {[d.text for d in self._docs]}')
+        with open(snapshot_file, 'wb') as f:
+            self._docs.save_binary(f)
+
+    def restore(self, snapshot_file: str):
+        self._docs = DocumentArray.load_binary(snapshot_file)
+        self.logger.warning(f'Restoring from {snapshot_file} with {len(self._docs)} documents')
+        self.logger.warning(f'Restoring with order {[d.text for d in self._docs]}')
+
+
+class MyStateExecutorNoSpanshot(Executor):
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._docs = DocumentArray()
+
+    @requests(on=['/index'])
+    @write
+    def index(self, docs, **kwargs):
+        time.sleep(0.2)
+        for doc in docs:
+            self.logger.debug(f'Indexing doc {doc.text}')
+            self._docs.append(doc)
+
+    @requests(on=['/search'])
+    def search(self, docs, **kwargs):
+        time.sleep(0.2)
+        for doc in docs:
+            self.logger.debug(f'Searching doc {doc.text} against {len(self._docs)} indexed documents')
+
+
+PORT_FLOW_SNAPSHOT = 12340
+PORTS_REPLICAS_SNAPSHOT = {'0': [12345, 12347, 12349]}
+
+PORTS_NEW_REPLICA_SNAPSHOT = 12351
+FOLDER_WORKSPACE_SNAPSHOT = './toy_workspace_with_snapshots'
+
+PORT_FLOW_NO_SNAPSHOT = 12360
+PORTS_REPLICAS_NO_SNAPSHOT = {'0': [12365, 12367, 12369]}
+PORTS_NEW_REPLICA_NO_SNAPSHOT = 12371
+FOLDER_WORKSPACE_NO_SNAPSHOT = './toy_workspace_no_snapshots'
+
+if __name__ == '__main__':
+    import sys
+
+    args = sys.argv
+    client = None
+    index = False
+    option = args[1]
+    if option == 'index_snapshot':
+        f = Flow(port=PORT_FLOW_SNAPSHOT).add(uses=MyStateExecutor,
+                                              replicas=3,
+                                              # shards=2,
+                                              workspace=FOLDER_WORKSPACE_SNAPSHOT,
+                                              peer_ports=PORTS_REPLICAS_SNAPSHOT,
+                                              stateful=True,
+                                              raft_configuration={'snapshot_interval': 10, 'snapshot_threshold': 5, 'trailing_logs': 10,
+                                                                  'LogLevel': 'INFO'})
+        with f:
+            f.block()
+    elif option == 'index_no_snapshot':
+        f = Flow(port=PORT_FLOW_NO_SNAPSHOT).add(uses=MyStateExecutorNoSpanshot,
+                                                 replicas=3,
+                                                 # shards=2,
+                                                 workspace=FOLDER_WORKSPACE_NO_SNAPSHOT,
+                                                 peer_ports=PORTS_REPLICAS_NO_SNAPSHOT,
+                                                 stateful=True,
+                                                 raft_configuration={'snapshot_interval': 10, 'snapshot_threshold': 5, 'trailing_logs': 10,
+                                                                     'LogLevel': 'INFO'})
+        with f:
+            f.block()
+    elif option == 'restore_snapshot':
+        f = Flow(port=PORT_FLOW_SNAPSHOT).add(uses=MyStateExecutor,
+                                              replicas=3,
+                                              # shards=2,
+                                              workspace=FOLDER_WORKSPACE_SNAPSHOT,
+                                              peer_ports=PORTS_REPLICAS_SNAPSHOT,
+                                              stateful=True,
+                                              raft_configuration={'snapshot_interval': 10, 'snapshot_threshold': 5, 'trailing_logs': 10,
+                                                                  'LogLevel': 'INFO'})
+        with f:
+            f.block()
+    elif option == 'restore_no_snapshot':
+        f = Flow(port=PORT_FLOW_NO_SNAPSHOT).add(uses=MyStateExecutorNoSpanshot,
+                                                 replicas=3,
+                                                 # shards=2,
+                                                 workspace=FOLDER_WORKSPACE_NO_SNAPSHOT,
+                                                 peer_ports=PORTS_REPLICAS_NO_SNAPSHOT,
+                                                 stateful=True,
+                                                 raft_configuration={'snapshot_interval': 10, 'snapshot_threshold': 5, 'trailing_logs': 10,
+                                                                     'LogLevel': 'INFO'})
+        with f:
+            f.block()
+    elif option == 'start_new_replica_snapshot':
+        args = set_pod_parser().parse_args([])
+        args.host = args.host[0]
+        args.port = [PORTS_NEW_REPLICA_SNAPSHOT]
+        args.stateful = True
+        args.workspace = FOLDER_WORKSPACE_SNAPSHOT
+        args.uses = 'MyStateExecutor'
+        args.replica_id = '4'
+        with PodFactory.build_pod(args) as p:
+            p.join()
+    elif option == 'start_new_replica_no_snapshot':
+        args = set_pod_parser().parse_args([])
+        args.host = args.host[0]
+        args.port = [PORTS_NEW_REPLICA_NO_SNAPSHOT]
+        args.stateful = True
+        args.workspace = FOLDER_WORKSPACE_NO_SNAPSHOT
+        args.uses = 'MyStateExecutorNoSnapshot'
+        args.replica_id = '4'
+        with PodFactory.build_pod(args) as p:
+            p.join()
+    elif option == 'add_new_snapshot_replica_to_cluster':
+        import jraft
+        leader_address = f'0.0.0.0:64937'
+        voter_address = f'0.0.0.0:55155'
+        jraft.add_voter(
+            leader_address, '4', voter_address
+        )
+    elif option == 'add_new_no_snapshot_replica_to_cluster':
+        import jraft
+        leader_address = f'0.0.0.0:{PORTS_REPLICAS_NO_SNAPSHOT[0]}'
+        voter_address = f'0.0.0.0:{PORTS_NEW_REPLICA_NO_SNAPSHOT}'
+        jraft.add_voter(
+            leader_address, '4', voter_address
+        )
+    elif option == 'client_index_search_snapshot':
+        client = Client(port=PORT_FLOW_SNAPSHOT)
+        index = True
+    elif option == 'client_search_snapshot':
+        client = Client(port=PORT_FLOW_SNAPSHOT)
+    elif option == 'client_index_search_no_snapshot':
+        client = Client(port=PORT_FLOW_NO_SNAPSHOT)
+        index = True
+    elif option == 'client_search_no_snapshot':
+        client = Client(port=PORT_FLOW_NO_SNAPSHOT)
+    else:
+        print(f' INVALID OPTION {option}')
+
+    if client is not None:
+        search_da = DocumentArray([Document(text='SEARCH') for _ in range(10)])
+        if index is True:
+            index_da = DocumentArray([Document(text=f'ID {i}') for i in range(1000)])
+            client.index(inputs=index_da[0:1], request_size=1)
+            client.search(inputs=search_da[0:10], request_size=1)
+            time.sleep(2)
+            client.index(inputs=index_da[10:20], request_size=1)
+            client.search(inputs=search_da[0:10], request_size=1)
+            time.sleep(2)
+            client.index(inputs=index_da[20:30], request_size=1)
+            client.search(inputs=search_da[0:10], request_size=1)
+            time.sleep(2)
+            client.index(inputs=index_da[30:40], request_size=1)
+            client.search(inputs=search_da[0:10], request_size=1)
+        else:
+            client.search(inputs=search_da[0:10], request_size=1)
+            client.search(inputs=search_da[0:10], request_size=1)
+            client.search(inputs=search_da[0:10], request_size=1)
+
+# JINA_LOG_LEVEL=DEBUG jina executor --uses MyStateExecutor --port 55555 --replica-id '4' --workspace toy_workspace
+# --stateful --py-modules toy_stateful.py


Code review comments:
diff_hunk: @@ -183,3 +183,33 @@ structure.
 ```{tip}
 In the `jina hub new` wizard you can choose from four Dockerfile templates: `cpu`, `tf-gpu`, `torch-gpu`, and `jax-gpu`.
 ```
+
+## Stateful-Executor (Beta)
+
+Executors sometimes may contain some internal state which changes when some of their methods are called. For instance, an Executor could contain an index of Documents
+to perform vector search.
+
+In these cases, orchestrating these Executors can be tougher than Executors that never change their inner state (Imagine a Machine Learning model served via an Executor that never updates its weights during its lifetime).
+The challenge comes at guaranteeing the consistency between `replicas` of the same Executor inside the same Deployment.
+
+In order to provide this consistency, Executors can mark some of their exposed methods as `write`. This indicates that calls to these endpoints must be consistently replicated between all the replicas
+such that other endpoints can serve independently of the replica that is hit.
+
+````{admonition} Deterministic state update
+:class: note
+
+Another consideration that needs to be there, is that the Executor inner state must evolve in a deterministic manner if we want `replicas` to behave consistently.
+````
+
+By considering this, {ref}`Executors can be scaled in a consistent manner<scale-consensus>`.
+
+### Snapshot and restoring
+
+When having an Stateful Executor, Jina uses RAFT consensus algorithm to guarantee that every replica eventually holds the same inner state. 
+RAFT writes the incoming requests as logs to local storage in every replica to make sure this is achieved. 
path: docs/concepts/serving/executor/create.md
text: @user1:

```suggestion
In a Stateful Executor Jina uses the RAFT consensus algorithm to guarantee that every replica eventually holds the same inner state. 
RAFT writes the incoming requests as logs to local storage in every replica to ensure this is achieved. 
```