Repository: deepset-ai/haystack
Pull request: ci: Add Github workflow to automate benchmark runs
### Related Issues

- fixes #4904
- depends on #5432 

### Proposed Changes:

 <!--- In case of a bug: Describe what caused the issue and how you solved it -->
 <!--- In case of a feature: Describe what did you add and how it works -->
This PR adapts the Github workflow `cml.yml` which does the following:
- Spin up an AWS EC2 machine of type p3.2xlarge using CML
- Install Haystack on that machine
- Sequentially run benchmarks for all configurations
- Send results to Datadog

### How did you test it?

<!-- unit tests, integration tests, manual verification, instructions for manual tests -->

### Notes for the reviewer

<!-- E.g. point out section where the reviewer  -->
This PR build on top of #5176, where we decided to use Github Actions instead of Lambda Functions to schedule the benchmarks. 

### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- I have updated the related issue with new insights and changes
- I added unit tests and updated the docstrings
- I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`.
- I documented my code
- I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
Automate running benchmarking script
We want to automate the benchmarking process using Github Actions to ensure that we can monitor the performance of Haystack easily. The benchmarks should be triggered weekly and whenever we branch off for a new version release. The action should spin up an EC2 instance for each configuration we want to benchmark, run the benchmarks, compare the results with previous benchmarking runs, and send a slack notification in case there is a significant change. This will help us identify performance changes quickly and take action accordingly.

Hints from issue discussion:

Commit message: Always terminate runner

Patch to review:
diff --git a/.github/workflows/cml.yml b/.github/workflows/cml.yml
index 545034427f..2b24757226 100644
--- a/.github/workflows/cml.yml
+++ b/.github/workflows/cml.yml
@@ -3,87 +3,249 @@ name: benchmarks
 on:
   workflow_dispatch:
 
+permissions:
+  id-token: write
+  contents: read
+
+env:
+  AWS_REGION: eu-central-1
+
 jobs:
-  deploy-cloud-runner:
-    runs-on: [ubuntu-latest]
-    container: docker://dvcorg/cml
+  deploy-runner:
+    runs-on: ubuntu-latest
+    outputs:
+      cml_runner_id: ${{ steps.deploy.outputs.cml_runner_id }}
     steps:
-      - name: deploy
+      - uses: actions/checkout@v3
+
+      - uses: iterative/setup-cml@v1
+
+      - name: AWS authentication
+        uses: aws-actions/configure-aws-credentials@5fd3084fc36e372ff1fff382a39b10d03659f355
+        with:
+          aws-region: ${{ env.AWS_REGION }}
+          role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }}
+
+      - name: Launch EC2 instance and deploy runner
+        id: deploy
         env:
           repo_token: ${{ secrets.HAYSTACK_BOT_TOKEN }}
-          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_ACCESS_KEY }}
-          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_SECRET_ACCESS_KEY }}
-          VPC: ${{ secrets.AWS_CI_VPC }}
+        run: |
+          cd test/benchmarks
+          OUTPUT=$(cml runner launch \
+          --cloud aws \
+          --cloud-region ${{ env.AWS_REGION }} \
+          --cloud-type=p3.2xlarge \
+          --cloud-hdd-size=64 \
+          --labels=cml 2>&1 | tee /dev/fd/2)
+
+          # Extract 'id' from the log and set it as an environment variable
+          ID_VALUE=$(echo "$OUTPUT" | jq -r '.message? | fromjson? | select(.id != null) | .id // empty')
+          echo "cml_runner_id=$ID_VALUE" >> "$GITHUB_OUTPUT"
+          echo "ID_VALUE: $ID_VALUE"
+
+  run-reader-benchmarks:
+    needs: deploy-runner
+    runs-on: [self-hosted, cml]
+    container:
+      image: docker://iterativeai/cml:0-dvc2-base1-gpu
+      options: --gpus all
+    timeout-minutes: 2880
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Install Haystack + Datadog requirements
+        run: |
+          pip install .[metrics,benchmarks,inference]
+          pip install -r test/benchmarks/datadog/requirements.txt
 
+      - name: Run benchmarks
+        working-directory: test/benchmarks
         run: |
-          echo "Deploying..."
-          RUNNER_LABELS="cml,aws"
-          RUNNER_REPO="https://github.com/${GITHUB_REPOSITORY}"
-          MACHINE="cml$(date +%s)"
-          docker-machine create \
-            --driver amazonec2 \
-            --amazonec2-instance-type p3.2xlarge \
-            --amazonec2-vpc-id "$VPC" \
-            --amazonec2-region us-east-1 \
-            --amazonec2-zone c \
-            --amazonec2-ssh-user ubuntu \
-            --amazonec2-ami ami-06a25ee8966373068 \
-            --amazonec2-root-size 150 \
-            "$MACHINE"
-          eval "$(docker-machine env --shell sh "$MACHINE")"
-
-          (
-          docker-machine ssh "$MACHINE" "sudo mkdir -p \
-            /docker_machine && \
-          sudo chmod 777 /docker_machine" && \
-          docker-machine scp -r -q ~/.docker/machine/ \
-            "$MACHINE:/docker_machine" && \
-          docker run --name elasticsearch -d \
-                     -p 9200:9200 \
-                     -e "discovery.type=single-node" \
-                    elasticsearch:7.9.2 && \
-          docker run --name postgres -d \
-                      -p 5432:5432 \
-                      --net host \
-                      -e POSTGRES_PASSWORD=password \
-                      -v /docker_machine/machine:/root/.docker/machine \
-                      -e "DOCKER_MACHINE=$MACHINE" \
-                      postgres && \
-          sleep 4 && \
-          docker exec -i postgres psql -U postgres -c "CREATE DATABASE haystack;" && \
-          docker run --name runner -d \
-            --gpus all \
-            -v /docker_machine/machine:/root/.docker/machine \
-            --net host \
-            -e "DOCKER_MACHINE=$MACHINE" \
-            -e "repo_token=$repo_token" \
-            -e "RUNNER_LABELS=$RUNNER_LABELS" \
-            -e "RUNNER_REPO=$RUNNER_REPO" \
-            -e RUNNER_IDLE_TIMEOUT=120 \
-            dvcorg/cml-py3:latest && \
-          sleep 20 && echo "Deployed $MACHINE"
-          ) || (echo "Shut down machine" && docker-machine rm -y -f "$MACHINE" && exit 1)
-  run-benchmark:
-    needs: deploy-cloud-runner
-    runs-on: [self-hosted,cml]
+          mkdir +p out
+          for f in ./configs/reader/*.yml; do
+            name="${f%.*}"
+            echo "=== Running benchmarks for $name ===";
+            config_name="$(basename "$name")"
+            python run.py --output "out/$config_name.json" "$f";
+            echo "=== Benchmarks done for $name (or failed) ===";
+          done
+
+      - name: Send Benchmark results to Datadog
+        working-directory: test/benchmarks
+        run: |
+          python datadog/send_metrics.py out/ ${{ secrets.CORE_DATADOG_API_KEY }} https://api.datadoghq.eu
+
+      - name: Archive benchmark results
+        uses: actions/upload-artifact@v3
+        with:
+          name: benchmark-results-reader
+          path: test/benchmarks/out/
+
+  run-elasticsearch-benchmarks:
+    needs:
+      - deploy-runner
+      - run-reader-benchmarks
+    runs-on: [self-hosted, cml]
+    container:
+      image: docker://iterativeai/cml:0-dvc2-base1-gpu
+      options: --gpus all
+    services:
+      elasticsearch:
+        image: elasticsearch:7.17.6
+        env:
+          discovery.type: "single-node"
+        ports:
+          - 9201:9200
+    timeout-minutes: 2880
+
     steps:
       - uses: actions/checkout@v3
-      - name: cml_run
+
+      - name: Install Haystack + Datadog requirements
+        run: |
+          pip install .[metrics,elasticsearch,benchmarks,inference]
+          pip install -r test/benchmarks/datadog/requirements.txt
+
+      - name: Run benchmarks
+        working-directory: test/benchmarks
+        run: |
+          mkdir +p out
+          for f in ./configs/**/*-elasticsearch-*.yml; do
+            name="${f%.*}"
+            echo "=== Running benchmarks for $name ===";
+            config_name="$(basename "$name")"
+            python run.py --output "out/$config_name.json" "$f";
+            echo "=== Benchmarks done for $name (or failed) ===";
+          done
+
+      - name: Send Benchmark results to Datadog
+        working-directory: test/benchmarks
+        run: |
+          python datadog/send_metrics.py out/ ${{ secrets.CORE_DATADOG_API_KEY }} https://api.datadoghq.eu
+
+      - name: Archive benchmark results
+        uses: actions/upload-artifact@v3
+        with:
+          name: benchmark-results-elasticsearch
+          path: test/benchmarks/out/
+
+  run-weaviate-benchmarks:
+    needs:
+      - deploy-runner
+      - run-elasticsearch-benchmarks
+    runs-on: [self-hosted, cml]
+    container:
+      image: docker://iterativeai/cml:0-dvc2-base1-gpu
+      options: --gpus all
+    services:
+      weaviate:
+        image: semitechnologies/weaviate:1.17.2
         env:
-          repo_token: ${{ secrets.HAYSTACK_BOT_TOKEN }}
-          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_CI_ACCESS_KEY }}
-          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_CI_SECRET_ACCESS_KEY }}
+          AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "true"
+          PERSISTENCE_DATA_PATH: "/var/lib/weaviate"
+        ports:
+          - 8080:8080
+    timeout-minutes: 2880
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Install Haystack + Datadog requirements
+        run: |
+          pip install .[metrics,weaviate,benchmarks,inference]
+          pip install -r test/benchmarks/datadog/requirements.txt
+
+      - name: Run benchmarks
+        working-directory: test/benchmarks
+        run: |
+          mkdir +p out
+          for f in ./configs/**/*-weaviate-*.yml; do
+            name="${f%.*}"
+            echo "=== Running benchmarks for $name ===";
+            config_name="$(basename "$name")"
+            python run.py --output "out/$config_name.json" "$f";
+            echo "=== Benchmarks done for $name (or failed) ===";
+          done
+
+      - name: Send Benchmark results to Datadog
+        working-directory: test/benchmarks
+        run: |
+          python datadog/send_metrics.py out/ ${{ secrets.CORE_DATADOG_API_KEY }} https://api.datadoghq.eu
+
+      - name: Archive benchmark results
+        uses: actions/upload-artifact@v3
+        with:
+          name: benchmark-results-weaviate
+          path: test/benchmarks/out/
+
+  run-opensearch-benchmarks:
+    needs:
+      - deploy-runner
+      - run-weaviate-benchmarks
+    runs-on: [ self-hosted, cml ]
+    container:
+      image: docker://iterativeai/cml:0-dvc2-base1-gpu
+      options: --gpus all
+    services:
+      opensearch:
+        image: opensearchproject/opensearch:1.3.5
+        env:
+          discovery.type: "single-node"
+          OPENSEARCH_JAVA_OPTS: "-Xms4096m -Xmx4096m"
+        ports:
+          - 9200:9200
+    timeout-minutes: 2880
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Install Haystack + Datadog requirements
+        run: |
+          pip install .[metrics,opensearch,benchmarks,inference]
+          pip install -r test/benchmarks/datadog/requirements.txt
+
+      - name: Run benchmarks
+        working-directory: test/benchmarks
         run: |
-          apt-get update -y
-          apt-get install python3-dev -y
-          pip install .[elasticsearch,faiss,weaviate,ray,rest,ui,dev]
-          cd test/benchmarks && python run.py --retriever_index --retriever_query --reader --ci --save_markdown
-          {
-            echo -en "## Benchmarks: Retriever Indexing\n"
-            cat retriever_index_results.md
-            echo -en "\n\n## Benchmarks: Retriever Querying\n"
-            cat retriever_query_results.md
-            echo -en "\n\n## Benchmarks: Reader\n"
-            cat reader_results.md
-          } >> report.md
-          cml-send-comment report.md
+          mkdir +p out
+          for f in ./configs/**/*-opensearch-*.yml; do
+            name="${f%.*}"
+            echo "=== Running benchmarks for $name ===";
+            config_name="$(basename "$name")"
+            python run.py --output "out/$config_name.json" "$f";
+            echo "=== Benchmarks done for $name (or failed) ===";
+          done
+
+      - name: Send Benchmark results to Datadog
+        working-directory: test/benchmarks
+        run: |
+          python datadog/send_metrics.py out/ ${{ secrets.CORE_DATADOG_API_KEY }} https://api.datadoghq.eu
+
+      - name: Archive benchmark results
+        uses: actions/upload-artifact@v3
+        with:
+          name: benchmark-results-opensearch
+          path: test/benchmarks/out/
+
+  terminate-runner:
+    if: always()
+    needs:
+      - deploy-runner
+      - run-opensearch-benchmarks
+    runs-on: ubuntu-latest
+    steps:
+    - name: AWS authentication
+      uses: aws-actions/configure-aws-credentials@5fd3084fc36e372ff1fff382a39b10d03659f355
+      with:
+        aws-region: ${{ env.AWS_REGION }}
+        role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }}
+
+    - name: Terminate EC2 instance
+      env:
+        CML_RUNNER_ID: ${{needs.deploy-runner.outputs.cml_runner_id}}
+      run: |
+        # Get the instance ID using its Name tag and terminate the instance
+        INSTANCE_ID=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=${{ env.CML_RUNNER_ID }}" --query "Reservations[*].Instances[*].[InstanceId]" --output text)
+        aws ec2 terminate-instances --instance-ids "$INSTANCE_ID"
diff --git a/test/benchmarks/configs/reader/debertabase.yaml b/test/benchmarks/configs/reader/debertabase.yml
similarity index 100%
rename from test/benchmarks/configs/reader/debertabase.yaml
rename to test/benchmarks/configs/reader/debertabase.yml
diff --git a/test/benchmarks/configs/reader/debertalarge.yaml b/test/benchmarks/configs/reader/debertalarge.yml
similarity index 100%
rename from test/benchmarks/configs/reader/debertalarge.yaml
rename to test/benchmarks/configs/reader/debertalarge.yml
diff --git a/test/benchmarks/configs/reader/tinyroberta.yaml b/test/benchmarks/configs/reader/tinyroberta.yml
similarity index 100%
rename from test/benchmarks/configs/reader/tinyroberta.yaml
rename to test/benchmarks/configs/reader/tinyroberta.yml
diff --git a/test/benchmarks/configs/retriever/bm25-elasticsearch-100k.yml b/test/benchmarks/configs/retriever/bm25-elasticsearch-100k.yml
index 9bd8debc10..1500cb50ee 100644
--- a/test/benchmarks/configs/retriever/bm25-elasticsearch-100k.yml
+++ b/test/benchmarks/configs/retriever/bm25-elasticsearch-100k.yml
@@ -3,6 +3,9 @@ version: ignore
 components:
   - name: DocumentStore
     type: ElasticsearchDocumentStore
+    params:
+      host: elasticsearch
+      port: 9200
   - name: TextConverter
     type: TextConverter
   - name: Retriever
diff --git a/test/benchmarks/configs/retriever/bm25-opensearch-100k.yml b/test/benchmarks/configs/retriever/bm25-opensearch-100k.yml
index 6aeb44d425..a798ca532d 100644
--- a/test/benchmarks/configs/retriever/bm25-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever/bm25-opensearch-100k.yml
@@ -3,6 +3,9 @@ version: ignore
 components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
+    params:
+      host: opensearch
+      port: 9200
   - name: TextConverter
     type: TextConverter
   - name: Retriever
diff --git a/test/benchmarks/configs/retriever/bm25-weaviate-100k.yml b/test/benchmarks/configs/retriever/bm25-weaviate-100k.yml
index 4b97bb31b9..0244104931 100644
--- a/test/benchmarks/configs/retriever/bm25-weaviate-100k.yml
+++ b/test/benchmarks/configs/retriever/bm25-weaviate-100k.yml
@@ -6,6 +6,8 @@ components:
     params:
       timeout_config: [5, 300]
       batch_size: 5000
+      host: http://weaviate
+      port: 8080
   - name: TextConverter
     type: TextConverter
   - name: Retriever
diff --git a/test/benchmarks/configs/retriever/minilm-elasticsearch-100k.yml b/test/benchmarks/configs/retriever/minilm-elasticsearch-100k.yml
index 8dedcc8cf1..bc74990bfc 100644
--- a/test/benchmarks/configs/retriever/minilm-elasticsearch-100k.yml
+++ b/test/benchmarks/configs/retriever/minilm-elasticsearch-100k.yml
@@ -7,6 +7,8 @@ components:
       batch_size: 5000
       similarity: cosine
       embedding_dim: 384
+      host: elasticsearch
+      port: 9200
   - name: TextConverter
     type: TextConverter
   - name: Retriever
diff --git a/test/benchmarks/configs/retriever/minilm-opensearch-100k.yml b/test/benchmarks/configs/retriever/minilm-opensearch-100k.yml
index 8f55e827b9..24875ebaad 100644
--- a/test/benchmarks/configs/retriever/minilm-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever/minilm-opensearch-100k.yml
@@ -7,6 +7,8 @@ components:
       batch_size: 5000
       similarity: cosine
       embedding_dim: 384
+      host: opensearch
+      port: 9200
   - name: TextConverter
     type: TextConverter
   - name: Retriever
diff --git a/test/benchmarks/configs/retriever/minilm-weaviate-100k.yml b/test/benchmarks/configs/retriever/minilm-weaviate-100k.yml
index e8cf5ad283..9ea0f10ab2 100644
--- a/test/benchmarks/configs/retriever/minilm-weaviate-100k.yml
+++ b/test/benchmarks/configs/retriever/minilm-weaviate-100k.yml
@@ -8,6 +8,8 @@ components:
       similarity: cosine
       embedding_dim: 384
       batch_size: 5000
+      host: http://weaviate
+      port: 8080
   - name: TextConverter
     type: TextConverter
   - name: Retriever
diff --git a/test/benchmarks/configs/retriever/mpnetbase-elasticsearch-100k.yml b/test/benchmarks/configs/retriever/mpnetbase-elasticsearch-100k.yml
index fbedabbe6a..926030ef08 100644
--- a/test/benchmarks/configs/retriever/mpnetbase-elasticsearch-100k.yml
+++ b/test/benchmarks/configs/retriever/mpnetbase-elasticsearch-100k.yml
@@ -7,6 +7,8 @@ components:
       batch_size: 5000
       similarity: dot_product
       embedding_dim: 768
+      host: elasticsearch
+      port: 9200
   - name: TextConverter
     type: TextConverter
   - name: Retriever
diff --git a/test/benchmarks/configs/retriever/mpnetbase-opensearch-100k.yml b/test/benchmarks/configs/retriever/mpnetbase-opensearch-100k.yml
index 4707187aaa..df4290afd1 100644
--- a/test/benchmarks/configs/retriever/mpnetbase-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever/mpnetbase-opensearch-100k.yml
@@ -7,6 +7,8 @@ components:
       batch_size: 5000
       similarity: dot_product
       embedding_dim: 768
+      host: opensearch
+      port: 9200
   - name: TextConverter
     type: TextConverter
   - name: Retriever
diff --git a/test/benchmarks/configs/retriever/mpnetbase-weaviate-100k.yml b/test/benchmarks/configs/retriever/mpnetbase-weaviate-100k.yml
index 6797379c85..60efa34b82 100644
--- a/test/benchmarks/configs/retriever/mpnetbase-weaviate-100k.yml
+++ b/test/benchmarks/configs/retriever/mpnetbase-weaviate-100k.yml
@@ -8,6 +8,8 @@ components:
       similarity: dot_product
       embedding_dim: 768
       batch_size: 5000
+      host: http://weaviate
+      port: 8080
   - name: TextConverter
     type: TextConverter
   - name: Retriever
diff --git a/test/benchmarks/configs/retriever_reader/bm25-debertabase-opensearch-100k.yml b/test/benchmarks/configs/retriever_reader/bm25-debertabase-opensearch-100k.yml
index 77c9d84351..ba70e81685 100644
--- a/test/benchmarks/configs/retriever_reader/bm25-debertabase-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever_reader/bm25-debertabase-opensearch-100k.yml
@@ -4,7 +4,9 @@ components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
     params:
-      batch_size: 5000
+      host: opensearch
+      port: 9200
+      batch_size: 1000
   - name: TextConverter
     type: TextConverter
   - name: Retriever
@@ -16,6 +18,7 @@ components:
     type: FARMReader
     params:
       model_name_or_path: deepset/deberta-v3-base-squad2
+      preprocessing_batch_size: 1000
       top_k: 10
 
 pipelines:
diff --git a/test/benchmarks/configs/retriever_reader/bm25-debertalarge-opensearch-100k.yml b/test/benchmarks/configs/retriever_reader/bm25-debertalarge-opensearch-100k.yml
index d8cac0c8be..d5e02ce0b5 100644
--- a/test/benchmarks/configs/retriever_reader/bm25-debertalarge-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever_reader/bm25-debertalarge-opensearch-100k.yml
@@ -4,7 +4,9 @@ components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
     params:
-      batch_size: 5000
+      host: opensearch
+      port: 9200
+      batch_size: 1000
   - name: TextConverter
     type: TextConverter
   - name: Retriever
@@ -17,6 +19,7 @@ components:
     params:
       model_name_or_path: deepset/deberta-v3-large-squad2
       top_k: 10
+      preprocessing_batch_size: 1000
 
 pipelines:
   - name: indexing
diff --git a/test/benchmarks/configs/retriever_reader/bm25-tinyroberta-opensearch-100k.yml b/test/benchmarks/configs/retriever_reader/bm25-tinyroberta-opensearch-100k.yml
index 9b18d22080..dc90b2749b 100644
--- a/test/benchmarks/configs/retriever_reader/bm25-tinyroberta-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever_reader/bm25-tinyroberta-opensearch-100k.yml
@@ -4,7 +4,9 @@ components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
     params:
-      batch_size: 5000
+      host: opensearch
+      port: 9200
+      batch_size: 1000
   - name: TextConverter
     type: TextConverter
   - name: Retriever
@@ -17,6 +19,7 @@ components:
     params:
       model_name_or_path: deepset/tinyroberta-squad2
       top_k: 10
+      preprocessing_batch_size: 1000
 
 pipelines:
   - name: indexing
diff --git a/test/benchmarks/configs/retriever_reader/minilm-debertabase-opensearch-100k.yml b/test/benchmarks/configs/retriever_reader/minilm-debertabase-opensearch-100k.yml
index 40761aa44d..6401906fef 100644
--- a/test/benchmarks/configs/retriever_reader/minilm-debertabase-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever_reader/minilm-debertabase-opensearch-100k.yml
@@ -4,7 +4,9 @@ components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
     params:
-      batch_size: 5000
+      host: opensearch
+      port: 9200
+      batch_size: 1000
       similarity: cosine
       embedding_dim: 384
   - name: TextConverter
@@ -20,6 +22,7 @@ components:
     params:
       model_name_or_path: deepset/deberta-v3-base-squad2
       top_k: 10
+      preprocessing_batch_size: 1000
 
 pipelines:
   - name: indexing
diff --git a/test/benchmarks/configs/retriever_reader/minilm-debertalarge-opensearch-100k.yml b/test/benchmarks/configs/retriever_reader/minilm-debertalarge-opensearch-100k.yml
index a14f6653b8..0e96acbaaf 100644
--- a/test/benchmarks/configs/retriever_reader/minilm-debertalarge-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever_reader/minilm-debertalarge-opensearch-100k.yml
@@ -4,7 +4,9 @@ components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
     params:
-      batch_size: 5000
+      host: opensearch
+      port: 9200
+      batch_size: 1000
       similarity: cosine
       embedding_dim: 384
   - name: TextConverter
@@ -20,6 +22,7 @@ components:
     params:
       model_name_or_path: deepset/deberta-v3-large-squad2
       top_k: 10
+      preprocessing_batch_size: 1000
 
 pipelines:
   - name: indexing
diff --git a/test/benchmarks/configs/retriever_reader/minilm-tinyroberta-opensearch-100k.yml b/test/benchmarks/configs/retriever_reader/minilm-tinyroberta-opensearch-100k.yml
index 22fc598f3f..f05cf7002a 100644
--- a/test/benchmarks/configs/retriever_reader/minilm-tinyroberta-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever_reader/minilm-tinyroberta-opensearch-100k.yml
@@ -4,7 +4,9 @@ components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
     params:
-      batch_size: 5000
+      host: opensearch
+      port: 9200
+      batch_size: 1000
       similarity: cosine
       embedding_dim: 384
   - name: TextConverter
@@ -20,6 +22,7 @@ components:
     params:
       model_name_or_path: deepset/tinyroberta-squad2
       top_k: 10
+      preprocessing_batch_size: 1000
 
 pipelines:
   - name: indexing
diff --git a/test/benchmarks/configs/retriever_reader/mpnetbase-debertabase-opensearch-100k.yml b/test/benchmarks/configs/retriever_reader/mpnetbase-debertabase-opensearch-100k.yml
index 25a3b84863..3fd8b47023 100644
--- a/test/benchmarks/configs/retriever_reader/mpnetbase-debertabase-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever_reader/mpnetbase-debertabase-opensearch-100k.yml
@@ -4,6 +4,8 @@ components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
     params:
+      host: opensearch
+      port: 9200
       batch_size: 1000
       similarity: dot_product
       embedding_dim: 768
@@ -20,6 +22,7 @@ components:
     params:
       model_name_or_path: deepset/deberta-v3-base-squad2
       top_k: 10
+      preprocessing_batch_size: 1000
 
 pipelines:
   - name: indexing
diff --git a/test/benchmarks/configs/retriever_reader/mpnetbase-debertalarge-opensearch-100k.yml b/test/benchmarks/configs/retriever_reader/mpnetbase-debertalarge-opensearch-100k.yml
index f2b4eb3940..3b961844e5 100644
--- a/test/benchmarks/configs/retriever_reader/mpnetbase-debertalarge-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever_reader/mpnetbase-debertalarge-opensearch-100k.yml
@@ -4,6 +4,8 @@ components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
     params:
+      host: opensearch
+      port: 9200
       batch_size: 1000
       similarity: dot_product
       embedding_dim: 768
@@ -20,6 +22,7 @@ components:
     params:
       model_name_or_path: deepset/deberta-v3-large-squad2
       top_k: 10
+      preprocessing_batch_size: 1000
 
 pipelines:
   - name: indexing
diff --git a/test/benchmarks/configs/retriever_reader/mpnetbase-tinyroberta-opensearch-100k.yml b/test/benchmarks/configs/retriever_reader/mpnetbase-tinyroberta-opensearch-100k.yml
index 94895e870c..4fb270090b 100644
--- a/test/benchmarks/configs/retriever_reader/mpnetbase-tinyroberta-opensearch-100k.yml
+++ b/test/benchmarks/configs/retriever_reader/mpnetbase-tinyroberta-opensearch-100k.yml
@@ -4,6 +4,8 @@ components:
   - name: DocumentStore
     type: OpenSearchDocumentStore
     params:
+      host: opensearch
+      port: 9200
       batch_size: 1000
       similarity: dot_product
       embedding_dim: 768
@@ -20,6 +22,7 @@ components:
     params:
       model_name_or_path: deepset/tinyroberta-squad2
       top_k: 10
+      preprocessing_batch_size: 1000
 
 pipelines:
   - name: indexing
diff --git a/test/benchmarks/datadog/send_metrics.py b/test/benchmarks/datadog/send_metrics.py
index 6d44449ed4..fcaefa1f5c 100644
--- a/test/benchmarks/datadog/send_metrics.py
+++ b/test/benchmarks/datadog/send_metrics.py
@@ -65,10 +65,10 @@ def get_retriever_tag(config: Dict) -> Tag:
                 return RetrieverModelTags.bm25
 
             model = comp["params"]["embedding_model"]
-            if "minilm" in model:
+            if "minilm" in model.lower():
                 return RetrieverModelTags.minilm
 
-            if "mpnet-base" in model:
+            if "mpnet-base" in model.lower():
                 return RetrieverModelTags.mpnetbase
 
     return NoneTag.none
diff --git a/test/benchmarks/retriever.py b/test/benchmarks/retriever.py
index 132ea76409..53d07fb28a 100644
--- a/test/benchmarks/retriever.py
+++ b/test/benchmarks/retriever.py
@@ -29,6 +29,9 @@ def benchmark_retriever(
     querying_results = benchmark_querying(querying_pipeline, eval_set)
 
     results = {"indexing": indexing_results, "querying": querying_results}
+
+    doc_store = indexing_pipeline.get_document_store()
+    doc_store.delete_index(index="document")
     return results
 
 
diff --git a/test/benchmarks/retriever_reader.py b/test/benchmarks/retriever_reader.py
index 7de03b58f7..e63e7305f2 100644
--- a/test/benchmarks/retriever_reader.py
+++ b/test/benchmarks/retriever_reader.py
@@ -30,6 +30,9 @@ def benchmark_retriever_reader(
     querying_results = benchmark_querying(querying_pipeline, eval_set)
 
     results = {"indexing": indexing_results, "querying": querying_results}
+
+    doc_store = indexing_pipeline.get_document_store()
+    doc_store.delete_index(index="document")
     return results
 
 
diff --git a/test/benchmarks/utils.py b/test/benchmarks/utils.py
index 04a4ebb29e..cfbf659458 100644
--- a/test/benchmarks/utils.py
+++ b/test/benchmarks/utils.py
@@ -36,10 +36,11 @@ def prepare_environment(pipeline_config: Dict, benchmark_config: Dict):
         )
 
     # Launch DocumentStore Docker container if needed
-    for comp in pipeline_config["components"]:
-        if comp["type"].endswith("DocumentStore"):
-            launch_document_store(comp["type"], n_docs=n_docs)
-            break
+    if "launch_document_store" in benchmark_config and benchmark_config["launch_document_store"]:
+        for comp in pipeline_config["components"]:
+            if comp["type"].endswith("DocumentStore"):
+                launch_document_store(comp["type"], n_docs=n_docs)
+                break
 
 
 def launch_document_store(document_store: str, n_docs: int = 0):


Code review comments:
path: .github/workflows/cml.yml
text: @author:
Should we change the name of the file to something that makes it clearer that this workflow is about running benchmarks, like `benchmarks.yml`?

@user1:
Yep, let's rename it right before merging.