Repository: redpanda-data/connect
Pull request #3354: configs: rag pipelines
Author: login: rockwotj
Created: 2025-04-18T21:54:52Z | Merged: 2025-04-28T14:37:25Z

In this PR 
* Fix a small bug in the cohere embeddings processor
* Add some example pipelines for RAG using various LLMs (more vector DBs coming soon)
- * I don't think these really need much look over
* Add a qdrant processor for querying from qdrant


Diff:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b9462df968..fa2513ef01 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ All notable changes to this project will be documented in this file.
 
 - (Benthos) Field `client_auth` added to the `socket_server` input. (@filippog)
 - (Benthos) New Bloblang string method `uuid_v5`. (@artemklevtsov)
+- New `qdrant` processor. (@rockwotj)
 
 ### Changed
 
diff --git a/Makefile b/Makefile
index 83088d54d4..9c4afc9a91 100644
--- a/Makefile
+++ b/Makefile
@@ -82,6 +82,7 @@ test: $(APPS)
 	@go test $(GO_FLAGS) -ldflags "$(LD_FLAGS)" -timeout 3m ./...
 	@$(PATHINSTBIN)/redpanda-connect template lint $(TEMPLATE_FILES)
 	@$(PATHINSTBIN)/redpanda-connect test ./config/test/...
+	@$(PATHINSTBIN)/redpanda-connect template lint ./config/rag/templates/...
 
 test-race: $(APPS)
 	@go test $(GO_FLAGS) -ldflags "$(LD_FLAGS)" -timeout 3m -race ./...
diff --git a/config/rag/.gitignore b/config/rag/.gitignore
new file mode 100644
index 0000000000..6a174bf7e0
--- /dev/null
+++ b/config/rag/.gitignore
@@ -0,0 +1,2 @@
+env
+results
diff --git a/config/rag/README.md b/config/rag/README.md
new file mode 100644
index 0000000000..9b3f036e5b
--- /dev/null
+++ b/config/rag/README.md
@@ -0,0 +1,57 @@
+## RAG with Redpanda Connect
+
+This folder hosts a series of RAG pipelines using Redpanda + Redpanda Connect.
+
+We have a series of ingestion pipelines in the `ingestion` directory, these all write
+data we want to ingest into our vector database in topics with the pattern: `rp.ai.rag.*`
+
+Next, there is a matrix of different indexing piplines that use the following sets of options:
+
+Vector Database: `{postgres, elasticsearch, qdrant}`
+
+Embeddings Model: `{openai, cohere}`
+
+These are implemented as resources in the `indexing/resources` directory, then each instance of a pipline
+is created in the `indexing` pipeline.
+
+Lastly, there is a set of retrivial pipelines that mirror each one of our indexing pipelines. These pipelines
+are exposed over HTTP and they can be used to retrive documents/chunks from. These pipelines all live in the
+`retrivial` directory.
+
+Lastly, there is an evalutation framework setup by running a final pipeline after indexing is complete, that exists
+in the `evaluation` directory, which also has some golden files/snapshots of various versions of the pipelines we can
+use to rank/compare the quality of different indexing options.
+
+## Running the pipelines
+
+First bootup the required services:
+
+```
+docker compose up -d
+```
+
+Then you need to set the environment variables, specifically the `*_API_KEY` ones.
+
+```
+cp env.sample env && vim env
+```
+
+Ingest and index all the documents for our knowledge graph
+
+```
+rpk connect streams -t 'templates/*.yaml' -e env indexing/* ingestion/*
+```
+
+Once everything has been ingested, then indexed, we can stop the pipeline and startup the retrival pipelines
+
+```
+rpk connect streams -t 'templates/*.yaml' -e env retrivial/*
+```
+
+Now concurrently run the eval pipeline to run the evaluations:
+
+```
+rpk connect run -e env eval.yaml
+```
+
+Now there should be a file in ./results with all the resulting fetched documents
diff --git a/config/rag/docker-compose.yml b/config/rag/docker-compose.yml
new file mode 100644
index 0000000000..2c34ad86ee
--- /dev/null
+++ b/config/rag/docker-compose.yml
@@ -0,0 +1,102 @@
+services:
+  ### VECTOR DATABASES ###
+  postgres:
+    image: pgvector/pgvector:0.8.0-pg17
+    environment:
+      POSTGRES_DB: mydatabase
+      POSTGRES_USER: myuser
+      POSTGRES_PASSWORD: mypassword
+    ports:
+      - "5432:5432"
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    networks:
+      - rag_network
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:9.0.0
+    environment:
+      - discovery.type=single-node
+      - ES_JAVA_OPTS=-Xms1G -Xmx4G
+    ports:
+      - "9200:9200"
+      - "9300:9300"
+    volumes:
+      - esdata:/usr/share/elasticsearch/data
+    networks:
+      - rag_network
+  qdrant:
+    image: qdrant/qdrant:latest
+    ports:
+      - "6333:6333"
+    volumes:
+      - qdrant_storage:/qdrant/storage
+    networks:
+      - rag_network
+  ### TRUSTY INGESTION ENGINE ###
+  redpanda:
+    image: redpandadata/redpanda:v25.1.1
+    command:
+      - redpanda
+      - start
+      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
+      # Address the broker advertises to clients that connect to the Kafka API.
+      # Use the internal addresses to connect to the Redpanda brokers'
+      # from inside the same Docker network.
+      # Use the external addresses to connect to the Redpanda brokers'
+      # from outside the Docker network.
+      - --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
+      - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
+      # Address the broker advertises to clients that connect to the HTTP Proxy.
+      - --advertise-pandaproxy-addr internal://redpanda:8082,external://localhost:18082
+      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
+      # Redpanda brokers use the RPC API to communicate with each other internally.
+      - --rpc-addr redpanda:33145
+      - --advertise-rpc-addr redpanda:33145
+      # Mode dev-container uses well-known configuration properties for development in containers.
+      - --mode dev-container
+      # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
+      - --smp 1
+      - --default-log-level=info
+      - --set redpanda.auto_create_topics_enabled=true
+    ports:
+      - 18081:18081
+      - 18082:18082
+      - 19092:19092
+      - 19644:9644
+    volumes:
+      - redpandadata:/var/lib/redpanda/data
+    networks:
+      - rag_network
+  redpanda-console:
+    image: redpandadata/console:v3.0.0
+    entrypoint: /bin/sh
+    command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
+    environment:
+      CONFIG_FILEPATH: /tmp/config.yml
+      CONSOLE_CONFIG_FILE: |
+        kafka:
+          brokers: ["redpanda:9092"]
+        schemaRegistry:
+          enabled: true
+          urls: ["http://redpanda:8081"]
+        redpanda:
+          adminApi:
+            enabled: true
+            urls: ["http://redpanda:9644"]
+    ports:
+      - 8080:8080
+    depends_on:
+      - redpanda
+    networks:
+      - rag_network
+
+volumes:
+  pgdata: null
+  redpandadata: null
+  esdata: null
+  qdrant_storage: null
+
+networks:
+  rag_network:
+    driver: bridge
+
diff --git a/config/rag/env.sample b/config/rag/env.sample
new file mode 100644
index 0000000000..7bd74946a9
--- /dev/null
+++ b/config/rag/env.sample
@@ -0,0 +1,11 @@
+# These environment variables are assuming the local docker compose setup is running
+REDPANDA_BROKERS="localhost:19092"
+REDPANDA_USER=""
+REDPANDA_PASS=""
+SASL_MECHANISM=""
+INDEXING_CONSUMER="rp.ai.rag.indexing.v1"
+POSTGRES_DSN="postgresql://myuser:mypassword@localhost:5432/mydatabase?sslmode=disable"
+POSTGRES_TABLE="rp_ai_rag"
+OPENAI_API_KEY=''
+COHERE_API_KEY=''
+GCP_PROJECT=''
diff --git a/config/rag/eval.yaml b/config/rag/eval.yaml
new file mode 100644
index 0000000000..3265b0f954
--- /dev/null
+++ b/config/rag/eval.yaml
@@ -0,0 +1,139 @@
+input:
+  generate:
+    count: 1
+    mapping: |
+      root = [
+        "I just deployed Redpanda on Kubernetes. What do I need to do to run it in production?",
+        "How do you join two arrays in bloblang?",
+        "What is the rpk command to describe the acls of a user?",
+        # Spoiler alert, it's that redis_hash output takes interpolated strings not blobl
+        """
+        What is wrong with this?
+
+        ```yaml
+        input: 
+          label: 'mqtt_topic'
+          mqtt:
+            urls: ['${MQTT_BROKER_HOST:${MQTT_BROKER_HOST}}']
+            topics: ['mytopics/+']
+            client_id: redpanda-connect-mqtt
+            qos: 0
+        pipeline:
+          processors: 
+            # If the topic is 'mytopics/connected', drop the message 
+            - mapping: |
+                root = if meta("mqtt_topic") == "mytopics/connected" { deleted() } else { this } 
+            # Extract fields from payload to match desired structure
+            - mapping: |
+                root = { 
+                  "A": this.a,
+                  "B": this.b.or(''),
+                  "C": this.c.or(''),
+                  "D": this.d.or(''),
+                }
+            # Set Redis key as 'foo_<A>'
+            - mapping: |
+                root = this
+                meta redis_key = "foo_" + this.A
+          output:
+            redis_hash:
+              url: redis://redis:6379
+              key: ${!meta("redis_key")} 
+              fields:
+                a: this.A
+                b: this.B
+                c: this.C
+                d: this.D
+          ```
+        """,
+        "What is the best thing about Redpanda?",
+        "Docker Compose for Redpanda",
+        "Give me an example of using the `http` processor within a `for_each` processor.",
+        "How do I enable Tiered Storage?",
+      ]
+pipeline:
+  processors:
+    - unarchive:
+        format: json_array
+    - mapping: 'root.query = this'
+    - workflow:
+        meta_path: workflow_result
+        branches:
+          cohere_pgvector:
+            processors:
+              - http:
+                  url: http://localhost:4195/cohere_pgvector/query
+            result_map: 'root.cohere_pgvector = this'
+          openai_pgvector:
+            processors:
+              - http:
+                  url: http://localhost:4195/openai_pgvector/query
+            result_map: 'root.openai_pgvector = this'
+          ollama_pgvector:
+            processors:
+              - http:
+                  url: http://localhost:4195/ollama_pgvector/query
+            result_map: 'root.ollama_pgvector = this'
+          score:
+            request_map: |
+              root.query = this.query
+              root.cohere_pgvector = this.cohere_pgvector
+              root.openai_pgvector = this.openai_pgvector
+              root.ollama_pgvector = this.ollama_pgvector
+            processors:
+              - mapping: |
+                  # This computes the combination of all retrival results
+                  let results = this.without("query").keys()
+                  root.processed = []
+                  root.unprocessed = range(0, $results.length()).fold([], i -> (
+                    i.tally.concat($results.slice(i.value+1).fold([], b -> (
+                      b.tally.append({
+                        "q": this.query,
+                        $results.index(i.value): this.get($results.index(i.value)), 
+                        b.value: this.get(b.value),
+                      })
+                    )))
+                  ))
+              # Loop over every combination of values and have gemini score them.
+              - while:
+                  check: 'this.unprocessed.length() > 0'
+                  processors:
+                  - branch:
+                      request_map: |
+                        root = this.unprocessed.0
+                      result_map: |
+                        root.processed = root.processed.append(this)
+                      processors:
+                      - gcp_vertex_ai_chat:
+                          project: ${GCP_PROJECT}
+                          model: gemini-2.5-pro-preview-03-25
+                          prompt: |
+                            Below is a YAML document with 3 keys, one is a query from a user,
+                            and two other keys contain documents that were retrived based on
+                            the user's query. Please rate which set of documents are better
+                            suited to give context to answer the user's question.
+
+                            ${!this.format_yaml()}
+
+                            Please respond in JSON with this format:
+                            {
+                              "winner": "<key in yaml doc that had better results>",
+                              "reasoning": "<the reason why it was better>"
+                            }
+                          response_format: json
+                  - mutation: 'root.unprocessed = this.unprocessed.slice(1)'
+            result_map: 'root.results = this.processed'
+    - mapping: |
+        root = this
+        # TODO: check the result
+        root.workflow_result = deleted()
+    - archive:
+        format: json_array
+    - mapping: |
+        root = this.format_yaml()
+output:
+  file:
+    path: results/${! now().ts_format("2006-01-02", "UTC") }.yaml
+
+http:
+  enabled: false
diff --git a/config/rag/indexing/cohere_pgvector.yaml b/config/rag/indexing/cohere_pgvector.yaml
new file mode 100644
index 0000000000..e23e4af872
--- /dev/null
+++ b/config/rag/indexing/cohere_pgvector.yaml
@@ -0,0 +1,50 @@
+input:
+  rag_topics:
+    seed_brokers: "${REDPANDA_BROKERS}"
+    consumer_group: "${INDEXING_CONSUMER}.cohere"
+    user: "${REDPANDA_USER}"
+    password: "${REDPANDA_PASS}"
+    batching: 
+      count: 100
+      period: 10s
+pipeline:
+  threads: 8
+  processors:
+    - try:
+        - mutation: |
+            if !@mime_type.or("text/plain").contains("text") {
+              root = deleted()
+            }
+            if (@kafka_key.not_empty() | null) == null {
+              meta kafka_key = content().hash("sha256").encode("hex")
+            }
+        - text_chunker:
+            strategy: recursive_character
+        - group_by_value:
+            value: ${! @kafka_key }
+        - mapping: |
+            root.document = content().string()
+            root.chunk_id = batch_index()
+        - branch:
+            request_map: root = this.document
+            processors:
+              - cohere_embed:
+                  api_key: ${COHERE_API_KEY}
+                  input_type: search_document
+                  dimensions: 1536
+            result_map: root.embeddings = this
+        - archive:
+            format: json_array
+    - catch:
+        - log:
+            level: ERROR
+            message: "ERROR: ${!error()}"
+        - mapping: 'root = deleted()'
+output:
+  fallback:
+  - reject_errored:
+      pgvector:
+        dsn: "${POSTGRES_DSN}"
+        table: "${POSTGRES_TABLE}_cohere"
+        dimensions: 1536
+  - reject: "error ${!@fallback_error}, processing document: ${!content().string()}"
diff --git a/config/rag/indexing/ollama_pgvector.yaml b/config/rag/indexing/ollama_pgvector.yaml
new file mode 100644
index 0000000000..4278503c23
--- /dev/null
+++ b/config/rag/indexing/ollama_pgvector.yaml
@@ -0,0 +1,44 @@
+input:
+  rag_topics:
+    seed_brokers: "${REDPANDA_BROKERS}"
+    consumer_group: "${INDEXING_CONSUMER}.ollama"
+    user: "${REDPANDA_USER}"
+    password: "${REDPANDA_PASS}"
+    batching: 
+      count: 100
+      period: 10s
+pipeline:
+  threads: 8
+  processors:
+    - try:
+        - mutation: |
+            if !@mime_type.or("text/plain").contains("text") {
+              root = deleted()
+            }
+            if (@kafka_key.not_empty() | null) == null {
+              meta kafka_key = content().hash("sha256").encode("hex")
+            }
+        - text_chunker:
+            strategy: recursive_character
+        - group_by_value:
+            value: ${! @kafka_key }
+        - mapping: |
+            root.document = content().string()
+            root.chunk_id = batch_index()
+        - label: embeddings
+          branch:
+            request_map: root = this.document
+            processors:
+              - ollama_embed:
+                  input_type: search_document
+            result_map: root.embeddings = this
+        - archive:
+            format: json_array
+output:
+  fallback:
+  - reject_errored:
+      pgvector:
+        dsn: "${POSTGRES_DSN}"
+        table: "${POSTGRES_TABLE}_ollama"
+        dimensions: 768
+  - reject: "error ${!@fallback_error}, processing document: ${!content().string()}"
diff --git a/config/rag/indexing/openai_pgvector.yaml b/config/rag/indexing/openai_pgvector.yaml
new file mode 100644
index 0000000000..496ac556e0
--- /dev/null
+++ b/config/rag/indexing/openai_pgvector.yaml
@@ -0,0 +1,45 @@
+input:
+  rag_topics:
+    seed_brokers: "${REDPANDA_BROKERS}"
+    consumer_group: "${INDEXING_CONSUMER}.openai"
+    user: "${REDPANDA_USER}"
+    password: "${REDPANDA_PASS}"
+    batching: 
+      count: 100
+      period: 10s
+pipeline:
+  threads: 8
+  processors:
+    - try:
+        - mutation: |
+            if !@mime_type.or("text/plain").contains("text") {
+              root = deleted()
+            }
+            if (@kafka_key.not_empty() | null) == null {
+              meta kafka_key = content().hash("sha256").encode("hex")
+            }
+        - text_chunker:
+            strategy: recursive_character
+        - group_by_value:
+            value: ${! @kafka_key }
+        - mapping: |
+            root.document = content().string()
+            root.chunk_id = batch_index()
+        - label: embeddings
+          branch:
+            request_map: root = this.document
+            processors:
+              - oai_embed:
+                  api_key: ${OPENAI_API_KEY}
+                  dimensions: 768
+            result_map: root.embeddings = this
+        - archive:
+            format: json_array
+output:
+  fallback:
+  - reject_errored:
+      pgvector:
+        dsn: "${POSTGRES_DSN}"
+        table: "${POSTGRES_TABLE}_openai"
+        dimensions: 768
+  - reject: "error ${!@fallback_error}, processing document: ${!content().string()}"
diff --git a/config/rag/ingestion/redpanda-docs.yaml b/config/rag/ingestion/redpanda-docs.yaml
new file mode 100644
index 0000000000..7593f3af01
--- /dev/null
+++ b/config/rag/ingestion/redpanda-docs.yaml
@@ -0,0 +1,28 @@
+input:
+  git:
+    repository_url: https://github.com/redpanda-data/docs.git
+    branch: main
+    poll_interval: "10s"
+    include_patterns:
+      - 'modules/**/*.adoc'
+    exclude_patterns:
+      - 'modules/ROOT/**'
+    max_file_size: 1048576
+
+pipeline:
+  processors:
+    - mapping: |
+        meta = @.map_each_key(key -> key.trim_prefix("git_"))
+        root = if @is_binary {
+          deleted()
+        }
+output:
+  kafka_franz:
+    seed_brokers: ["${REDPANDA_BROKERS}"]
+    sasl: []
+    tls:
+      enabled: false
+    topic: "rp.ai.rag.rpdocs"
+    key: ${!meta("git_file_path")}
+    metadata:
+      include_patterns: [".*"]
diff --git a/config/rag/retrieval/cohere_pgvector.yaml b/config/rag/retrieval/cohere_pgvector.yaml
new file mode 100644
index 0000000000..6487be941e
--- /dev/null
+++ b/config/rag/retrieval/cohere_pgvector.yaml
@@ -0,0 +1,40 @@
+input:
+  http_server:
+    path: /query
+    allowed_verbs:
+      - POST
+    sync_response:
+      status: "${!this.status.or(200)}"
+    timeout: 60s
+pipeline:
+  processors:
+    - try:
+      - json_schema:
+          schema: |
+            {
+              "$schema": "http://json-schema.org/draft-07/schema#",
+              "title": "HTTP Request schema",
+              "type": "object",
+              "properties": {
+                "query": {
+                  "type": "string"
+                }
+              },
+              "required": [
+                "query"
+              ]
+            }
+      - cohere_embed:
+          api_key: ${COHERE_API_KEY}
+          input_type: search_query
+          dimensions: 1536
+      - pgvector:
+          dsn: ${POSTGRES_DSN}
+          table: ${POSTGRES_TABLE}_cohere
+output:
+  processors:
+    - mutation: |
+        if errored() {
+          root = {"status": 500, "error": error()}
+        }
+  sync_response: {}
diff --git a/config/rag/retrieval/ollama_pgvector.yaml b/config/rag/retrieval/ollama_pgvector.yaml
new file mode 100644
index 0000000000..e4e7f610d6
--- /dev/null
+++ b/config/rag/retrieval/ollama_pgvector.yaml
@@ -0,0 +1,38 @@
+input:
+  http_server:
+    path: /query
+    allowed_verbs:
+      - POST
+    sync_response:
+      status: "${!this.status.or(200)}"
+    timeout: 60s
+pipeline:
+  processors:
+    - try:
+      - json_schema:
+          schema: |
+            {
+              "$schema": "http://json-schema.org/draft-07/schema#",
+              "title": "HTTP Request schema",
+              "type": "object",
+              "properties": {
+                "query": {
+                  "type": "string"
+                }
+              },
+              "required": [
+                "query"
+              ]
+            }
+      - ollama_embed:
+          input_type: search_query
+      - pgvector:
+          dsn: ${POSTGRES_DSN}
+          table: ${POSTGRES_TABLE}_ollama
+output:
+  processors:
+    - mutation: |
+        if errored() {
+          root = {"status": 500, "error": error()}
+        }
+  sync_response: {}
diff --git a/config/rag/retrieval/openai_pgvector.yaml b/config/rag/retrieval/openai_pgvector.yaml
new file mode 100644
index 0000000000..d5b78c530a
--- /dev/null
+++ b/config/rag/retrieval/openai_pgvector.yaml
@@ -0,0 +1,39 @@
+input:
+  http_server:
+    path: /query
+    allowed_verbs:
+      - POST
+    sync_response:
+      status: "${!this.status.or(200)}"
+    timeout: 60s
+pipeline:
+  processors:
+    - try:
+      - json_schema:
+          schema: |
+            {
+              "$schema": "http://json-schema.org/draft-07/schema#",
+              "title": "HTTP Request schema",
+              "type": "object",
+              "properties": {
+                "query": {
+                  "type": "string"
+                }
+              },
+              "required": [
+                "query"
+              ]
+            }
+      - oai_embed:
+          api_key: ${OPENAI_API_KEY}
+          dimensions: 768
+      - pgvector:
+          dsn: ${POSTGRES_DSN}
+          table: ${POSTGRES_TABLE}_openai
+output:
+  processors:
+    - mutation: |
+        if errored() {
+          root = {"status": 500, "error": error()}
+        }
+  sync_response: {}
diff --git a/config/rag/rpk.profile.yaml b/config/rag/rpk.profile.yaml
new file mode 100644
index 0000000000..551efd389d
--- /dev/null
+++ b/config/rag/rpk.profile.yaml
@@ -0,0 +1,14 @@
+name: docker-compose
+description: ""
+prompt: ""
+from_cloud: false
+kafka_api:
+    brokers:
+        - localhost:19092
+admin_api:
+    addresses:
+        - localhost:19644
+schema_registry:
+    addresses:
+        - localhost:18081
+
diff --git a/config/rag/templates/cohere_embeddings.yaml b/config/rag/templates/cohere_embeddings.yaml
new file mode 100644
index 0000000000..6a2e0c826c
--- /dev/null
+++ b/config/rag/templates/cohere_embeddings.yaml
@@ -0,0 +1,31 @@
+name: cohere_embed
+type: processor
+
+fields:
+  - name: api_key
+    type: string
+  - name: input_type
+    type: string
+  - name: dimensions
+    type: int
+
+mapping: |
+  root.cohere_embeddings = {
+    "api_key": this.api_key,
+    "model": "embed-v4.0",
+    "input_type": this.input_type,
+    "dimensions": this.dimensions
+  }
+
+tests:
+  - name: cohere_embeddings test
+    config: 
+      api_key: "sk-foo"
+      input_type: "search_document"
+      dimensions: 1536
+    expected:
+      cohere_embeddings:
+        api_key: sk-foo
+        model: embed-v4.0
+        input_type: search_document
+        dimensions: 1536
diff --git a/config/rag/templates/ollama_embeddings.yaml b/config/rag/templates/ollama_embeddings.yaml
new file mode 100644
index 0000000000..901575b810
--- /dev/null
+++ b/config/rag/templates/ollama_embeddings.yaml
@@ -0,0 +1,21 @@
+name: ollama_embed
+type: processor
+
+fields:
+  - name: input_type
+    type: string
+
+mapping: |
+  root.ollama_embeddings = {
+    "model": "nomic-embed-text",
+    "text": "%s: ${!content().string()}".format(this.input_type),
+  }
+
+tests:
+  - name: ollama_embeddings test
+    config: 
+      input_type: "search_document"
+    expected:
+      ollama_embeddings:
+        model: nomic-embed-text
+        text: "search_document: ${!content().string()}"
diff --git a/config/rag/templates/openai_embeddings.yaml b/config/rag/templates/openai_embeddings.yaml
new file mode 100644
index 0000000000..0fdc6e1265
--- /dev/null
+++ b/config/rag/templates/openai_embeddings.yaml
@@ -0,0 +1,26 @@
+name: oai_embed
+type: processor
+
+fields:
+  - name: api_key
+    type: string
+  - name: dimensions
+    type: int
+
+mapping: |
+  root.openai_embeddings = {
+    "api_key": this.api_key,
+    "model": "text-embedding-3-small",
+    "dimensions": this.dimensions,
+  }
+
+tests:
+  - name: openai_embeddings test
+    config: 
+      api_key: "sk-foo"
+      dimensions: 768
+    expected:
+      openai_embeddings:
+        api_key: sk-foo
+        model: text-embedding-3-small
+        dimensions: 768
diff --git a/config/rag/templates/pgvector_output.yaml b/config/rag/templates/pgvector_output.yaml
new file mode 100644
index 0000000000..caf267fe91
--- /dev/null
+++ b/config/rag/templates/pgvector_output.yaml
@@ -0,0 +1,81 @@
+name: pgvector
+type: output
+
+fields:
+  - name: table
+    type: string
+  - name: dsn
+    type: string
+  - name: max_in_flight
+    type: int
+    default: 8
+  - name: dimensions
+    type: int
+  - name: batching
+    type: unknown
+    default:
+      count: 100
+      period: 10s
+
+mapping: |
+  root.sql_raw = {
+    "driver": "postgres",
+    "dsn": this.dsn,
+    "init_statement": """
+      CREATE EXTENSION IF NOT EXISTS vector;
+      CREATE TABLE IF NOT EXISTS %s (
+        topic text,
+        key text,
+        chunk_id integer,
+        document text,
+        embeddings vector(%d),
+        PRIMARY KEY(topic, key, chunk_id)
+      );""".format(this.table, this.dimensions).trim("\n"),
+    "queries": [
+      { 
+        "query": "DELETE FROM %s WHERE (topic, key) = ($1, $2)".format(this.table),
+        "args_mapping": "root = [ @kafka_topic, @kafka_key ]",
+      },
+      {
+        "query": """
+          INSERT INTO %s (topic, key, chunk_id, document, embeddings) SELECT $1, $2, (chunk->>'chunk_id')::INT, chunk->>'document', (chunk->>'embeddings')::text::vector FROM jsonb_array_elements($3) AS chunk
+        """.format(this.table).trim(),
+        "args_mapping": "root = [@kafka_topic, @kafka_key, this.format_json(no_indent: true, escape_html: false)]",
+        },
+      ],
+    "max_in_flight": this.max_in_flight,
+    "batching": this.batching,
+  }
+
+tests:
+  - name: pgvector output test
+    config: 
+      dsn: "postgres://localhost"
+      table: "foo"
+      dimensions: 768
+    expected:
+      sql_raw: 
+        driver: "postgres"
+        dsn: "postgres://localhost"
+        init_statement: |-2
+              CREATE EXTENSION IF NOT EXISTS vector;
+              CREATE TABLE IF NOT EXISTS foo (
+                topic text,
+                key text,
+                chunk_id integer,
+                document text,
+                embeddings vector(768),
+                PRIMARY KEY(topic, key, chunk_id)
+              );
+        queries:
+          - args_mapping: "root = [ @kafka_topic, @kafka_key ]"
+            query: "DELETE FROM foo WHERE (topic, key) = ($1, $2)"
+          - args_mapping: 'root = [@kafka_topic, @kafka_key, this.format_json(no_indent: true, escape_html: false)]'
+            query: >-
+              INSERT INTO foo (topic, key, chunk_id, document, embeddings)
+              SELECT $1, $2, (chunk->>'chunk_id')::INT, chunk->>'document', (chunk->>'embeddings')::text::vector
+              FROM jsonb_array_elements($3) AS chunk
+        max_in_flight: 8
+        batching:
+            count: 100
+            period: 10s
diff --git a/config/rag/templates/pgvector_query.yaml b/config/rag/templates/pgvector_query.yaml
new file mode 100644
index 0000000000..4d59f4b44f
--- /dev/null
+++ b/config/rag/templates/pgvector_query.yaml
@@ -0,0 +1,29 @@
+name: pgvector
+type: processor
+
+fields:
+  - name: table
+    type: string
+  - name: dsn
+    type: string
+  - name: limit
+    type: int
+    default: 3
+
+mapping: |
+  root.sql_raw = {
+    "driver": "postgres",
+    "dsn": this.dsn,
+    "query": """
+      SELECT (
+        SELECT STRING_AGG(t2.document, '' ORDER BY chunk_id ASC) 
+        FROM %s t2 
+        WHERE t1.key = t2.key AND t1.topic = t2.topic
+        GROUP BY key
+      ) AS document, key, topic
+      FROM %s t1
+      ORDER BY embeddings <-> $1
+      LIMIT %d
+    """.format(this.table, this.table, this.limit),
+    "args_mapping": "[ this.vector() ]",
+  }
diff --git a/config/rag/templates/redpanda.yaml b/config/rag/templates/redpanda.yaml
new file mode 100644
index 0000000000..3a9ab5e79c
--- /dev/null
+++ b/config/rag/templates/redpanda.yaml
@@ -0,0 +1,79 @@
+name: rag_topics
+type: input 
+fields:
+  - name: seed_brokers
+    type: string
+  - name: user
+    type: string
+    default: ""
+  - name: password
+    type: string
+    default: ""
+  - name: consumer_group
+    type: string
+  - name: batching
+    type: unknown 
+    default: {}
+mapping: |
+    root.kafka_franz = {
+      "seed_brokers": [this.seed_brokers],
+      "sasl": if this.user != "" { 
+          [{
+            "username": this.user,
+            "password": this.password,
+            "mechanism": "SCRAM-SHA-256",
+          }]
+        } else {
+          []
+        },
+      "tls": {"enabled": this.user != ""},
+      "topics": ["^rp\\.ai\\.rag\\..*$"],
+      "regexp_topics": true,
+      "consumer_group": this.consumer_group,
+      "batching": this.batching,
+    }
+
+tests:
+  - name: no auth test
+    config:
+      seed_brokers: "localhost:9092"
+      consumer_group: "foo_cg"
+      batching:
+        count: 100
+        period: 10s
+    expected:
+      kafka_franz:
+        seed_brokers: [localhost:9092]
+        sasl: []
+        tls:
+          enabled: false
+        topics: ['^rp\.ai\.rag\..*$']
+        consumer_group: foo_cg
+        regexp_topics: true
+        batching:
+          count: 100
+          period: 10s
+  - name: yes auth test
+    config:
+      seed_brokers: "localhost:9092"
+      consumer_group: "foo_cg"
+      batching:
+        count: 100
+        period: 10s
+      user: me
+      password: 12345
+    expected:
+      kafka_franz:
+        seed_brokers: [localhost:9092]
+        sasl:
+          - username: me
+            password: "12345"
+            mechanism: SCRAM-SHA-256
+        tls:
+          enabled: true
+        topics: ['^rp\.ai\.rag\..*$']
+        consumer_group: foo_cg
+        regexp_topics: true
+        batching:
+          count: 100
+          period: 10s
diff --git a/docs/modules/components/pages/processors/qdrant.adoc b/docs/modules/components/pages/processors/qdrant.adoc
new file mode 100644
index 0000000000..021a4be645
--- /dev/null
+++ b/docs/modules/components/pages/processors/qdrant.adoc
@@ -0,0 +1,374 @@
+= qdrant
+:type: processor
+:status: experimental
+:categories: ["AI"]
+
+
+
+////
+     THIS FILE IS AUTOGENERATED!
+
+     To make changes, edit the corresponding source file under:
+
+     https://github.com/redpanda-data/connect/tree/main/internal/impl/<provider>.
+
+     And:
+
+     https://github.com/redpanda-data/connect/tree/main/cmd/tools/docs_gen/templates/plugin.adoc.tmpl
+////
+
+// © 2024 Redpanda Data Inc.
+
+
+component_type_dropdown::[]
+
+
+Query items within a https://qdrant.tech/[Qdrant^] collection.
+
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Common config fields, showing default values
+label: ""
+qdrant:
+  grpc_host: localhost:6334 # No default (required)
+  api_token: ""
+  collection_name: "" # No default (required)
+  vector_mapping: root = [1.2, 0.5, 0.76] # No default (required)
+  filter: | # No default (optional)
+    root.must = [
+    	{"has_id":{"has_id":[{"num": 8}, { "uuid":"1234-5678-90ab-cdef" }]}},
+    	{"field":{"key": "city", "match": {"text": "London"}}},
+    ]
+  payload_fields: []
+  payload_filter: include
+  limit: 10
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All config fields, showing default values
+label: ""
+qdrant:
+  grpc_host: localhost:6334 # No default (required)
+  api_token: ""
+  tls:
+    enabled: false
+    skip_cert_verify: false
+    enable_renegotiation: false
+    root_cas: ""
+    root_cas_file: ""
+    client_certs: []
+  collection_name: "" # No default (required)
+  vector_mapping: root = [1.2, 0.5, 0.76] # No default (required)
+  filter: | # No default (optional)
+    root.must = [
+    	{"has_id":{"has_id":[{"num": 8}, { "uuid":"1234-5678-90ab-cdef" }]}},
+    	{"field":{"key": "city", "match": {"text": "London"}}},
+    ]
+  payload_fields: []
+  payload_filter: include
+  limit: 10
+```
+
+--
+======
+
+== Fields
+
+=== `grpc_host`
+
+The gRPC host of the Qdrant server.
+
+
+*Type*: `string`
+
+
+```yml
+# Examples
+
+grpc_host: localhost:6334
+
+grpc_host: xyz-example.eu-central.aws.cloud.qdrant.io:6334
+```
+
+=== `api_token`
+
+The Qdrant API token for authentication. Defaults to an empty string.
+[CAUTION]
+====
+This field contains sensitive information that usually shouldn't be added to a config directly, read our xref:configuration:secrets.adoc[secrets page for more info].
+====
+
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `tls`
+
+TLS(HTTPS) config to use when connecting
+
+
+*Type*: `object`
+
+
+=== `tls.enabled`
+
+Whether custom TLS settings are enabled.
+
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `tls.skip_cert_verify`
+
+Whether to skip server side certificate verification.
+
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `tls.enable_renegotiation`
+
+Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you're seeing the error message `local error: tls: no renegotiation`.
+
+
+*Type*: `bool`
+
+*Default*: `false`
+Requires version 3.45.0 or newer
+
+=== `tls.root_cas`
+
+An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate.
+[CAUTION]
+====
+This field contains sensitive information that usually shouldn't be added to a config directly, read our xref:configuration:secrets.adoc[secrets page for more info].
+====
+
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+root_cas: |-
+  -----BEGIN CERTIFICATE-----
+  ...
+  -----END CERTIFICATE-----
+```
+
+=== `tls.root_cas_file`
+
+An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate.
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+root_cas_file: ./root_cas.pem
+```
+
+=== `tls.client_certs`
+
+A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both.
+
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+client_certs:
+  - cert: foo
+    key: bar
+
+client_certs:
+  - cert_file: ./example.pem
+    key_file: ./example.key
+```
+
+=== `tls.client_certs[].cert`
+
+A plain text certificate to use.
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `tls.client_certs[].key`
+
+A plain text certificate key to use.
+[CAUTION]
+====
+This field contains sensitive information that usually shouldn't be added to a config directly, read our xref:configuration:secrets.adoc[secrets page for more info].
+====
+
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `tls.client_certs[].cert_file`
+
+The path of a certificate to use.
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `tls.client_certs[].key_file`
+
+The path of a certificate key to use.
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `tls.client_certs[].password`
+
+A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format.
+
+Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext.
+[CAUTION]
+====
+This field contains sensitive information that usually shouldn't be added to a config directly, read our xref:configuration:secrets.adoc[secrets page for more info].
+====
+
+
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+password: foo
+
+password: ${KEY_PASSWORD}
+```
+
+=== `collection_name`
+
+The name of the collection in Qdrant.
+This field supports xref:configuration:interpolation.adoc#bloblang-queries[interpolation functions].
+
+
+*Type*: `string`
+
+
+=== `vector_mapping`
+
+The mapping to extract the search vector from the document.
+
+
+*Type*: `string`
+
+
+```yml
+# Examples
+
+vector_mapping: root = [1.2, 0.5, 0.76]
+
+vector_mapping: root = this.vector
+
+vector_mapping: root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]
+
+vector_mapping: 'root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}}'
+
+vector_mapping: 'root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]}'
+
+vector_mapping: 'root = {"some_dense": [0.352,0.532,0.532,0.234]}'
+```
+
+=== `filter`
+
+Additional filtering to perform on the results. The mapping should return a valid filter (using the proto3 encoded form) in qdrant. See the https://qdrant.tech/documentation/concepts/filtering/[^Qdrant documentation] for examples.
+
+
+*Type*: `string`
+
+
+```yml
+# Examples
+
+filter: |2
+  root.must = [
+  	{"has_id":{"has_id":[{"num": 8}, { "uuid":"1234-5678-90ab-cdef" }]}},
+  	{"field":{"key": "city", "match": {"text": "London"}}},
+  ]
+
+filter: |2
+  root.must = [
+  	{"field":{"key": "city", "match": {"text": "London"}}},
+  ]
+  root.must_not = [
+  	{"field":{"color": "city", "match": {"text": "red"}}},
+  ]
+```
+
+=== `payload_fields`
+
+The fields to include or exclude in returned result based on the `payload_filter`.
+
+
+*Type*: `array`
+
+*Default*: `[]`
+
+=== `payload_filter`
+
+The way the fields in `payload_fields` are filtered in the result.
+
+
+*Type*: `string`
+
+*Default*: `"include"`
+
+|===
+| Option | Summary
+
+| `exclude`
+| Exclude the payload fields specified in `payload_fields`.
+| `include`
+| Include the payload fields specified in `payload_fields`.
+
+|===
+
+=== `limit`
+
+The maximum number of points to return.
+
+
+*Type*: `int`
+
+*Default*: `10`
+
+
diff --git a/internal/impl/cohere/embeddings_processor.go b/internal/impl/cohere/embeddings_processor.go
index c450e56b8f..36f020ebd7 100644
--- a/internal/impl/cohere/embeddings_processor.go
+++ b/internal/impl/cohere/embeddings_processor.go
@@ -145,6 +145,7 @@ func (p *embeddingsProcessor) Process(ctx context.Context, msg *service.Message)
 	body.Model = p.model
 	body.InputType = p.inputType
 	body.OutputDimension = p.dimensions
+	body.EmbeddingTypes = []cohere.EmbeddingType{cohere.EmbeddingTypeFloat}
 	if p.text != nil {
 		s, err := msg.BloblangQuery(p.text)
 		if err != nil {
diff --git a/internal/impl/qdrant/client.go b/internal/impl/qdrant/client.go
index c6951cf5c2..93db52de25 100644
--- a/internal/impl/qdrant/client.go
+++ b/internal/impl/qdrant/client.go
@@ -82,6 +82,30 @@ func (c *qdrantClient) Upsert(ctx context.Context, collectionName string, points
 	return err
 }
 
+func (c *qdrantClient) Query(
+	ctx context.Context,
+	collectionName string,
+	vectorName *string,
+	vector *qdrant.VectorInput,
+	payload *qdrant.WithPayloadSelector,
+	filter *qdrant.Filter,
+	limit uint64,
+) ([]*qdrant.ScoredPoint, error) {
+	request := &qdrant.QueryPoints{
+		CollectionName: collectionName,
+		Using:          vectorName,
+		Query: &qdrant.Query{
+			Variant: &qdrant.Query_Nearest{
+				Nearest: vector,
+			},
+		},
+		Filter:      filter,
+		WithPayload: payload,
+		Limit:       &limit,
+	}
+	return c.client.Query(ctx, request)
+}
+
 func (c *qdrantClient) Connect(ctx context.Context) error {
 	c.logger.Debug("Checking connection to Qdrant")
 	_, err := c.client.HealthCheck(ctx)
diff --git a/internal/impl/qdrant/integration_test.go b/internal/impl/qdrant/integration_test.go
index 7f4e396c02..3eb01be6fc 100644
--- a/internal/impl/qdrant/integration_test.go
+++ b/internal/impl/qdrant/integration_test.go
@@ -17,12 +17,16 @@ package qdrant
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"strconv"
 	"strings"
+	"sync"
 	"testing"
 
 	"github.com/qdrant/go-client/qdrant"
+	_ "github.com/redpanda-data/benthos/v4/public/components/pure"
+	"github.com/redpanda-data/benthos/v4/public/service"
 	"github.com/redpanda-data/benthos/v4/public/service/integration"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -44,13 +48,13 @@ output:
 `
 )
 
-func TestIntegrationQdrant(t *testing.T) {
+func TestIntegrationQdrant_Output(t *testing.T) {
 	integration.CheckSkip(t)
 
 	t.Parallel()
 
 	ctx := context.Background()
-	qdrantContainer, err := qc.Run(ctx, "qdrant/qdrant:v1.10.1")
+	qdrantContainer, err := qc.Run(ctx, "qdrant/qdrant:v1.14.0")
 	require.NoError(t, err, "failed to start container")
 
 	testCases := []struct {
@@ -152,6 +156,139 @@ func TestIntegrationQdrant(t *testing.T) {
 	require.NoError(t, qdrantContainer.Terminate(ctx), "failed to terminate container")
 }
 
+func TestIntegrationQdrant_Processor(t *testing.T) {
+	integration.CheckSkip(t)
+
+	t.Parallel()
+
+	ctx := context.Background()
+	qdrantContainer, err := qc.Run(ctx, "qdrant/qdrant:v1.14.0")
+	require.NoError(t, err, "failed to start container")
+
+	vectors := []any{
+		[]any{0.352, 0.532, 0.532},
+		map[string]any{"some_sparse": map[string]any{"indices": []any{23, 325, 532}, "values": []any{0.352, 0.532, 0.532}}},
+		map[string]any{"some_dense": []any{0.352, 0.532, 0.532}, "some_sparse": map[string]any{"indices": []any{23, 325, 532}, "values": []any{0.352, 0.532, 0.532}}},
+	}
+
+	payloads := []map[string]any{
+		{
+			"city":  "London",
+			"color": "red",
+		},
+		{
+			"city":  "London",
+			"color": "blue",
+		},
+		{
+			"city":  "New York",
+			"color": "blue",
+		},
+	}
+
+	addr, err := qdrantContainer.GRPCEndpoint(ctx)
+	require.NoError(t, err, "failed to get container grpc endpoint")
+
+	host, port, err := parseHostAndPort(addr)
+	require.NoError(t, err, "failed to parse host and port")
+
+	err = setupCollection(ctx, addr, collectionName)
+	require.NoError(t, err, "failed to setup collection")
+
+	client, err := qdrant.NewClient(&qdrant.Config{
+		Host: host,
+		Port: port,
+	})
+	require.NoError(t, err, "failed to create qdrant client")
+	var points []*qdrant.PointStruct
+	for i, vector := range vectors {
+		v, err := newVectors(vector)
+		require.NoError(t, err, "failed to create vector")
+		for j, payload := range payloads {
+			points = append(points, &qdrant.PointStruct{
+				Id:      qdrant.NewIDNum(uint64((i * len(payloads)) + j)),
+				Payload: qdrant.NewValueMap(payload),
+				Vectors: qdrant.NewVectorsMap(v),
+			})
+		}
+	}
+	wait := true
+	_, err = client.Upsert(ctx, &qdrant.UpsertPoints{
+		CollectionName: collectionName,
+		Points:         points,
+		Wait:           &wait,
+		Ordering:       &qdrant.WriteOrdering{Type: qdrant.WriteOrderingType_Strong},
+	})
+	require.NoError(t, err, "failed to upsert point")
+
+	builder := service.NewStreamBuilder()
+	err = builder.AddProcessorYAML(strings.NewReplacer(
+		"$PORT", strconv.Itoa(port),
+		"$COLLECTION_NAME", collectionName,
+	).Replace(`
+qdrant:
+  grpc_host: 'localhost:$PORT'
+  collection_name: $COLLECTION_NAME
+  vector_mapping: this.vector
+  filter: this.filter
+  payload_fields: ['city']
+  payload_filter: exclude
+  limit: 1`))
+	require.NoError(t, err, "failed to create processor")
+	produce, err := builder.AddProducerFunc()
+	require.NoError(t, err, "failed to create producer")
+	output := service.MessageBatch{}
+	var mu sync.Mutex
+	err = builder.AddConsumerFunc(func(ctx context.Context, m *service.Message) error {
+		mu.Lock()
+		defer mu.Unlock()
+		output = append(output, m)
+		return nil
+	})
+	require.NoError(t, err, "failed to create conusmer")
+	stream, err := builder.Build()
+	require.NoError(t, err, "failed to create stream")
+	streamCtx, cancel := context.WithCancel(ctx)
+	streamDone := make(chan any)
+	go func() {
+		err := stream.Run(streamCtx)
+		if errors.Is(err, streamCtx.Err()) {
+			err = nil
+		}
+		require.NoError(t, err)
+		close(streamDone)
+	}()
+	err = produce(ctx, service.NewMessage([]byte(`{
+		"vector": [0.352,0.532,0.532],
+		"filter": {"must": [{"field":{"key": "color", "match": {"text": "red"}}}]}
+	}`)))
+	require.NoError(t, err, "failed to produce message")
+	err = produce(ctx, service.NewMessage([]byte(`{
+		"vector": {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}},
+		"filter": {
+			"must": [{"has_id":{"has_id":[{"num": 8}]}}],
+			"must_not": [{"field":{"key": "city", "match": {"text": "London"}}}]
+		}
+	}`)))
+	require.NoError(t, err, "failed to produce message")
+	cancel()
+	<-streamDone
+
+	expected := []string{
+		`[{"id":{"num":"0"},"payload":{"color":{"stringValue":"red"}},"score":0.9999999}]`,
+		`[{"id":{"num":"8"},"payload":{"color":{"stringValue":"blue"}},"score":0.689952}]`,
+	}
+
+	for i, m := range output {
+		require.NoError(t, m.GetError(), "message had error")
+		b, err := m.AsBytes()
+		require.NoError(t, err, "failed to get message bytes")
+		require.Equal(t, expected[i], string(b))
+	}
+
+	require.NoError(t, qdrantContainer.Terminate(ctx), "failed to terminate container")
+}
+
 func setupCollection(ctx context.Context, addr, collectionName string) error {
 
 	host, port, err := parseHostAndPort(addr)
diff --git a/internal/impl/qdrant/output.go b/internal/impl/qdrant/output.go
index b479f92d9c..4a4a13904f 100644
--- a/internal/impl/qdrant/output.go
+++ b/internal/impl/qdrant/output.go
@@ -219,7 +219,7 @@ func (w *outputWriter) batchPointsByCollection(batch service.MessageBatch) (map[
 		if err != nil {
 			return nil, fmt.Errorf("%s extraction failed: %w", qoFieldPayloadMapping, err)
 		}
-		maybePayloadMap, ok := maybePayload.(map[string]interface{})
+		maybePayloadMap, ok := maybePayload.(map[string]any)
 		if !ok {
 			return nil, errors.New("unable to coerce payload output type")
 		}
@@ -231,7 +231,7 @@ func (w *outputWriter) batchPointsByCollection(batch service.MessageBatch) (map[
 
 		batches[collectionName] = append(batches[collectionName], &qdrant.PointStruct{
 			Id:      id,
-			Vectors: vec,
+			Vectors: qdrant.NewVectorsMap(vec),
 			Payload: payload,
 		})
 	}
diff --git a/internal/impl/qdrant/processor.go b/internal/impl/qdrant/processor.go
new file mode 100644
index 0000000000..f5bc5c79d9
--- /dev/null
+++ b/internal/impl/qdrant/processor.go
@@ -0,0 +1,283 @@
+// Copyright 2025 Redpanda Data, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package qdrant
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"slices"
+
+	"github.com/qdrant/go-client/qdrant"
+	"github.com/redpanda-data/benthos/v4/public/bloblang"
+	"github.com/redpanda-data/benthos/v4/public/service"
+	"google.golang.org/protobuf/encoding/protojson"
+)
+
+const (
+	qpFieldGrpcHost       = "grpc_host"
+	qpFieldAPIToken       = "api_token"
+	qpFieldTLS            = "tls"
+	qpFieldCollectionName = "collection_name"
+	qpFieldVectorMapping  = "vector_mapping"
+	qpFieldFilter         = "filter"
+	qpFieldPayloadFields  = "payload_fields"
+	qpFieldPayloadFilter  = "payload_filter"
+	qpFieldLimit          = "limit"
+)
+
+func processorSpec() *service.ConfigSpec {
+	return service.NewConfigSpec().
+		Categories("AI").
+		Summary("Query items within a https://qdrant.tech/[Qdrant^] collection.").
+		Fields(
+			service.NewStringField(qpFieldGrpcHost).
+				Description("The gRPC host of the Qdrant server.").
+				Example("localhost:6334").
+				Example("xyz-example.eu-central.aws.cloud.qdrant.io:6334"),
+			service.NewStringField(qpFieldAPIToken).
+				Secret().
+				Description("The Qdrant API token for authentication. Defaults to an empty string.").Default(""),
+			service.NewTLSToggledField(qpFieldTLS).Description("TLS(HTTPS) config to use when connecting"),
+			service.NewInterpolatedStringField(qpFieldCollectionName).
+				Description("The name of the collection in Qdrant."),
+			service.NewBloblangField(qpFieldVectorMapping).
+				Description("The mapping to extract the search vector from the document.").
+				Example(`root = [1.2, 0.5, 0.76]`).
+				Example(`root = this.vector`).
+				Example(`root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]`).
+				Example(`root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}}`).
+				Example(`root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]}`).
+				Example(`root = {"some_dense": [0.352,0.532,0.532,0.234]}`),
+			service.NewBloblangField(qpFieldFilter).
+				Optional().
+				Description("Additional filtering to perform on the results. The mapping should return a valid filter (using the proto3 encoded form) in qdrant. See the https://qdrant.tech/documentation/concepts/filtering/[^Qdrant documentation] for examples.").
+				Example(`
+root.must = [
+	{"has_id":{"has_id":[{"num": 8}, { "uuid":"1234-5678-90ab-cdef" }]}},
+	{"field":{"key": "city", "match": {"text": "London"}}},
+]
+`).Example(`
+root.must = [
+	{"field":{"key": "city", "match": {"text": "London"}}},
+]
+root.must_not = [
+	{"field":{"color": "city", "match": {"text": "red"}}},
+]
+`),
+			service.NewStringListField(qpFieldPayloadFields).
+				Default([]any{}).
+				Description("The fields to include or exclude in returned result based on the `payload_filter`."),
+			service.NewStringAnnotatedEnumField(qpFieldPayloadFilter, map[string]string{
+				"include": "Include the payload fields specified in `payload_fields`.",
+				"exclude": "Exclude the payload fields specified in `payload_fields`.",
+			}).
+				Default("include").
+				Description("The way the fields in `payload_fields` are filtered in the result."),
+			service.NewIntField(qpFieldLimit).
+				Default(10).
+				Description("The maximum number of points to return."),
+		)
+}
+
+func init() {
+	err := service.RegisterProcessor(
+		"qdrant",
+		processorSpec(),
+		newProcessor,
+	)
+	if err != nil {
+		panic(err)
+	}
+}
+
+func newProcessor(conf *service.ParsedConfig, mgr *service.Resources) (service.Processor, error) {
+	collectionName, err := conf.FieldInterpolatedString(qpFieldCollectionName)
+	if err != nil {
+		return nil, err
+	}
+
+	vectorMapping, err := conf.FieldBloblang(qpFieldVectorMapping)
+	if err != nil {
+		return nil, err
+	}
+
+	var filter *bloblang.Executor
+	if conf.Contains(qpFieldFilter) {
+		filter, err = conf.FieldBloblang(qpFieldFilter)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	payloadFields, err := conf.FieldStringList(qpFieldPayloadFields)
+	if err != nil {
+		return nil, err
+	}
+
+	payloadFilter, err := conf.FieldString(qpFieldPayloadFilter)
+	if err != nil {
+		return nil, err
+	}
+
+	var payloadSelector *qdrant.WithPayloadSelector
+	if payloadFilter == "include" {
+		if len(payloadFields) > 0 {
+			payloadSelector = qdrant.NewWithPayloadInclude(payloadFields...)
+		} else {
+			payloadSelector = qdrant.NewWithPayloadEnable(false)
+		}
+	} else {
+		if len(payloadFields) > 0 {
+			payloadSelector = qdrant.NewWithPayloadExclude(payloadFields...)
+		} else {
+			payloadSelector = qdrant.NewWithPayloadEnable(true)
+		}
+	}
+
+	limit, err := conf.FieldInt(qpFieldLimit)
+	if err != nil {
+		return nil, err
+	}
+
+	host, err := conf.FieldString(qpFieldGrpcHost)
+	if err != nil {
+		return nil, err
+	}
+
+	apiToken, err := conf.FieldString(qpFieldAPIToken)
+	if err != nil {
+		return nil, err
+	}
+
+	tlsConfig, enabled, err := conf.FieldTLSToggled(qpFieldTLS)
+	if err != nil {
+		return nil, err
+	}
+
+	client, err := newQdrantClient(host, apiToken, enabled, tlsConfig, mgr.Logger())
+	if err != nil {
+		return nil, err
+	}
+	return &processor{
+		client:         client,
+		filter:         filter,
+		collectionName: collectionName,
+		vectorMapping:  vectorMapping,
+		payload:        payloadSelector,
+		limit:          uint64(limit),
+	}, nil
+}
+
+type processor struct {
+	client *qdrantClient
+
+	collectionName *service.InterpolatedString
+	payload        *qdrant.WithPayloadSelector
+	vectorMapping  *bloblang.Executor
+	filter         *bloblang.Executor
+	limit          uint64
+}
+
+var _ service.Processor = (*processor)(nil)
+
+// Process implements service.Processor.
+func (p *processor) Process(ctx context.Context, msg *service.Message) (service.MessageBatch, error) {
+	collection, err := p.collectionName.TryString(msg)
+	if err != nil {
+		return nil, fmt.Errorf("failed to interpolate `%s`: %w", qpFieldCollectionName, err)
+	}
+	var filter qdrant.Filter
+	if p.filter != nil {
+		rawFilter, err := msg.BloblangQuery(p.filter)
+		if err != nil {
+			return nil, fmt.Errorf("failed to execute `%s`: %w", qpFieldFilter, err)
+		}
+		b, err := rawFilter.AsBytes()
+		if err != nil {
+			return nil, fmt.Errorf("%s extraction failed: %w", qpFieldFilter, err)
+		}
+		if string(b) != `null` {
+			if err = protojson.Unmarshal(b, &filter); err != nil {
+				return nil, fmt.Errorf("invalid filter, filters should result in JSON data that is parsable into a qdrant Filter proto3 message. Error: %w", err)
+			}
+		}
+	}
+	rawVec, err := msg.BloblangQuery(p.vectorMapping)
+	if err != nil {
+		return nil, fmt.Errorf("failed to execute `%s`: %w", qpFieldVectorMapping, err)
+	}
+	maybeVec, err := rawVec.AsStructured()
+	if err != nil {
+		return nil, fmt.Errorf("%s extraction failed: %w", qpFieldVectorMapping, err)
+	}
+	vec, err := newVectors(maybeVec)
+	if err != nil {
+		return nil, fmt.Errorf("unable to coerce vector output type: %w", err)
+	}
+	if len(vec) != 1 {
+		return nil, fmt.Errorf("expected only a single vector to search on, got: %d", len(vec))
+	}
+	var vectorName *string
+	var vector *qdrant.VectorInput
+	for k, v := range vec {
+		if k != "" {
+			vectorName = &k
+		}
+		if v.GetVectorsCount() > 0 {
+			var vecs [][]float32
+			for chunk := range slices.Chunk(v.Data, int(v.GetVectorsCount())) {
+				vecs = append(vecs, chunk)
+			}
+			vector = qdrant.NewVectorInputMulti(vecs)
+		} else if v.Indices != nil {
+			vector = qdrant.NewVectorInputSparse(v.Indices.Data, v.Data)
+		} else {
+			vector = qdrant.NewVectorInputDense(v.Data)
+		}
+	}
+	results, err := p.client.Query(
+		ctx,
+		collection,
+		vectorName,
+		vector,
+		p.payload,
+		&filter,
+		p.limit,
+	)
+	if err != nil {
+		return nil, fmt.Errorf("failed to query qdrant: %w", err)
+	}
+	points := []json.RawMessage{}
+	for _, result := range results {
+		b, err := protojson.Marshal(result)
+		if err != nil {
+			return nil, err
+		}
+		points = append(points, json.RawMessage(b))
+	}
+	b, err := json.Marshal(points)
+	if err != nil {
+		return nil, err
+	}
+	msg = msg.Copy()
+	msg.SetBytes(b)
+	return service.MessageBatch{msg}, nil
+}
+
+// Close implements service.Processor.
+func (p *processor) Close(ctx context.Context) error {
+	return p.client.Close()
+}
diff --git a/internal/impl/qdrant/vectors.go b/internal/impl/qdrant/vectors.go
index a1bbbcb786..2ee13121c5 100644
--- a/internal/impl/qdrant/vectors.go
+++ b/internal/impl/qdrant/vectors.go
@@ -22,7 +22,7 @@ import (
 )
 
 // newVectors converts the input into the appropriate *pb.Vectors format
-func newVectors(input any) (*qdrant.Vectors, error) {
+func newVectors(input any) (map[string]*qdrant.Vector, error) {
 	namedVectors := make(map[string]*qdrant.Vector)
 
 	switch vec := input.(type) {
@@ -78,7 +78,7 @@ func newVectors(input any) (*qdrant.Vectors, error) {
 		return nil, fmt.Errorf("unsupported vector input type: %T", input)
 	}
 
-	return qdrant.NewVectorsMap(namedVectors), nil
+	return namedVectors, nil
 }
 
 // Handle dense and multi-vectors
diff --git a/internal/plugins/info.csv b/internal/plugins/info.csv
index b5d98f01ed..2bebd62e84 100644
--- a/internal/plugins/info.csv
+++ b/internal/plugins/info.csv
@@ -190,6 +190,7 @@ pulsar                    ,input     ,pulsar                    ,3.43.0  ,commun
 pulsar                    ,output    ,pulsar                    ,3.43.0  ,community  ,n          ,n     ,n
 pusher                    ,output    ,pusher                    ,4.3.0   ,community  ,n          ,n     ,n
 qdrant                    ,output    ,qdrant                    ,4.33.0  ,certified  ,n          ,y     ,y
+qdrant                    ,processor ,qdrant                    ,4.54.0  ,certified  ,n          ,y     ,y
 questdb                   ,output    ,questdb                   ,4.37.0  ,certified  ,n          ,y     ,y
 rate_limit                ,processor ,rate_limit                ,0.0.0   ,certified  ,n          ,y     ,y
 re_match                  ,scanner   ,re_match                  ,0.0.0   ,certified  ,n          ,y     ,y


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This pull request adds several new configuration templates and updates to support RAG pipelines, including embedding processors, ingestion configurations, indexing pipelines, retrieval endpoints, evaluation scenarios, and a docker-compose setup for local testing.  
- Introduces a new field in the Cohere embeddings processor (EmbeddingTypes).  
- Adds various YAML configuration templates for Redpanda, pgvector, OpenAI, Cohere, and Ollama pipelines.  
- Provides infrastructure configuration files including docker-compose and README updates.

### Reviewed Changes

Copilot reviewed 18 out of 20 changed files in this pull request and generated 2 comments.

<details>
<summary>Show a summary per file</summary>

| File | Description |
| ---- | ----------- |
| internal/impl/cohere/embeddings_processor.go | Adds an EmbeddingTypes field to the request body. |
| config/rag/templates/redpanda.yaml | New Redpanda template configuration with ingestion and mapping settings. |
| config/rag/templates/pgvector_query.yaml | New configuration for pgvector query processor. |
| config/rag/templates/pgvector_output.yaml | Defines pgvector output template with custom batching defaults. |
| config/rag/templates/openai_embeddings.yaml | New OpenAI embeddings processor template. |
| config/rag/templates/ollama_embeddings.yaml | New Ollama embeddings processor template. |
| config/rag/templates/cohere_embeddings.yaml | New Cohere embeddings processor template. |
| config/rag/rpk.profile.yaml | New rpk profile file for containerized environments. |
| config/rag/retrieval/*.yaml | Retrieval pipelines for OpenAI, Ollama, and Cohere configurations. |
| config/rag/ingestion/redpanda-docs.yaml | Redpanda ingestion template for documentation sources. |
| config/rag/indexing/*.yaml | Indexing pipelines for OpenAI, Ollama, and Cohere configurations. |
| config/rag/eval.yaml | Evaluation pipeline definition with multiple test queries and workflow. |
| config/rag/docker-compose.yml | Docker Compose configuration for required services. |
| config/rag/README.md | Documentation update for RAG pipeline usage. |
</details>


<details>
<summary>Files not reviewed (2)</summary>

* **config/rag/.gitignore**: Language not supported
* **config/rag/env.sample**: Language not supported
</details>



state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T17:44:36Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR introduces configuration templates for various RAG pipelines including ingestion, indexing, retrieval, and evaluation, along with a minor fix to the cohere embeddings processor and the addition of a new qdrant processor.  
- Fix the cohere embeddings processor bug  
- Add multiple example pipelines using various LLMs and vector databases  
- Introduce a new qdrant processor for querying

### Reviewed Changes

Copilot reviewed 26 out of 28 changed files in this pull request and generated 3 comments.

<details>
<summary>Show a summary per file</summary>

| File                                        | Description                                                      |
| ------------------------------------------- | ---------------------------------------------------------------- |
| config/rag/templates/redpanda.yaml          | Adds an input configuration for Redpanda with optional auth      |
| config/rag/templates/pgvector_query.yaml      | Introduces a PGVector query processor configuration                |
| config/rag/templates/pgvector_output.yaml     | Provides a PGVector output processor configuration                  |
| config/rag/templates/openai_embeddings.yaml   | Defines an OpenAI embeddings processor configuration                |
| config/rag/templates/ollama_embeddings.yaml    | Sets up an Ollama embeddings processor configuration                 |
| config/rag/templates/cohere_embeddings.yaml    | Updates the cohere embeddings processor configuration (bug fix)       |
| config/rag/rpk.profile.yaml                    | Introduces a Docker Compose profile configuration                       |
| config/rag/retrieval/*                         | Adds retrieval pipeline configurations for different embeddings models  |
| config/rag/ingestion/redpanda-docs.yaml         | Adds an ingestion pipeline for Redpanda documentation                    |
| config/rag/indexing/*                          | Adds indexing pipeline configurations for various embeddings models       |
| config/rag/eval.yaml                          | Adds an evaluation pipeline that aggregates and scores retrieval results  |
| config/rag/docker-compose.yml                  | Updates service definitions for supporting the new pipelines and services    |
| config/rag/README.md                           | Updates documentation to reflect the new RAG pipeline configurations         |
| CHANGELOG.md                                  | Documents the addition of the new qdrant processor and other changes         |
</details>


<details>
<summary>Files not reviewed (2)</summary>

* **config/rag/.gitignore**: Language not supported
* **config/rag/env.sample**: Language not supported
</details>



state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T16:11:35Z
user: login: mihaitodor
body: LGTM, just a few small nits. eel free to 🐑 🚀 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T00:52:53Z
user: login: rockwotj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T13:54:32Z
user: login: rockwotj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T14:00:24Z

Review comments:
diff_hunk: @@ -0,0 +1,79 @@
+name: rag_topics
+type: input 
+fields:
+  - name: seed_brokers
+    type: string
+  - name: user
+    type: string
+    default: ""
+  - name: password
+    type: string
+    default: ""
+  - name: consumer_group
+    type: string
+  - name: batching
+    type: unknown 
path: config/rag/templates/redpanda.yaml
user: login: Copilot
body: The field 'batching' is defined with type 'unknown'. Consider using a more explicit, supported type to avoid potential configuration issues.
```suggestion
    type: map
    keys:
      count:
        type: integer
      period:
        type: string
```
created_at: 2025-04-23T17:44:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,81 @@
+name: pgvector
+type: output
+
+fields:
+  - name: table
+    type: string
+  - name: dsn
+    type: string
+  - name: max_in_flight
+    type: int
+    default: 8
+  - name: dimensions
+    type: int
+  - name: batching
+    type: unknown
path: config/rag/templates/pgvector_output.yaml
user: login: Copilot
body: The 'batching' field here is also set to type 'unknown'. It would be beneficial to specify a clear type to ensure consistent behavior.
```suggestion
    type: object
    properties:
      count:
        type: int
      period:
        type: string
    required:
      - count
      - period
```
created_at: 2025-04-23T17:44:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,79 @@
+name: rag_topics
+type: input 
+fields:
+  - name: seed_brokers
+    type: string
+  - name: user
+    type: string
+    default: ""
+  - name: password
+    type: string
+    default: ""
+  - name: consumer_group
+    type: string
+  - name: batching
+    type: unknown 
path: config/rag/templates/redpanda.yaml
user: login: Copilot
body: The 'batching' field is assigned a type of 'unknown', which may lead to unexpected behavior. Consider replacing it with a concrete type or adding documentation to clarify its intended use.
```suggestion
    type: object
    properties:
      count:
        type: integer
      period:
        type: string
```
created_at: 2025-04-24T16:11:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+## RAG with Redpanda Connect
+
+This folder hosts a series of RAG pipelines using Redpanda + Redpanda Connect.
+
+We have a series of ingestion pipelines in the `ingestion` directory, these all write
+data we want to ingest into our vector database in topics with the pattern: `rp.ai.rag.*`
+
+Next, there is a matrix of different indexing piplines that use the following sets of options:
+
+Vector Database: `{postgres, elasticsearch, qdrant}`
+
+Embeddings Model: `{openai, cohere}`
+
+These are implemented as resources in the `indexing/resources` directory, then each instance of a pipline
path: config/rag/README.md
user: login: Copilot
body: Typo detected: 'piplines' should be corrected to 'pipelines'.
```suggestion
Next, there is a matrix of different indexing pipelines that use the following sets of options:

Vector Database: `{postgres, elasticsearch, qdrant}`

Embeddings Model: `{openai, cohere}`

These are implemented as resources in the `indexing/resources` directory, then each instance of a pipeline
```
created_at: 2025-04-24T16:11:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+## RAG with Redpanda Connect
+
+This folder hosts a series of RAG pipelines using Redpanda + Redpanda Connect.
+
+We have a series of ingestion pipelines in the `ingestion` directory, these all write
+data we want to ingest into our vector database in topics with the pattern: `rp.ai.rag.*`
+
+Next, there is a matrix of different indexing piplines that use the following sets of options:
+
+Vector Database: `{postgres, elasticsearch, qdrant}`
+
+Embeddings Model: `{openai, cohere}`
+
+These are implemented as resources in the `indexing/resources` directory, then each instance of a pipline
+is created in the `indexing` pipeline.
+
+Lastly, there is a set of retrivial pipelines that mirror each one of our indexing pipelines. These pipelines
+are exposed over HTTP and they can be used to retrive documents/chunks from. These pipelines all live in the
+`retrivial` directory.
path: config/rag/README.md
user: login: Copilot
body: Typo detected: 'retrivial' should be corrected to 'retrieval'.
```suggestion
Lastly, there is a set of retrieval pipelines that mirror each one of our indexing pipelines. These pipelines
are exposed over HTTP and they can be used to retrieve documents/chunks from. These pipelines all live in the
`retrieval` directory.
```
created_at: 2025-04-24T16:11:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,102 @@
+services:
+  ### VECTOR DATABASES ###
+  postgres:
+    image: pgvector/pgvector:0.8.0-pg17
+    environment:
+      POSTGRES_DB: mydatabase
+      POSTGRES_USER: myuser
+      POSTGRES_PASSWORD: mypassword
+    ports:
+      - "5432:5432"
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    networks:
+      - the_network
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:9.0.0
+    environment:
+      - discovery.type=single-node
+      - ES_JAVA_OPTS=-Xms1G -Xmx4G
+    ports:
+      - "9200:9200"
+      - "9300:9300"
+    volumes:
+      - esdata:/usr/share/elasticsearch/data
+    networks:
+      - the_network
+  qdrant:
+    image: qdrant/qdrant:latest
+    ports:
+      - "6333:6333"
+    volumes:
+      - qdrant_storage:/qdrant/storage
+    networks:
+      - the_network
+  ### TRUSTY INGESTION ENGINE ###
+  redpanda:
+    image: redpandadata/redpanda:v25.1.1
+    command:
+      - redpanda
+      - start
+      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
+      # Address the broker advertises to clients that connect to the Kafka API.
+      # Use the internal addresses to connect to the Redpanda brokers'
+      # from inside the same Docker network.
+      # Use the external addresses to connect to the Redpanda brokers'
+      # from outside the Docker network.
+      - --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
+      - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
+      # Address the broker advertises to clients that connect to the HTTP Proxy.
+      - --advertise-pandaproxy-addr internal://redpanda:8082,external://localhost:18082
+      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
+      # Redpanda brokers use the RPC API to communicate with each other internally.
+      - --rpc-addr redpanda:33145
+      - --advertise-rpc-addr redpanda:33145
+      # Mode dev-container uses well-known configuration properties for development in containers.
+      - --mode dev-container
+      # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
+      - --smp 1
+      - --default-log-level=info
+      - --set redpanda.auto_create_topics_enabled=true
+    ports:
+      - 18081:18081
+      - 18082:18082
+      - 19092:19092
+      - 19644:9644
+    volumes:
+      - redpandadata:/var/lib/redpanda/data
+    networks:
+      - the_network
+  redpanda-console:
+    image: redpandadata/console:v3.0.0
+    entrypoint: /bin/sh
+    command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
+    environment:
+      CONFIG_FILEPATH: /tmp/config.yml
+      CONSOLE_CONFIG_FILE: |
+        kafka:
+          brokers: ["redpanda:9092"]
+        schemaRegistry:
+          enabled: true
+          urls: ["http://redpanda:8081"]
+        redpanda:
+          adminApi:
+            enabled: true
+            urls: ["http://redpanda:9644"]
+    ports:
+      - 8080:8080
+    depends_on:
+      - redpanda
+    networks:
+      - the_network
+
+volumes:
+  pgdata: null
+  redpandadata: null
+  esdata: null
+  qdrant_storage: null
+
+networks:
+  the_network:
path: config/rag/docker-compose.yml
user: login: mihaitodor
body: Nit: Might be worth giving this a more descriptive name
created_at: 2025-04-28T00:30:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,283 @@
+// Copyright 2025 Redpanda Data, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package qdrant
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"slices"
+
+	"github.com/qdrant/go-client/qdrant"
+	"github.com/redpanda-data/benthos/v4/public/bloblang"
+	"github.com/redpanda-data/benthos/v4/public/service"
+	"google.golang.org/protobuf/encoding/protojson"
+)
+
+const (
+	qpFieldGrpcHost       = "grpc_host"
+	qpFieldAPIToken       = "api_token"
+	qpFieldTLS            = "tls"
+	qpFieldCollectionName = "collection_name"
+	qpFieldVectorMapping  = "vector_mapping"
+	qpFieldFilter         = "filter"
+	qpFieldPayloadFields  = "payload_fields"
+	qpFieldPayloadFilter  = "payload_filter"
+	qpFieldLimit          = "limit"
+)
+
+func processorSpec() *service.ConfigSpec {
+	return service.NewConfigSpec().
+		Categories("AI").
+		Summary("Query items within a https://qdrant.tech/[Qdrant^] collection.").
+		Fields(
+			service.NewStringField(qpFieldGrpcHost).
+				Description("The gRPC host of the Qdrant server.").
+				Example("localhost:6334").
+				Example("xyz-example.eu-central.aws.cloud.qdrant.io:6334"),
+			service.NewStringField(qpFieldAPIToken).
+				Secret().
+				Description("The Qdrant API token for authentication. Defaults to an empty string.").Default(""),
+			service.NewTLSToggledField(qpFieldTLS).Description("TLS(HTTPS) config to use when connecting"),
+			service.NewInterpolatedStringField(qpFieldCollectionName).
+				Description("The name of the collection in Qdrant."),
+			service.NewBloblangField(qpFieldVectorMapping).
+				Description("The mapping to extract the search vector from the document.").
+				Example(`root = [1.2, 0.5, 0.76]`).
+				Example(`root = this.vector`).
+				Example(`root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]`).
+				Example(`root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}}`).
+				Example(`root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]}`).
+				Example(`root = {"some_dense": [0.352,0.532,0.532,0.234]}`),
+			service.NewBloblangField(qpFieldFilter).
+				Optional().
+				Description("Additional filtering to perform on the results. The mapping should return a valid filter (using the proto3 encoded form) in qdrant. See the https://qdrant.tech/documentation/concepts/filtering/[^Qdrant documentation] for examples.").
+				Example(`
+root.must = [
+	{"has_id":{"has_id":[{"num": 8}, { "uuid":"1234-5678-90ab-cdef" }]}},
+	{"field":{"key": "city", "match": {"text": "London"}}},
+]
+`).Example(`
+root.must = [
+	{"field":{"key": "city", "match": {"text": "London"}}},
+]
+root.must_not = [
+	{"field":{"color": "city", "match": {"text": "red"}}},
+]
+`),
+			service.NewStringListField(qpFieldPayloadFields).
+				Default([]any{}).
+				Description("The fields to include or exclude in returned result based on the `payload_filter`."),
+			service.NewStringAnnotatedEnumField(qpFieldPayloadFilter, map[string]string{
+				"include": "Include the payload fields specified in `payload_fields`.",
+				"exclude": "Exclude the payload fields specified in `payload_fields`.",
+			}).
+				Default("include").
+				Description("The way the fields in `payload_fields` are filtered in the result."),
+			service.NewIntField(qpFieldLimit).
+				Default(10).
+				Description("The maximum number of points to return."),
+		)
+}
+
+func init() {
+	err := service.RegisterProcessor(
+		"qdrant",
+		processorSpec(),
+		newProcessor,
+	)
+	if err != nil {
+		panic(err)
+	}
+}
+
+func newProcessor(conf *service.ParsedConfig, mgr *service.Resources) (service.Processor, error) {
+	collectionName, err := conf.FieldInterpolatedString(qpFieldCollectionName)
+	if err != nil {
+		return nil, err
+	}
+
+	vectorMapping, err := conf.FieldBloblang(qpFieldVectorMapping)
+	if err != nil {
+		return nil, err
+	}
+
+	var filter *bloblang.Executor
+	if conf.Contains(qpFieldFilter) {
+		filter, err = conf.FieldBloblang(qpFieldFilter)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	payloadFields, err := conf.FieldStringList(qpFieldPayloadFields)
+	if err != nil {
+		return nil, err
+	}
+
+	payloadFilter, err := conf.FieldString(qpFieldPayloadFilter)
+	if err != nil {
+		return nil, err
+	}
+
+	var payloadSelector *qdrant.WithPayloadSelector
+	if payloadFilter == "include" {
+		if len(payloadFields) > 0 {
+			payloadSelector = qdrant.NewWithPayloadInclude(payloadFields...)
+		} else {
+			payloadSelector = qdrant.NewWithPayloadEnable(false)
+		}
+	} else {
+		if len(payloadFields) > 0 {
+			payloadSelector = qdrant.NewWithPayloadExclude(payloadFields...)
+		} else {
+			payloadSelector = qdrant.NewWithPayloadEnable(true)
+		}
+	}
+
+	limit, err := conf.FieldInt(qpFieldLimit)
+	if err != nil {
+		return nil, err
+	}
+
+	host, err := conf.FieldString(qpFieldGrpcHost)
+	if err != nil {
+		return nil, err
+	}
+
+	apiToken, err := conf.FieldString(qpFieldAPIToken)
+	if err != nil {
+		return nil, err
+	}
+
+	tlsConfig, enabled, err := conf.FieldTLSToggled(qpFieldTLS)
+	if err != nil {
+		return nil, err
+	}
+
+	client, err := newQdrantClient(host, apiToken, enabled, tlsConfig, mgr.Logger())
+	if err != nil {
+		return nil, err
+	}
+	return &processor{
+		client:         client,
+		filter:         filter,
+		collectionName: collectionName,
+		vectorMapping:  vectorMapping,
+		payload:        payloadSelector,
+		limit:          uint64(limit),
+	}, nil
+}
+
+type processor struct {
+	client *qdrantClient
+
+	collectionName *service.InterpolatedString
+	payload        *qdrant.WithPayloadSelector
+	vectorMapping  *bloblang.Executor
+	filter         *bloblang.Executor
+	limit          uint64
+}
+
+var _ service.Processor = (*processor)(nil)
+
+// Process implements service.Processor.
+func (p *processor) Process(ctx context.Context, msg *service.Message) (service.MessageBatch, error) {
+	collection, err := p.collectionName.TryString(msg)
+	if err != nil {
+		return nil, fmt.Errorf("failed to interpolate `%s`: %w", qpFieldCollectionName, err)
+	}
+	var filter qdrant.Filter
+	if p.filter != nil {
+		rawFilter, err := msg.BloblangQuery(p.filter)
+		if err != nil {
+			return nil, fmt.Errorf("failed to execute `%s`: %w", qpFieldFilter, err)
+		}
+		b, err := rawFilter.AsBytes()
+		if err != nil {
+			return nil, fmt.Errorf("%s extraction failed: %w", qpFieldFilter, err)
+		}
+		if string(b) != `null` {
+			if err = protojson.Unmarshal(b, &filter); err != nil {
+				return nil, fmt.Errorf("invalid filter, filters should result in JSON data that is parsable into a qdrant Filter proto3 message. Error: %w", err)
+			}
+		}
+	}
+	rawVec, err := msg.BloblangQuery(p.vectorMapping)
+	if err != nil {
+		return nil, fmt.Errorf("failed to execute `%s`: %w", qpFieldVectorMapping, err)
+	}
+	maybeVec, err := rawVec.AsStructured()
+	if err != nil {
+		return nil, fmt.Errorf("%s extraction failed: %w", qpFieldVectorMapping, err)
+	}
+	vec, err := newVectors(maybeVec)
+	if err != nil {
+		return nil, fmt.Errorf("unable to coerce vector output type: %w", err)
+	}
+	if len(vec) != 1 {
+		return nil, fmt.Errorf("expected only a single vector to search on, got: %d", len(vec))
+	}
+	var vectorName *string
+	var vector *qdrant.VectorInput
+	for k, v := range vec {
+		if k != "" {
+			vectorName = &k
+		}
+		if v.GetVectorsCount() > 0 {
+			var vecs [][]float32
+			for chunk := range slices.Chunk(v.Data, int(v.GetVectorsCount())) {
+				vecs = append(vecs, chunk)
+			}
+			vector = qdrant.NewVectorInputMulti(vecs)
+		} else if v.Indices != nil {
+			vector = qdrant.NewVectorInputSparse(v.Indices.Data, v.Data)
+		} else {
+			vector = qdrant.NewVectorInputDense(v.Data)
+		}
+	}
+	results, err := p.client.Query(
+		ctx,
+		collection,
+		vectorName,
+		vector,
+		p.payload,
+		&filter,
+		p.limit,
+	)
+	if err != nil {
+		return nil, fmt.Errorf("failed to query qdrant: %w", err)
+	}
+	points := []json.RawMessage{}
+	for _, result := range results {
+		b, err := protojson.Marshal(result)
+		if err != nil {
+			return nil, err
+		}
+		points = append(points, json.RawMessage(b))
+	}
+	b, err := json.Marshal(points)
+	if err != nil {
+		return nil, err
path: internal/impl/qdrant/processor.go
user: login: mihaitodor
body: Nit: Might be worth annotating these errors
created_at: 2025-04-28T00:52:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,283 @@
+// Copyright 2025 Redpanda Data, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package qdrant
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"slices"
+
+	"github.com/qdrant/go-client/qdrant"
+	"github.com/redpanda-data/benthos/v4/public/bloblang"
+	"github.com/redpanda-data/benthos/v4/public/service"
+	"google.golang.org/protobuf/encoding/protojson"
+)
+
+const (
+	qpFieldGrpcHost       = "grpc_host"
+	qpFieldAPIToken       = "api_token"
+	qpFieldTLS            = "tls"
+	qpFieldCollectionName = "collection_name"
+	qpFieldVectorMapping  = "vector_mapping"
+	qpFieldFilter         = "filter"
+	qpFieldPayloadFields  = "payload_fields"
+	qpFieldPayloadFilter  = "payload_filter"
+	qpFieldLimit          = "limit"
+)
+
+func processorSpec() *service.ConfigSpec {
+	return service.NewConfigSpec().
+		Categories("AI").
+		Summary("Query items within a https://qdrant.tech/[Qdrant^] collection.").
+		Fields(
+			service.NewStringField(qpFieldGrpcHost).
+				Description("The gRPC host of the Qdrant server.").
+				Example("localhost:6334").
+				Example("xyz-example.eu-central.aws.cloud.qdrant.io:6334"),
+			service.NewStringField(qpFieldAPIToken).
+				Secret().
+				Description("The Qdrant API token for authentication. Defaults to an empty string.").Default(""),
+			service.NewTLSToggledField(qpFieldTLS).Description("TLS(HTTPS) config to use when connecting"),
+			service.NewInterpolatedStringField(qpFieldCollectionName).
+				Description("The name of the collection in Qdrant."),
+			service.NewBloblangField(qpFieldVectorMapping).
+				Description("The mapping to extract the search vector from the document.").
+				Example(`root = [1.2, 0.5, 0.76]`).
+				Example(`root = this.vector`).
+				Example(`root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]`).
+				Example(`root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}}`).
+				Example(`root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]}`).
+				Example(`root = {"some_dense": [0.352,0.532,0.532,0.234]}`),
+			service.NewBloblangField(qpFieldFilter).
+				Optional().
+				Description("Additional filtering to perform on the results. The mapping should return a valid filter (using the proto3 encoded form) in qdrant. See the https://qdrant.tech/documentation/concepts/filtering/[^Qdrant documentation] for examples.").
+				Example(`
+root.must = [
+	{"has_id":{"has_id":[{"num": 8}, { "uuid":"1234-5678-90ab-cdef" }]}},
+	{"field":{"key": "city", "match": {"text": "London"}}},
+]
+`).Example(`
+root.must = [
+	{"field":{"key": "city", "match": {"text": "London"}}},
+]
+root.must_not = [
+	{"field":{"color": "city", "match": {"text": "red"}}},
+]
+`),
+			service.NewStringListField(qpFieldPayloadFields).
+				Default([]any{}).
+				Description("The fields to include or exclude in returned result based on the `payload_filter`."),
+			service.NewStringAnnotatedEnumField(qpFieldPayloadFilter, map[string]string{
+				"include": "Include the payload fields specified in `payload_fields`.",
+				"exclude": "Exclude the payload fields specified in `payload_fields`.",
+			}).
+				Default("include").
+				Description("The way the fields in `payload_fields` are filtered in the result."),
+			service.NewIntField(qpFieldLimit).
+				Default(10).
+				Description("The maximum number of points to return."),
+		)
+}
+
+func init() {
+	err := service.RegisterProcessor(
+		"qdrant",
+		processorSpec(),
+		newProcessor,
+	)
+	if err != nil {
+		panic(err)
+	}
+}
+
+func newProcessor(conf *service.ParsedConfig, mgr *service.Resources) (service.Processor, error) {
+	collectionName, err := conf.FieldInterpolatedString(qpFieldCollectionName)
+	if err != nil {
+		return nil, err
+	}
+
+	vectorMapping, err := conf.FieldBloblang(qpFieldVectorMapping)
+	if err != nil {
+		return nil, err
+	}
+
+	var filter *bloblang.Executor
+	if conf.Contains(qpFieldFilter) {
+		filter, err = conf.FieldBloblang(qpFieldFilter)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	payloadFields, err := conf.FieldStringList(qpFieldPayloadFields)
+	if err != nil {
+		return nil, err
+	}
+
+	payloadFilter, err := conf.FieldString(qpFieldPayloadFilter)
+	if err != nil {
+		return nil, err
+	}
+
+	var payloadSelector *qdrant.WithPayloadSelector
+	if payloadFilter == "include" {
+		if len(payloadFields) > 0 {
+			payloadSelector = qdrant.NewWithPayloadInclude(payloadFields...)
+		} else {
+			payloadSelector = qdrant.NewWithPayloadEnable(false)
+		}
+	} else {
+		if len(payloadFields) > 0 {
+			payloadSelector = qdrant.NewWithPayloadExclude(payloadFields...)
+		} else {
+			payloadSelector = qdrant.NewWithPayloadEnable(true)
+		}
+	}
+
+	limit, err := conf.FieldInt(qpFieldLimit)
+	if err != nil {
+		return nil, err
+	}
+
+	host, err := conf.FieldString(qpFieldGrpcHost)
+	if err != nil {
+		return nil, err
+	}
+
+	apiToken, err := conf.FieldString(qpFieldAPIToken)
+	if err != nil {
+		return nil, err
+	}
+
+	tlsConfig, enabled, err := conf.FieldTLSToggled(qpFieldTLS)
+	if err != nil {
+		return nil, err
+	}
+
+	client, err := newQdrantClient(host, apiToken, enabled, tlsConfig, mgr.Logger())
+	if err != nil {
+		return nil, err
+	}
+	return &processor{
+		client:         client,
+		filter:         filter,
+		collectionName: collectionName,
+		vectorMapping:  vectorMapping,
+		payload:        payloadSelector,
+		limit:          uint64(limit),
+	}, nil
+}
+
+type processor struct {
+	client *qdrantClient
+
+	collectionName *service.InterpolatedString
+	payload        *qdrant.WithPayloadSelector
+	vectorMapping  *bloblang.Executor
+	filter         *bloblang.Executor
+	limit          uint64
+}
+
+var _ service.Processor = (*processor)(nil)
+
+// Process implements service.Processor.
+func (p *processor) Process(ctx context.Context, msg *service.Message) (service.MessageBatch, error) {
+	collection, err := p.collectionName.TryString(msg)
+	if err != nil {
+		return nil, fmt.Errorf("failed to interpolate `%s`: %w", qpFieldCollectionName, err)
+	}
+	var filter qdrant.Filter
+	if p.filter != nil {
+		rawFilter, err := msg.BloblangQuery(p.filter)
+		if err != nil {
+			return nil, fmt.Errorf("failed to execute `%s`: %w", qpFieldFilter, err)
+		}
+		b, err := rawFilter.AsBytes()
+		if err != nil {
+			return nil, fmt.Errorf("%s extraction failed: %w", qpFieldFilter, err)
+		}
+		if string(b) != `null` {
+			if err = protojson.Unmarshal(b, &filter); err != nil {
+				return nil, fmt.Errorf("invalid filter, filters should result in JSON data that is parsable into a qdrant Filter proto3 message. Error: %w", err)
+			}
+		}
+	}
+	rawVec, err := msg.BloblangQuery(p.vectorMapping)
+	if err != nil {
+		return nil, fmt.Errorf("failed to execute `%s`: %w", qpFieldVectorMapping, err)
+	}
+	maybeVec, err := rawVec.AsStructured()
+	if err != nil {
+		return nil, fmt.Errorf("%s extraction failed: %w", qpFieldVectorMapping, err)
+	}
+	vec, err := newVectors(maybeVec)
+	if err != nil {
+		return nil, fmt.Errorf("unable to coerce vector output type: %w", err)
+	}
+	if len(vec) != 1 {
+		return nil, fmt.Errorf("expected only a single vector to search on, got: %d", len(vec))
+	}
+	var vectorName *string
+	var vector *qdrant.VectorInput
+	for k, v := range vec {
+		if k != "" {
+			vectorName = &k
+		}
+		if v.GetVectorsCount() > 0 {
+			var vecs [][]float32
+			for chunk := range slices.Chunk(v.Data, int(v.GetVectorsCount())) {
+				vecs = append(vecs, chunk)
+			}
+			vector = qdrant.NewVectorInputMulti(vecs)
+		} else if v.Indices != nil {
+			vector = qdrant.NewVectorInputSparse(v.Indices.Data, v.Data)
+		} else {
+			vector = qdrant.NewVectorInputDense(v.Data)
+		}
+	}
+	results, err := p.client.Query(
+		ctx,
+		collection,
+		vectorName,
+		vector,
+		p.payload,
+		&filter,
+		p.limit,
+	)
+	if err != nil {
+		return nil, fmt.Errorf("failed to query qdrant: %w", err)
+	}
+	points := []json.RawMessage{}
+	for _, result := range results {
+		b, err := protojson.Marshal(result)
+		if err != nil {
+			return nil, err
+		}
+		points = append(points, json.RawMessage(b))
+	}
+	b, err := json.Marshal(points)
+	if err != nil {
+		return nil, err
path: internal/impl/qdrant/processor.go
user: login: rockwotj
body: I didn't annotate the ones that should never fail (because it's always valid JSON). I probably should I support 🤷 
created_at: 2025-04-28T13:54:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,102 @@
+services:
+  ### VECTOR DATABASES ###
+  postgres:
+    image: pgvector/pgvector:0.8.0-pg17
+    environment:
+      POSTGRES_DB: mydatabase
+      POSTGRES_USER: myuser
+      POSTGRES_PASSWORD: mypassword
+    ports:
+      - "5432:5432"
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    networks:
+      - the_network
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:9.0.0
+    environment:
+      - discovery.type=single-node
+      - ES_JAVA_OPTS=-Xms1G -Xmx4G
+    ports:
+      - "9200:9200"
+      - "9300:9300"
+    volumes:
+      - esdata:/usr/share/elasticsearch/data
+    networks:
+      - the_network
+  qdrant:
+    image: qdrant/qdrant:latest
+    ports:
+      - "6333:6333"
+    volumes:
+      - qdrant_storage:/qdrant/storage
+    networks:
+      - the_network
+  ### TRUSTY INGESTION ENGINE ###
+  redpanda:
+    image: redpandadata/redpanda:v25.1.1
+    command:
+      - redpanda
+      - start
+      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
+      # Address the broker advertises to clients that connect to the Kafka API.
+      # Use the internal addresses to connect to the Redpanda brokers'
+      # from inside the same Docker network.
+      # Use the external addresses to connect to the Redpanda brokers'
+      # from outside the Docker network.
+      - --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
+      - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
+      # Address the broker advertises to clients that connect to the HTTP Proxy.
+      - --advertise-pandaproxy-addr internal://redpanda:8082,external://localhost:18082
+      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
+      # Redpanda brokers use the RPC API to communicate with each other internally.
+      - --rpc-addr redpanda:33145
+      - --advertise-rpc-addr redpanda:33145
+      # Mode dev-container uses well-known configuration properties for development in containers.
+      - --mode dev-container
+      # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
+      - --smp 1
+      - --default-log-level=info
+      - --set redpanda.auto_create_topics_enabled=true
+    ports:
+      - 18081:18081
+      - 18082:18082
+      - 19092:19092
+      - 19644:9644
+    volumes:
+      - redpandadata:/var/lib/redpanda/data
+    networks:
+      - the_network
+  redpanda-console:
+    image: redpandadata/console:v3.0.0
+    entrypoint: /bin/sh
+    command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
+    environment:
+      CONFIG_FILEPATH: /tmp/config.yml
+      CONSOLE_CONFIG_FILE: |
+        kafka:
+          brokers: ["redpanda:9092"]
+        schemaRegistry:
+          enabled: true
+          urls: ["http://redpanda:8081"]
+        redpanda:
+          adminApi:
+            enabled: true
+            urls: ["http://redpanda:9644"]
+    ports:
+      - 8080:8080
+    depends_on:
+      - redpanda
+    networks:
+      - the_network
+
+volumes:
+  pgdata: null
+  redpandadata: null
+  esdata: null
+  qdrant_storage: null
+
+networks:
+  the_network:
path: config/rag/docker-compose.yml
user: login: rockwotj
body: done
created_at: 2025-04-28T14:00:24Z
author_association: CONTRIBUTOR

PR comments:
user: login: mmatczuk
created_at: 2025-04-25T09:12:46Z
body: It would be very cool if we could run the Benthos tests in CI.
author_association: CONTRIBUTOR
user: login: rockwotj
created_at: 2025-04-25T09:16:24Z
body: Can you clarify? We do this in CI: https://github.com/redpanda-data/connect/blob/main/Makefile#L80
author_association: CONTRIBUTOR
user: login: mmatczuk
created_at: 2025-04-25T09:22:42Z
body: Do you think we could be running the Benthos tests added in this PR in CI?
Correct me if I'm wrong but it looks like we run tests in `./config/test/...` but not in `config/rag`.
author_association: CONTRIBUTOR
user: login: rockwotj
created_at: 2025-04-28T14:00:33Z
body: > Do you think we could be running the Benthos tests added in this PR in CI? Correct me if I'm wrong but it looks like we run tests in `./config/test/...` but not in `config/rag`.

fixed
author_association: CONTRIBUTOR