Repository: redpanda-data/connect

Issue #1677: `sql_raw` output should support batched/transactional sends when possible
Created: 2023-01-16T08:47:26Z
I have a processing pipeline that takes a few Kafka topic inputs and upserts the records in a Postgres database. There is a switch output config that selects the output resource by the input topic. When using Benthos 3.65 this pipeline is very fast, it upserts more than one million Kafka records in the DB in a few minutes. After upgrading to Benthos 4.11 the processing became much slower and I also see errors after a few tens of thousands records are processed.
The input for Benthos 3.65 is configured like this:
resources:
  inputs:
    input-topics:
      kafka:
      ...
        topics: T1, T2
        checkpoint_limit: 1
        batching:
          count: 1500
          period: 100ms      


The output for Benthos 3.65 is configured using a sql configuration like this:
  outputs:
    output-table1:
      retry:
        max_retries: 2
        backoff:
          initial_interval: 500ms
          max_interval: 3s
        output:
          sql:
            max_in_flight: 1
            driver: postgres
            data_source_name: postgres://...
           query: INSERT INTO output_table1 (..) VALUES (..) ON CONFLICT(pk) DO UPDATE SET(..) = (..)
     output-table2:

After upgrading to Benthos 4.11 we had to change the output config since sql output config is not supported anymore:
  outputs:
    output-table1:
      retry:
        max_retries: 2
        backoff:
          initial_interval: 500ms
          max_interval: 3s
        output:
          sql_raw:
            max_in_flight: 1
            batching:
              count: 1500
              period: 1s
            driver: postgres
            dsn: postgres://...
            query: INSERT INTO output_table (..) VALUES (..) ON CONFLICT(pk) DO UPDATE SET(..) = (..)

    output-table2:
         ...

I've set max_in_flight=1 to ensure the upserts are correctly ordered. I also tried using the sql_insert configuration but I don't know how to correctly configure an ON CONFLICT.. DO UPDATE SET with params and also I am seeing errors when multiple records with the same key are part of a batch (since it's not possible to have a batch insert in a DB tx for records having the same key).
What am I doing wrong? Is there a config option that could bring this config to the 3.65 performance level?
Thanks!
florin

Duplicate issue #2346: Behaviour of error in sql_raw output
Created: 2024-01-26T13:56:58Z
I'm setting up an http_server ingestion that inserts to a Postgres, but I'm confused by the error handling when it comes to multiple messages in a batch. I will outline my case below in as much detail as I can, hopefully I'm misconfiguring something somewhere so that the error is on my side. Thank you!
In the Benthos configuration example below, the http_server expects a JSON like
{
  "data": [
    {
      "a": "1",
      "b": "1"
    },
    {
      "a": "2",
      "b": "2"
    }
  ]
}

which it splits into multiple Benthos messages for each item in the data array. The database consists of a primary key that's the concatenation of a and b and is intended to fail with pq: duplicate key value violates unique constraint if the same pair gets inserted again.
Issue
curl --request POST --url http://localhost:4195/ingest --header 'Content-Type: application/json' --data '{ "data": [ { "a": "1", "b": "1" }, { "a": "2", "b": "2" } ] }'
Both 1-1 and 2-2 gets inserted, as expected.
curl --request POST --url http://localhost:4195/ingest --header 'Content-Type: application/json' --data '{ "data": [ { "a": "2", "b": "1" }, { "a": "2", "b": "2" } ] }'
2-1 gets inserted, not 2-2 due to primary key.
Benthos internally logs a single error with "Failed to send message", but the fallback treats the whole batch as a failure and thus logs a duplicate key for both 2-1 and 2-2, even though 2-1 got inserted correctly.
ERRO Failed to send message to sql_raw: pq: duplicate key value violates unique constraint "table_x_pkey"  @service=benthos label="" path=root.output.fallback.0
ERRO ERROR: pq: duplicate key value violates unique constraint "table_x_pkey", Content: {"a":"2","b":"1"}  @service=benthos label="" path=root.output.fallback.1.processors.0
ERRO ERROR: pq: duplicate key value violates unique constraint "table_x_pkey", Content: {"a":"2","b":"2"}  @service=benthos label="" path=root.output.fallback.1.processors.0

curl --request POST --url http://localhost:4195/ingest --header 'Content-Type: application/json' --data '{ "data": [ { "a": "1", "b": "1" }, { "a": "5", "b": "5" } ] }'
None of 1-1 or 5-5 gets inserted
Again, Benthos logs a duplicate key error for both.
ERRO Failed to send message to sql_raw: pq: duplicate key value violates unique constraint "table_x_pkey"  @service=benthos label="" path=root.output.fallback.0
ERRO ERROR: pq: duplicate key value violates unique constraint "table_x_pkey", Content: {"a":"1","b":"1"}  @service=benthos label="" path=root.output.fallback.1.processors.0
ERRO ERROR: pq: duplicate key value violates unique constraint "table_x_pkey", Content: {"a":"5","b":"5"}  @service=benthos label="" path=root.output.fallback.1.processors.0

Expected
I expect

Benthos to treat the messages the same regardless which order they come in. There should be a way of configuring to either commit all messages or none.
Better examples in the documentation for fallback for treating the errors of individual messages separately.

Benthos configuration
input:
  http_server:
    path: /ingest
    allowed_verbs:
      - POST
    timeout: 5s
    sync_response:
      status: '${! meta("http_status_code").or("500") }'
  processors:
    - log:
        level: DEBUG
        message: 'Received API message: ${! content() }, Meta: ${! meta() }'

    - mapping: |
        root = this.data

    - unarchive:
        format: json_array
    - catch:
      - log:
          level: ERROR
          message: 'Dropping message due to invalid JSON: ${! meta("raw_message") }'
      - bloblang: |
          meta http_status_code = "400"
      - sync_response: {}

output:
  fallback:
    - sql_raw:
        driver: postgres
        dsn: "postgres://postgres:postgres@localhost:5432/cloud-kpi-data-service?sslmode=disable"
        query: |
          INSERT INTO table_x (id, a, b)
          VALUES ($1, $2, $3)
        args_mapping: |
          root = [
            "%v_%v".format(this.a, this.b),
            this.a,
            this.b
          ]
        init_statement: |
          CREATE TABLE IF NOT EXISTS table_x (
            id TEXT PRIMARY KEY,
            a TEXT NOT NULL,
            b TEXT NOT NULL
          );

      processors:
        - log:
            level: DEBUG
            message: "Inserting into table A: ${! content() }"

    - sync_response: {}
      processors:
      - log:
          level: ERROR
          message: 'ERROR: ${! meta("fallback_error") }, Content: ${! content() }'
      - mapping: |
          meta http_status_code = "500"
          root = {"error": "Failed inserting metrics into the database", "details": meta("fallback_error"), "message": content() }

Closing comment (2024-02-03T23:39:36Z): Hey @mwindmark, I believe this is a duplicate of #1677. If you only need a simple query, in your case I think it should be possible to use sql_insert and that one has the desired semantics. With sql_raw it's a bit difficult, because what you need is this query INSERT INTO table_x(id, a, b) VALUES('1_2', '1', '2'), ('2_2', '2', '2') and the number of tuples after VALUES depends on the batch size, which makes it extra-annoying. One can use Bloblang to craft such a query dynamically and set unsafe_dynamic_query: true to achieve the desired behaviour, but it's a bit cumbersome. Alternatively, at least some SQL drivers should support passing an array of values to each tuple element when calling db.ExecContext(...), but it requires some investigation to have it work similarly to sql_insert.
LE: I shared a bloblang-based workaround in a comment under #1677.
LLE: Here's a Postgres-specific approach which uses arrays: https://klotzandrew.com/blog/postgres-passing-65535-parameter-limit