Repository: argoproj/argo-events

Issue #1277: Allow SSL connections to Kafka, even when not using TLS authentication
Created: 2021-07-14T16:16:12Z
Is your feature request related to a problem? Please describe.
The kafka library used by argo-events supports SASL_SCRAM authentication over an SSL connection. See for instance https://github.com/Shopify/sarama/blob/master/examples/sasl_scram_client/main.go
However, when configuring an EventSource for kafka like in the snippet below, the Eventsource pod attempts to connect to Kafka in Plaintext.
 kafka:
    cdm:
      url: blahblah-msk-endpoint.eu-central-1.amazonaws.com:9096
      topic: test
      jsonBody: true
      # partition id
      partition: "1"
      # optional backoff time for connection retries.
      # if not provided, default connection backoff time will be used.
      connectionBackoff:
        # duration in nanoseconds, or strings like "3s", "2m". following value is 10 seconds
        duration: 1000000000 # "10s" is reported invalid here, but that's another issue
        # how many backoffs
        steps: 5
        factor: 2
        jitter: 0.2
     # tls: TLS IS NOT the answer. TLS enables an SSL connection, but it's for client authentication using SSL. Something that MSK supports only with a very expensive private CA. AWS, amirite? 🤷 
      sasl:
        mechanism: SCRAM-SHA-512
        passwordSecret:
          key: password
          name: kafkauser
        userSecret:
          key: user
          name: kafkauser
namespace=orekit-fetcher, eventSourceName=kafka, eventSourceType=kafka, eventName=cdm, level=info, time=2021-07-14T16
:14:23Z, msg=start kafka event source...
namespace=orekit-fetcher, eventSourceName=kafka, eventSourceType=kafka, eventName=cdm, level=info, time=2021-07-14T16
:14:23Z, msg=connecting to Kafka cluster...
namespace=orekit-fetcher, eventSourceName=kafka, eventSourceType=kafka, eventName=cdm, level=error, time=2021-07-14T1
6:14:23Z, msg=failed to start listening eventsource
namespace=orekit-fetcher, eventSourceName=kafka, level=error, time=2021-07-14T16:14:23Z, msg=Erroring out, no active
event server running
namespace=orekit-fetcher, eventSourceName=kafka, level=info, time=2021-07-14T16:14:23Z, msg=exiting eventbus connecti
on daemon...
namespace=orekit-fetcher, eventSourceName=kafka, level=error, time=2021-07-14T16:14:23Z, msg=failed to start
namespace=orekit-fetcher, eventSourceName=kafka, level=error, time=2021-07-14T16:14:23Z, msg=NATS connection lost

AWS MSK does not allow to set up SASL_SCRAM while allowing plaintext connections, therefore I cannot connect to my MSK cluster with an EventSource.
Describe the solution you'd like
I believe Sarama allows to configure a TLS socket, without client authentication (but optional server certificate validation). However, there is no way, that I could find, to configure an EventSource to connect to Kafka with TLS, when not using TLS authentication.
Describe alternatives you've considered
I'm considering temporarily dropping TLS and Authentication in MSK to continue integrating argo-events with our systems, but that's not how I'd like to run in production.


Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

Duplicate issue #2580: Kafka Trigger - `unexpected EOF` error
Created: 2023-04-18T21:30:01Z
Describe the bug
We want to produce messages on our MSK Cluster for S3 Bucket Object Events.
S3 Bucket Object Event -> SNS -> ArgoEvents SNS Event Source -> ArgoEvents JetStream EventBus -> ArgoEvents Sensor w/ Kafka Trigger -> MSK Cluster
We've validated the S3 Bucket Object Event -> ArgoEvents Sensor w/ LogTrigger already.  The missing piece is producing the message on our MSK Cluster.
When attempting to produce a message on our AWS MSK Cluster, we are facing a crypting unexpected EOF (x3) error.
We've already verified routing/security group configurations between our EKS cluster and MSK cluster via AWS Reachability Analyzer AND via other services running on the same EKS cluster that talk to our MSK cluster.
Also tried with only one broker in the kafka.url error but hit the same error.
Error Message:
{
    "level": "info",
    "ts": 1681852034.6398003,
    "logger": "argo-events.sensor",
    "caller": "sensor/trigger_conn.go:348",
    "msg": "Triggering actions after receiving dependency s3-sns-kafka-event-source-dep",
    "sensorName": "s3-sns-kafka-events",
    "triggerName": "kafka-trigger"
} {
    "level": "error",
    "ts": 1681852035.4367816,
    "logger": "argo-events.sensor",
    "caller": "sensors/trigger.go:104",
    "msg": "failed to new a Kafka trigger",
    "sensorName": "s3-sns-kafka-events",
    "triggerName": "kafka-trigger",
    "error": "kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n",
    "stacktrace": "github.com/argoproj/argo-events/sensors.(*SensorContext).GetTrigger\n\t/home/runner/work/argo-events/argo-events/sensors/trigger.go:104\ngithub.com/argoproj/argo-events/sensors.(*SensorContext).triggerOne\n\t/home/runner/work/argo-events/argo-events/sensors/listener.go:377\ngithub.com/argoproj/argo-events/sensors.(*SensorContext).triggerWithRateLimit\n\t/home/runner/work/argo-events/argo-events/sensors/listener.go:354"
} {
    "level": "error",
    "ts": 1681852035.436886,
    "logger": "argo-events.sensor",
    "caller": "sensors/listener.go:356",
    "msg": "Failed to execute a trigger",
    "sensorName": "s3-sns-kafka-events",
    "error": "invalid trigger kafka-trigger, could not find an implementation",
    "triggerName": "kafka-trigger",
    "triggeredBy": ["s3-sns-kafka-event-source-dep"],
    "triggeredByEvents": ["62386564613733322d636161322d346334632d393332302d653264343630623863363632"],
    "stacktrace": "github.com/argoproj/argo-events/sensors.(*SensorContext).triggerWithRateLimit\n\t/home/runner/work/argo-events/argo-events/sensors/listener.go:356"
}
To Reproduce
Steps to reproduce the behavior:

Apply Sensor Configuration

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: s3-sns-kafka-events
spec:
  dependencies:
    - name: s3-sns-kafka-event-source-dep
      eventSourceName: s3-sns-kafka-events
      eventName: s3-sns-events
  triggers:
    - template:
        name: kafka-trigger
        kafka:
          # Kafka URL
          url: b-2.sandbox-msk-cluster.kafka.us-west-2.amazonaws.com:9094,b-3.sandbox-msk-cluster.kafka.us-west-2.amazonaws.com:9094,b-1.sandbox-msk-cluster.kafka.us-west-2.amazonaws.com:9094
          # Name of the topic
          topic: s3-sns-events
          payload:
            - src:
                dependencyName: s3-sns-kafka-event-source-dep
                dataKey: notification.0.s3.object.key
              dest: fileName
            - src:
                dependencyName: s3-sns-kafka-event-source-dep
                dataKey: notification.0.s3.bucket.name
              dest: bucket


Create / Delete S3 Object

Expected behavior
Message produced on MSK Cluster
Environment (please complete the following information):

Kubernetes: v1.23.16-eks-48e63af
ArgoCD: v2.4.11
Argo Events: v1.7.6



Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

Closing comment (2023-04-18T21:48:02Z): This looks like it's related to #1277. Specifically this comment in the thread.
Our MSK Clusters are not using TLS/SASL/SCRAM authentication, but we ARE using TLS Encryption for connections between our brokers/clients.
Amending our Sensor configuration as follows helped moved me along (to a different, but more workable error)
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: s3-sns-kafka-events
spec:
  dependencies:
    - name: s3-sns-kafka-event-source-dep
      eventSourceName: s3-sns-kafka-events
      eventName: s3-sns-events
  triggers:
    - template:
        name: kafka-trigger
        kafka:
          tls:
            insecureSkipVerify: true
          # Kafka URL
          url: b-2.sandbox-msk-cluster.kafka.us-west-2.amazonaws.com:9094,b-3.sandbox-msk-cluster.kafka.us-west-2.amazonaws.com:9094,b-1.sandbox-msk-cluster.kafka.us-west-2.amazonaws.com:9094
          # Name of the topic
          topic: s3-sns-events
          payload:
            - src:
                dependencyName: s3-sns-kafka-event-source-dep
                dataKey: notification.0.s3.object.key
              dest: fileName
            - src:
                dependencyName: s3-sns-kafka-event-source-dep
                dataKey: notification.0.s3.bucket.name
              dest: bucket
New Error:
failed to get value by key: key notification.0.s3.object.key does not exist to in the event payload


Edit: Leveraged Sprig Templates (per #1295) to get a working implementation.

          payload:
            - src:
                dependencyName: s3-sns-kafka-event-source-dep
                dataTemplate: '{{ (first (.Input.body.Message | mustFromJson).Records) | dig "s3" "object" "key" "<missing-s3-fileName>" }}'
              dest: fileName
            - src:
                dependencyName: s3-sns-kafka-event-source-dep
                dataTemplate: '{{ (first (.Input.body.Message | mustFromJson).Records) | dig "s3" "bucket" "name" "<missing-s3-bucket>" }}'
              dest: bucket