Repository: argoproj/argo-events

Issue #520: data.body from redis and aws-sqs are different: sqs data.body base64 encoded
Created: 2020-03-03T01:30:56Z
Describe the bug
A sensor receives raw payload from redis gateway but base64 encoded from aws-sqs gateway.
To Reproduce

Create a sensor like this:

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: the-sensor
  labels:
    sensors.argoproj.io/sensor-controller-instanceid: argo-events
spec:
  template:
    spec:
      containers:
        - name: sensor
          image: argoproj/sensor:v0.13.0-rc
          imagePullPolicy: Always
      serviceAccountName: argo-events-sa
  subscription:
    http:
      port: 9300
  dependencies:
    - name: test-dep
      gatewayName: the-gateway
      eventName: example
  triggers:
    - template:
        name: workflow-trigger
        k8s:
          group: argoproj.io
          version: v1alpha1
          resource: workflows
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: event-job-
                namespace: default
              spec:
                entrypoint: whalesay
                arguments:
                  parameters:
                    - name: message
                      value: hello world
                templates:
                  - name: whalesay
                    inputs:
                      parameters:
                        - name: message
                    container:
                      image: docker/whalesay:latest
                      command: [cowsay]
                      args: ["{{inputs.parameters.message}}"]
          parameters:
            - src:
                dependencyName: test-dep
                dataKey: body
              dest: spec.arguments.parameters.0.value

If the-gateway is redis and publish like this:
PUBLISH the_channel "hello from redis"

cowsay: "hello from redis"
But the-gateway is aws-sqs and send message like this:
aws sqs send-message --queue-url="https://queue-url" --message-body="hello from sqs"

cawsay: "aGVsbG8gZnJvbSBzcXM="
"aGVsbG8gZnJvbSBzcXM=" is base64 encoded "hello from sqs"
Expected behavior
I think the body should not to be encoded.
Environment (please complete the following information):

OS: Ubuntu 18.04
Version: v0.13.0-rc

Duplicate issue #549: GCP Pubsub Gateway/Sensor not decoding data?
Created: 2020-03-20T03:52:28Z
Describe the bug
Unsure what changed -- I've had argo-events flows for a while now set up with argoproj/sensor:v0.12.1 and argoproj/gateway-client:v0.12.1 images, but when trying to run my events I'm getting:
ERRO[2020-03-20 03:17:40] failed to operate on the event notification   error="failed to apply filter on Event dependency pubsub-dep"

Looking into it, it seems that argo-events is passing the event data as base64 and not json when running the filters or filling in parameters. When I remove the filter, my workflow gets handed base64 of the json despite using the dataKey source to pass it through.
This exact deployment has been working for months, not sure what changed -- is it possible that the argoproj/xxxx:v0.12.1 images got updated?
Here's the (mostly full) yaml of the event source / gateway / sensor:
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: gcp-pubsub-event-source
  namespace: argo-events
spec:
  type: "pubsub"
  pubSub:
    gcp-es-name:
      topic: "gcp-es-topic"

---

apiVersion: argoproj.io/v1alpha1
kind: Gateway
metadata:
  name: gcp-pubsub-gateway
  labels:
    gateways.argoproj.io/gateway-controller-instanceid: argo-events
    argo-events-gateway-version: v0.12.1
spec:
  replica: 1
  type: pubsub
  eventSourceRef:
    name: gcp-pubsub-event-source
  template:
    metadata:
      name: gcp-pubsub-gateway
      labels:
        gateway-name: gcp-pubsub-gateway
    spec:
      containers:
        - name: gateway-client
          image: argoproj/gateway-client:v0.12.1
          imagePullPolicy: Always
          command: ["/bin/gateway-client"]
        - name: gcp-pubsub-events
          image: argoproj/gcp-pubsub-gateway:v0.12.1
          imagePullPolicy: Always
          command: ["/bin/gcp-pubsub-gateway"]
      serviceAccountName: argo-events-sa
  subscribers:
    http:
      - "http://gcp-pubsub-sensor.argo-events.svc:9300/"

---

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: gcp-pubsub-sensor
  namespace: argo-events
  labels:
    sensors.argoproj.io/sensor-controller-instanceid: argo-events
    argo-events-sensor-version: v0.12.1
spec:
  template:
    spec:
      containers:
        - name: sensor
          image: argoproj/sensor:v0.12.1
          imagePullPolicy: Always
      serviceAccountName: argo-events-sa
  subscription:
    http:
      port: 9300
  dependencies:
    - name: pubsub-dep
      gatewayName: gcp-pubsub-gateway
      eventName: gcp-es-name
      filters:
        name: file-extension-filter
        data:
        - path: name
          type: string
          value:
            - \.tar(\.gz)?$

  triggers:
    - template:
        name: gcp-workflow
        k8s:
          group: argoproj.io
          version: v1alpha1
          resource: workflows
          operation: create
          source:
            # the source to trigger [not relevant]

          parameters:
            - src:
                dependencyName: pubsub-dep
                dataKey: body
              dest: spec.arguments.parameters.0.value

Closing comment (2020-03-20T13:03:31Z): This may be related to #520 . The fix is already available in the master branch and will be released within the next couple of days.