Repository: argoproj/argo-events

Issue #1449: Filter issues
Created: 2021-12-05T01:10:46Z
the Filter documentation:

is really incomplete
examples are too simple (it would be good to have a real example based for example on GitHub and GitLab events)
should have a dedicated section and not be placed only under "tutorials", as imo it's a really important feature of Argo Events


I suggest to add an option to change the evaluation process of the list of filters and conditions under each filter, offering possibility to specify if evaluated in AND or in OR, because:

there is no option to have multiple conditions under a filter evaluated in OR. I'm forced to create multiple sensors, increasing complexity (more yaml manifests) and costs of maintainability and resources in the cluster (more sensors = more resource consumption)
Multiple paths could fast become really complex and it's completely not flexible as Expr (see examples of differences below)
declaring or-conditions in Multiple paths is complex and limited, because based on regex



The Filter doc is really incomplete because:

Nowhere is documented that "the list of filters is evaluated in AND condition"
Nowhere is documented that "the list of conditions under a filter is evaluated in AND condition"
There are 4 types of filter, not 3. Expr filter is completely missing and it's one of the most important and flexible
Data filter Multiple path example is not really clear and probably wrong
There is no clear example for Context filter (like for Data filter)
There is no clear example for Time filter (like for Data filter)
There is no real examples or use case, for example filtering an event coming from GitHub or GitLab webhook

"Data filter Multiple paths" is not comparable to "Expr filter", for example:

declaring multiple conditions within a regex becomes fast really complex and error prone
declaring something like b != "" is not so easy together with other conditions
it's not possible to declare conditions in || (OR)
it's not possible to declare a condition like b > 2.5 as number or b == true as bool
no "Parenthesis to control order of evaluation" available (based on https://github.com/Knetic/govaluate)
no "Arrays" available (based on https://github.com/Knetic/govaluate)
For these reasons (and others I didn't list), I don't consider "Data filter Multiple paths" a valid alternative to "Expr filter". And for sure also not a valid, easy, fast alternative to have conditions under a filter evaluated in OR.

@bygui86

Duplicate issue #1442: Sensor expression filter "not really mutual exclusive"
Created: 2021-11-29T13:51:50Z
Describe the bug
I created a Sensor containing some dependencies on different kind of events, coming from an EventSource listening for GitLab webhook events.
I'm trying to filter those events basing on a list of expressions declared under spec.dependencies.[].filters.exprs.[]).
At the first failing expression (spec.dependencies.[].filters.exprs.[].expr), the Sensor just discards the event without evaluating all following expressions in the list.
To Reproduce
It's pretty difficult to reproduce exactly my environment, because it requires:

GitLab account
GitLab repository
GitLab webhook configured to point ArgoEvents EventSource
EventSource deployed and exposed on the internet

Anyway here the steps to reproduce:

Deploy ArgoEvents Sensor
Trigger a GitLab webhook event with a commit or a change to a MR
Look at Sensor logs, searching for something like

{"level":"error","ts":1638190242.5967605,"logger":"argo-events.sensor","caller":"sensors/listener.go:174","msg":"failed to apply filters","sensorName":"sbt-gitlab-mr","error":"path body.object_attributes.oldrev does not exist","stacktrace":"github.com/argoproj/argo-events/sensors.(*SensorContext).listenEvents.func1.2\n\t/home/runner/work/argo-events/argo-events/sensors/listener.go:174\ngithub.com/argoproj/argo-events/eventbus/driver.(*natsStreaming).processEventSourceMsg\n\t/home/runner/work/argo-events/argo-events/eventbus/driver/nats.go:229\ngithub.com/argoproj/argo-events/eventbus/driver.(*natsStreaming).SubscribeEventSources.func1\n\t/home/runner/work/argo-events/argo-events/eventbus/driver/nats.go:150\ngithub.com/nats-io/stan%2ego.(*conn).processMsg\n\t/home/runner/go/pkg/mod/github.com/nats-io/stan.go@v0.6.0/stan.go:863\ngithub.com/nats-io/nats%2ego.(*Conn).waitForMsgs\n\t/home/runner/go/pkg/mod/github.com/nats-io/nats.go@v1.10.0/nats.go:2211"}
Here the Sensor:
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: gitlab-sensor
spec:
  replicas: 1
  dependencies:
    - name: gitlab-dependency
      eventSourceName: gitlab-event-source
      eventName: gitlab-event
      filters:
        exprs:
          # expression 1
          # GitLab webhook event always contains fields "event_type", "object_attributes.work_in_progress", "object_attributes.state", "object_attributes.action", so no issue here
          - expr: event_type == "merge_request" && work_in_progress == false && state == "opened" && action == "open"
            fields:
              - name: event_type
                path: body.event_type
              - name: work_in_progress
                path: body.object_attributes.work_in_progress
              - name: state
                path: body.object_attributes.state
              - name: action
                path: body.object_attributes.action
          # expression 2
          # GitLab webhook event does not always contain field "object_attributes.oldrev", in this case the evaluationa process stops here ignoring all following expressions
          - expr: event_type == "merge_request" && work_in_progress == false && state == "opened" && action == "update" && oldrev != ""
            fields:
              - name: event_type
                path: body.event_type
              - name: work_in_progress
                path: body.object_attributes.work_in_progress
              - name: state
                path: body.object_attributes.state
              - name: action
                path: body.object_attributes.action
              - name: oldrev
                path: body.object_attributes.oldrev
          # expression 3
          # This expression is never evaluated as GitLab webhook event does not always contain fiel "object_attributes.oldrev" and the evaluation process stops before reaching this expression
          - expr: event_type == "merge_request" && work_in_progress == false && state == "opened" && action == "update" && title_previous =~ "Draft:.*"
            fields:
              - name: event_type
                path: body.event_type
              - name: work_in_progress
                path: body.object_attributes.work_in_progress
              - name: state
                path: body.object_attributes.state
              - name: action
                path: body.object_attributes.action
              - name: title_previous
                path: body.changes.title.previous
  triggers:
    - template:
        conditions: gitlab-dependency
        name: gitlab-trigger
        # ...
Expected behaviour
Expressions listed in the filter are mutual exclusive: the first one that matches, triggers corresponding action declared in the Sensor.
This behaviour is fine, as soon as ALL expressions are evaluated.
Right now at the first failure (e.g. "path ... does not exist") the evaluation process gets interrupted and all following expressions are completely ignored. This cannot be the right behaviour, otherwise it does not make sense to allow the declaration of multiple mutual exclusive expressions.
I found the piece of code responsible for filtering events here: https://github.com/argoproj/argo-events/blob/master/sensors/dependencies/filter.go
And here the method:
// filterExpr applies expression based filters against event data
func filterExpr(filters []v1alpha1.ExprFilter, event *v1alpha1.Event) (bool, error) {
	if filters == nil {
		return true, nil
	}
	if event == nil {
		return false, fmt.Errorf("nil event")
	}
	payload := event.Data
	if payload == nil {
		return true, nil
	}
	var js *json.RawMessage
	if err := json.Unmarshal(payload, &js); err != nil {
		return false, err
	}
	var jsData []byte
	jsData, err := json.Marshal(js)
	if err != nil {
		return false, err
	}

	for _, filter := range filters {
		parameters := map[string]interface{}{}
		for _, field := range filter.Fields {
			result := gjson.GetBytes(jsData, field.Path)
			if !result.Exists() {
				return false, fmt.Errorf("path %s does not exist", field.Path)
			}
			parameters[field.Name] = result.Value()
		}
		if len(parameters) == 0 {
			continue
		}
		expr, err := govaluate.NewEvaluableExpression(filter.Expr)
		if err != nil {
			return false, err
		}
		result, err := expr.Evaluate(parameters)
		if err != nil {
			return false, err
		}
		if result == true {
			return true, nil
		}
	}

	return false, nil
}
In my opinion we should fix the method above, like following:
// filterExpr applies expression based filters against event data
func filterExpr(filters []v1alpha1.ExprFilter, event *v1alpha1.Event) (bool, error) {
	if filters == nil {
		return true, nil
	}
	if event == nil {
		return false, fmt.Errorf("nil event")
	}
	payload := event.Data
	if payload == nil {
		return true, nil
	}
	var js *json.RawMessage
	if err := json.Unmarshal(payload, &js); err != nil {
		return false, err
	}
	var jsData []byte
	jsData, err := json.Marshal(js)
	if err != nil {
		return false, err
	}

	errMsgs := make([]string, 0)
	for _, filter := range filters {
		parameters := map[string]interface{}{}
		for _, field := range filter.Fields {
			result := gjson.GetBytes(jsData, field.Path)
			if !result.Exists() {
				errMsgs = append(errMsgs, fmt.Sprintf("path %s does not exist", field.Path))
				continue
			}
			parameters[field.Name] = result.Value()
		}
		if len(parameters) == 0 {
			continue
		}
		expr, err := govaluate.NewEvaluableExpression(filter.Expr)
		if err != nil {
			errMsgs = append(errMsgs, err.Error())
			continue
		}
		result, err := expr.Evaluate(parameters)
		if err != nil {
			errMsgs = append(errMsgs, err.Error())
			continue
		}
		if result == true {
			return true, nil
		}
	}

	return false, errors.New(strings.Join(errMsgs, " / "))
}
We should fix also all other filtering options allowing list of mutual exclusive matches (e.g. data).
Environment (please complete the following information):

Kubernetes:   v1.19.14-gke.1900
Argo:   v3.1.3
Argo Events:   v1.4.0


Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍  reaction to this issue! We often sort issues this way to know what to prioritise.

Closing comment (2021-12-17T15:46:11Z): @whynowy based on our call, I think we should close this as "not really a bug" and proceed with improvements tracked by #1449
what do you think?