Repository: netobserv/flowlogs-pipeline
Pull request #601 (2024-02-20 09:01:30)

NETOBSERV-1466 Update eBPF dependency for FLP
## Description

See eBPF update https://github.com/netobserv/netobserv-ebpf-agent/pull/275 and followup to remove Duplicate field https://github.com/netobserv/netobserv-ebpf-agent/pull/276

## Dependencies

Update go mod once eBPF PR is merged

## Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

* [x] Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
* [x] Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix _(in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes)._
* [ ] Does this PR require product documentation?
  * [ ] If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
* [ ] Does this PR require a product release notes entry?
  * [ ] If so, fill in "Release Note Text" in the JIRA.
* [ ] Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
  * [ ] If so, make sure it is described in the JIRA ticket.
* QE requirements (check 1 from the list):
  * [x] Standard QE validation, with pre-merge tests unless stated otherwise.
  * [ ] Regression tests only (e.g. refactoring with no user-facing change).
  * [ ] No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).


openshift-ci[bot]: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
**Once this PR has been reviewed and has the lgtm label**, please ask for approval from [jpinsonneau](https://github.com/jpinsonneau). For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=netobserv%2Fflowlogs-pipeline).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/netobserv/flowlogs-pipeline/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["jpinsonneau"]} -->
openshift-ci[bot]: @jpinsonneau: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
ci/prow/unit | a7f3f6af89bdafcb61f9b54bd84afcc710f59c8a | [link](https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/netobserv_flowlogs-pipeline/601/pull-ci-netobserv-flowlogs-pipeline-main-unit/1760278007433400320) | true | `/test unit`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=netobserv&repo=flowlogs-pipeline&pr=601). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:jpinsonneau).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jotak: /lgtm
github-actions[bot]: New image:
quay.io/netobserv/flowlogs-pipeline:013a597

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:
```bash
USER=netobserv VERSION=013a597 make set-flp-image
```

openshift-ci[bot]: New changes are detected. LGTM label has been removed.
openshift-ci[bot]: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
**Once this PR has been reviewed and has the lgtm label**, please ask for approval from [jotak](https://github.com/jotak). For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=netobserv%2Fflowlogs-pipeline).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/netobserv/flowlogs-pipeline/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["jotak"]} -->
openshift-ci[bot]: [APPROVALNOTIFIER] This PR is **APPROVED**

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=netobserv%2Fflowlogs-pipeline).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/netobserv/flowlogs-pipeline/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["jotak"]} -->
openshift-ci[bot]: @jpinsonneau: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
ci/prow/unit | 6d5843ebbedce772e72c7a655abefd2b64993c03 | [link](https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/netobserv_flowlogs-pipeline/601/pull-ci-netobserv-flowlogs-pipeline-main-unit/1767481727325835264) | true | `/test unit`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=netobserv&repo=flowlogs-pipeline&pr=601). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:jpinsonneau).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
openshift-ci[bot]: New changes are detected. LGTM label has been removed.
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/netobserv/flowlogs-pipeline/pull/601?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv) Report
Attention: Patch coverage is `0%` with `6 lines` in your changes are missing coverage. Please review.
> Project coverage is 67.20%. Comparing base [(`df307e0`)](https://app.codecov.io/gh/netobserv/flowlogs-pipeline/commit/df307e0ef280d54ca73d19407809be14652be146?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv) to head [(`8f10ff5`)](https://app.codecov.io/gh/netobserv/flowlogs-pipeline/pull/601?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv).

| [Files](https://app.codecov.io/gh/netobserv/flowlogs-pipeline/pull/601?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv) | Patch % | Lines |
|---|---|---|
| [pkg/pipeline/write/write\_ipfix.go](https://app.codecov.io/gh/netobserv/flowlogs-pipeline/pull/601?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv#diff-cGtnL3BpcGVsaW5lL3dyaXRlL3dyaXRlX2lwZml4Lmdv) | 0.00% | [6 Missing :warning: ](https://app.codecov.io/gh/netobserv/flowlogs-pipeline/pull/601?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv) |

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #601   +/-   ##
=======================================
  Coverage   67.19%   67.20%           
=======================================
  Files         104      104           
  Lines        7459     7461    +2     
=======================================
+ Hits         5012     5014    +2     
- Misses       2144     2145    +1     
+ Partials      303      302    -1     
```

| [Flag](https://app.codecov.io/gh/netobserv/flowlogs-pipeline/pull/601/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/netobserv/flowlogs-pipeline/pull/601/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv) | `67.20% <0.00%> (+<0.01%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv#carryforward-flags-in-the-pull-request-comment) to find out more.


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/netobserv/flowlogs-pipeline/pull/601?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv).


Patch:
diff --git a/go.mod b/go.mod
index ff852a37..92590646 100644
--- a/go.mod
+++ b/go.mod
@@ -16,7 +16,7 @@ require (
 	github.com/mitchellh/mapstructure v1.5.0
 	github.com/netobserv/gopipes v0.3.0
 	github.com/netobserv/loki-client-go v0.0.0-20220927092034-f37122a54500
-	github.com/netobserv/netobserv-ebpf-agent v0.3.3
+	github.com/netobserv/netobserv-ebpf-agent v0.3.4-0.20240312085757-e9bf0d0d5a09
 	github.com/netsampler/goflow2 v1.3.7
 	github.com/pkg/errors v0.9.1
 	github.com/prometheus/client_golang v1.19.0
diff --git a/go.sum b/go.sum
index 4176ed43..ff729249 100644
--- a/go.sum
+++ b/go.sum
@@ -631,8 +631,8 @@ github.com/netobserv/gopipes v0.3.0 h1:IYmPnnAVCdSK7VmHmpFhrVBOEm45qpgbZmJz1sSW+
 github.com/netobserv/gopipes v0.3.0/go.mod h1:N7/Gz05EOF0CQQSKWsv3eof22Cj2PB08Pbttw98YFYU=
 github.com/netobserv/loki-client-go v0.0.0-20220927092034-f37122a54500 h1:RmnoJe/ci5q+QdM7upFdxiU+D8F3L3qTd5wXCwwHefw=
 github.com/netobserv/loki-client-go v0.0.0-20220927092034-f37122a54500/go.mod h1:LHXpc5tjKvsfZn0pwLKrvlgEhZcCaw3Di9mUEZGAI4E=
-github.com/netobserv/netobserv-ebpf-agent v0.3.3 h1:4e/mDVhSTd8JzaqKapaQ78BFLJP/GYpYWQ/LHPArJeI=
-github.com/netobserv/netobserv-ebpf-agent v0.3.3/go.mod h1:R3zRIhu5faj91s9pCU735oULLC+oBDPzA/xVwiUJ9PY=
+github.com/netobserv/netobserv-ebpf-agent v0.3.4-0.20240312085757-e9bf0d0d5a09 h1:9Y7nPXhMc8PToNX4aQltGNcpSDBehWoTHlg/LWToils=
+github.com/netobserv/netobserv-ebpf-agent v0.3.4-0.20240312085757-e9bf0d0d5a09/go.mod h1:W26Mf03LriOdylI/3T0yKgmU3bR4gy/ihzRd0SzJb3w=
 github.com/netsampler/goflow2 v1.3.7 h1:XZaTy8kkMnGXpJ9hS3KbO1McyrFTpVNhVFEx9rNhMmc=
 github.com/netsampler/goflow2 v1.3.7/go.mod h1:4UZsVGVAs//iMCptUHn3WNScztJeUhZH7kDW2+/vDdQ=
 github.com/oklog/oklog v0.3.2/go.mod h1:FCV+B7mhrz4o+ueLpx+KqkyXRGMWOYEvfiXtdGtbWGs=
diff --git a/pkg/pipeline/write/write_ipfix.go b/pkg/pipeline/write/write_ipfix.go
index e76bb9e1..cf984a26 100644
--- a/pkg/pipeline/write/write_ipfix.go
+++ b/pkg/pipeline/write/write_ipfix.go
@@ -246,8 +246,9 @@ func setStandardIEValue(record config.GenericMap, ieValPtr *entities.InfoElement
 			return fmt.Errorf("unable to find ethernet type (Etype) in record")
 		}
 	case "flowDirection":
-		if record["FlowDirection"] != nil {
-			ieVal.SetUnsigned8Value(uint8(record["FlowDirection"].(int)))
+		dirs := record["IfDirections"].([]int)
+		if len(dirs) > 0 {
+			ieVal.SetUnsigned8Value(uint8(dirs[0]))
 		} else {
 			return fmt.Errorf("unable to find flow direction (flowDirection) in record")
 		}
@@ -336,8 +337,9 @@ func setStandardIEValue(record config.GenericMap, ieValPtr *entities.InfoElement
 			return fmt.Errorf("unable to find packets in record")
 		}
 	case "interfaceName":
-		if record["Interface"] != nil {
-			ieVal.SetStringValue(record["Interface"].(string))
+		interfaces := record["Interfaces"].([]string)
+		if len(interfaces) > 0 {
+			ieVal.SetStringValue(interfaces[0])
 		} else {
 			return fmt.Errorf("unable to find interface in record")
 		}
diff --git a/vendor/github.com/netobserv/netobserv-ebpf-agent/pkg/decode/decode_protobuf.go b/vendor/github.com/netobserv/netobserv-ebpf-agent/pkg/decode/decode_protobuf.go
index 4d0ea2bb..0bd286c6 100644
--- a/vendor/github.com/netobserv/netobserv-ebpf-agent/pkg/decode/decode_protobuf.go
+++ b/vendor/github.com/netobserv/netobserv-ebpf-agent/pkg/decode/decode_protobuf.go
@@ -46,18 +46,19 @@ func PBFlowToMap(flow *pbflow.Record) config.GenericMap {
 		return config.GenericMap{}
 	}
 	out := config.GenericMap{
-		"FlowDirection":   int(flow.Direction.Number()),
 		"SrcMac":          macToStr(flow.DataLink.GetSrcMac()),
 		"DstMac":          macToStr(flow.DataLink.GetDstMac()),
 		"Etype":           flow.EthProtocol,
-		"Duplicate":       flow.Duplicate,
 		"TimeFlowStartMs": flow.TimeFlowStart.AsTime().UnixMilli(),
 		"TimeFlowEndMs":   flow.TimeFlowEnd.AsTime().UnixMilli(),
 		"TimeReceived":    time.Now().Unix(),
-		"Interface":       flow.Interface,
 		"AgentIP":         ipToStr(flow.AgentIp),
 	}
 
+	if flow.Duplicate {
+		out["Duplicate"] = true
+	}
+
 	if flow.Bytes != 0 {
 		out["Bytes"] = flow.Bytes
 	}
@@ -65,17 +66,20 @@ func PBFlowToMap(flow *pbflow.Record) config.GenericMap {
 	if flow.Packets != 0 {
 		out["Packets"] = flow.Packets
 	}
-	var interfaces []interface{}
-	var flowDirections []interface{}
 
+	var interfaces []string
+	var directions []int
 	if len(flow.GetDupList()) != 0 {
 		for _, entry := range flow.GetDupList() {
 			interfaces = append(interfaces, entry.Interface)
-			flowDirections = append(flowDirections, entry.Direction)
+			directions = append(directions, int(entry.Direction))
 		}
-		out["Interfaces"] = interfaces
-		out["FlowDirections"] = flowDirections
+	} else {
+		interfaces = append(interfaces, flow.Interface)
+		directions = append(directions, int(flow.Direction))
 	}
+	out["Interfaces"] = interfaces
+	out["IfDirections"] = directions
 
 	ethType := ethernet.EtherType(flow.EthProtocol)
 	if ethType == ethernet.EtherTypeIPv4 || ethType == ethernet.EtherTypeIPv6 {
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 9eb96574..7a99cbdf 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -250,7 +250,7 @@ github.com/netobserv/loki-client-go/pkg/labelutil
 github.com/netobserv/loki-client-go/pkg/logproto
 github.com/netobserv/loki-client-go/pkg/metric
 github.com/netobserv/loki-client-go/pkg/urlutil
-# github.com/netobserv/netobserv-ebpf-agent v0.3.3
+# github.com/netobserv/netobserv-ebpf-agent v0.3.4-0.20240312085757-e9bf0d0d5a09
 ## explicit; go 1.20
 github.com/netobserv/netobserv-ebpf-agent/pkg/decode
 github.com/netobserv/netobserv-ebpf-agent/pkg/grpc


Test patch:
diff --git a/pkg/pipeline/pipeline_test.go b/pkg/pipeline/pipeline_test.go
index d9742f2d..fa8476fc 100644
--- a/pkg/pipeline/pipeline_test.go
+++ b/pkg/pipeline/pipeline_test.go
@@ -228,7 +228,7 @@ parameters:
 	assert.NotZero(t, capturedRecord["TimeReceived"])
 	delete(capturedRecord, "TimeReceived")
 	assert.EqualValues(t, map[string]interface{}{
-		"FlowDirection":          float64(1),
+		"IfDirections":           []interface{}{float64(1)},
 		"Bytes":                  float64(456),
 		"SrcAddr":                "1.2.3.4",
 		"DstAddr":                "5.6.7.8",
@@ -237,13 +237,12 @@ parameters:
 		"SrcMac":                 "01:02:03:04:05:06",
 		"SrcPort":                float64(23000),
 		"DstPort":                float64(443),
-		"Duplicate":              false,
 		"Etype":                  float64(2048),
 		"Packets":                float64(123),
 		"Proto":                  float64(17),
 		"TimeFlowStartMs":        float64(startTime.UnixMilli()),
 		"TimeFlowEndMs":          float64(endTime.UnixMilli()),
-		"Interface":              "eth0",
+		"Interfaces":             []interface{}{"eth0"},
 		"AgentIP":                "10.11.12.13",
 		"PktDropBytes":           float64(100),
 		"PktDropPackets":         float64(10),
diff --git a/pkg/pipeline/write/testnorace/write_ipfix_test.go b/pkg/pipeline/write/testnorace/write_ipfix_test.go
index 075764d4..13d5aa94 100644
--- a/pkg/pipeline/write/testnorace/write_ipfix_test.go
+++ b/pkg/pipeline/write/testnorace/write_ipfix_test.go
@@ -147,7 +147,7 @@ func matchElement(t *testing.T, element entities.InfoElementWithValue, flow conf
 	case "ethernetType":
 		assert.Equal(t, flow["Etype"], uint32(element.GetUnsigned16Value()))
 	case "flowDirection":
-		assert.Equal(t, flow["FlowDirection"], int(element.GetUnsigned8Value()))
+		assert.Equal(t, flow["IfDirections"], []int{int(element.GetUnsigned8Value())})
 	case "protocolIdentifier":
 		assert.Equal(t, flow["Proto"], uint32(element.GetUnsigned8Value()))
 	case "sourceTransportPort":
@@ -163,7 +163,7 @@ func matchElement(t *testing.T, element entities.InfoElementWithValue, flow conf
 	case "packetDeltaCount":
 		assert.Equal(t, flow["Packets"], element.GetUnsigned64Value())
 	case "interfaceName":
-		assert.Equal(t, flow["Interface"], element.GetStringValue())
+		assert.Equal(t, flow["Interfaces"], []string{element.GetStringValue()})
 	case "sourcePodNamespace":
 		assert.Equal(t, flow["SrcK8S_Namespace"], element.GetStringValue())
 	case "sourcePodName":