Repository: elastic/integrations
Pull request #13099: [aws] Set event.type and event.action fields in vpcflow logs
Author: login: brijesh-elastic
Labels: name: Integration:aws
description: AWS
name: Team:Security-Service Integrations
description: Security Service Integrations team [elastic/security-service-integrations]
name: Team:Security-External Integrations
description: Label for the Security External Integrations team
name: mapping/pipeline issue
name: Team:SDE-Crest
description: Crest developers on the Security Integrations team [elastic/sit-crest-contractors]
Created: 2025-03-13T10:27:46Z | Merged: 2025-03-18T13:19:29Z

<!-- Type of change
Please label this PR with one of the following labels, depending on the scope of your change:
- Bug
- Enhancement
- Breaking change
- Deprecation
-->

## Proposed commit message
Set `event.type` and `event.action` values based on `aws.vpcflow.action`.
See reference : https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html

## Checklist

- [x] I have reviewed [tips for building integrations](https://github.com/elastic/integrations/blob/main/docs/tips_for_building_integrations.md) and this pull request is aligned with them.
- [x] I have verified that all data streams collect metrics or logs.
- [x] I have added an entry to my package's `changelog.yml` file.
- [ ] I have verified that Kibana version constraints are current according to [guidelines](https://github.com/elastic/elastic-package/blob/master/docs/howto/stack_version_support.md#when-to-update-the-condition).
- [ ] I have verified that any added dashboard complies with Kibana's [Dashboard good practices](https://docs.elastic.dev/ux-guidelines/data-viz/dashboard-good-practices) 


## How to test this PR locally

- Clone integrations repo.
- Install elastic package locally.
- Start elastic stack using elastic-package.
- Move to integrations/packages/aws directory.
- Run the following command to run tests.
> elastic-package test

## Related issues

- Closes #5478 



Linked issue #5478: aws vpcflow integration should properly set event.type
For aws vpcflow integration with aws.vpcflow.action:REJECT, I only see event.outcome:failure.  It should set event.type based on aws.vpcflow.action.

aws.vpcflow.action can be ACCEPT or REJECT according to https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html

@jamiehynds 
Linked issue comments: user: login: elasticmachine
created_at: 2023-03-08T14:35:20Z
body: Pinging @elastic/security-external-integrations (Team:Security-External Integrations)
author_association: NONE

Diff:
diff --git a/packages/aws/changelog.yml b/packages/aws/changelog.yml
index abbc033f8e1..7ea0cd5d5d7 100644
--- a/packages/aws/changelog.yml
+++ b/packages/aws/changelog.yml
@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.43.0"
+  changes:
+    - description: Set `event.type` and `event.action` fields in vpcflow logs.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/13099
 - version: "2.42.0"
   changes:
     - description: Add support to configure start_timestamp and ignore_older configurations for AWS S3 backed inputs.
diff --git a/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-extra-samples.log-expected.json b/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-extra-samples.log-expected.json
index 9e112d0bfd0..5d8a4aa17da 100644
--- a/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-extra-samples.log-expected.json
+++ b/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-extra-samples.log-expected.json
@@ -35,6 +35,7 @@
                 "version": "8.11.0"
             },
             "event": {
+                "action": "ACCEPT",
                 "category": [
                     "network"
                 ],
@@ -44,7 +45,8 @@
                 "outcome": "success",
                 "start": "2016-10-31T11:35:08.000Z",
                 "type": [
-                    "connection"
+                    "connection",
+                    "allowed"
                 ]
             },
             "network": {
@@ -195,6 +197,7 @@
                 "version": "8.11.0"
             },
             "event": {
+                "action": "ACCEPT",
                 "category": [
                     "network"
                 ],
@@ -204,7 +207,8 @@
                 "outcome": "success",
                 "start": "2014-12-14T04:06:50.000Z",
                 "type": [
-                    "connection"
+                    "connection",
+                    "allowed"
                 ]
             },
             "network": {
@@ -294,6 +298,7 @@
                 "version": "8.11.0"
             },
             "event": {
+                "action": "REJECT",
                 "category": [
                     "network"
                 ],
@@ -303,7 +308,8 @@
                 "outcome": "failure",
                 "start": "2014-12-14T04:06:50.000Z",
                 "type": [
-                    "connection"
+                    "connection",
+                    "denied"
                 ]
             },
             "network": {
@@ -375,6 +381,7 @@
                 "version": "8.11.0"
             },
             "event": {
+                "action": "ACCEPT",
                 "category": [
                     "network"
                 ],
@@ -384,7 +391,8 @@
                 "outcome": "success",
                 "start": "2015-05-29T16:30:27.000Z",
                 "type": [
-                    "connection"
+                    "connection",
+                    "allowed"
                 ]
             },
             "network": {
@@ -474,6 +482,7 @@
                 "version": "8.11.0"
             },
             "event": {
+                "action": "REJECT",
                 "category": [
                     "network"
                 ],
@@ -483,7 +492,8 @@
                 "outcome": "failure",
                 "start": "2015-05-29T16:31:34.000Z",
                 "type": [
-                    "connection"
+                    "connection",
+                    "denied"
                 ]
             },
             "network": {
@@ -512,4 +522,4 @@
             ]
         }
     ]
-}
\ No newline at end of file
+}
diff --git a/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-tcp-flag-sequence.log-expected.json b/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-tcp-flag-sequence.log-expected.json
index d7d092f63a2..00c5609d7d5 100644
--- a/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-tcp-flag-sequence.log-expected.json
+++ b/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-tcp-flag-sequence.log-expected.json
@@ -39,6 +39,7 @@
                 "version": "8.11.0"
             },
             "event": {
+                "action": "ACCEPT",
                 "category": [
                     "network"
                 ],
@@ -48,7 +49,8 @@
                 "outcome": "success",
                 "start": "2019-08-26T19:47:55.000Z",
                 "type": [
-                    "connection"
+                    "connection",
+                    "allowed"
                 ]
             },
             "network": {
@@ -177,4 +179,4 @@
             ]
         }
     ]
-}
\ No newline at end of file
+}
diff --git a/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-v5-all-fields.log-expected.json b/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-v5-all-fields.log-expected.json
index 9e82baf7f28..5db3c34717f 100644
--- a/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-v5-all-fields.log-expected.json
+++ b/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-v5-all-fields.log-expected.json
@@ -45,6 +45,7 @@
                 "version": "8.11.0"
             },
             "event": {
+                "action": "REJECT",
                 "category": [
                     "network"
                 ],
@@ -54,7 +55,8 @@
                 "outcome": "failure",
                 "start": "1975-03-14T03:34:26.000Z",
                 "type": [
-                    "connection"
+                    "connection",
+                    "denied"
                 ]
             },
             "network": {
@@ -102,4 +104,4 @@
             ]
         }
     ]
-}
\ No newline at end of file
+}
diff --git a/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-with-message-field.log-expected.json b/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-with-message-field.log-expected.json
index fcadc8fbc74..1119fccc126 100644
--- a/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-with-message-field.log-expected.json
+++ b/packages/aws/data_stream/vpcflow/_dev/test/pipeline/test-with-message-field.log-expected.json
@@ -36,4 +36,4 @@
             ]
         }
     ]
-}
\ No newline at end of file
+}
diff --git a/packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml b/packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml
index 3d5d4bc5ebb..0cae3876ee1 100644
--- a/packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml
@@ -114,6 +114,18 @@ processors:
       field: event.outcome
       value: failure
       if: ctx.aws?.vpcflow?.action == "REJECT"
+  - append:
+      field: event.type
+      value: allowed
+      if: ctx.aws?.vpcflow?.action == "ACCEPT"
+  - append:
+      field: event.type
+      value: denied
+      if: ctx.aws?.vpcflow?.action == "REJECT"
+  - set:
+      field: event.action
+      copy_from: aws.vpcflow.action
+      ignore_empty_value: true
   - rename:
       field: aws.vpcflow.srcaddr
       target_field: source.address
diff --git a/packages/aws/data_stream/vpcflow/sample_event.json b/packages/aws/data_stream/vpcflow/sample_event.json
index 1ab87d5dff1..991572dd071 100644
--- a/packages/aws/data_stream/vpcflow/sample_event.json
+++ b/packages/aws/data_stream/vpcflow/sample_event.json
@@ -1,17 +1,17 @@
 {
     "@timestamp": "2014-12-14T04:07:50.000Z",
     "agent": {
-        "ephemeral_id": "1ffa6099-2893-4e54-8f41-e84bf2244161",
-        "id": "acba78ef-1401-4689-977c-d8c2e5d6a8fa",
-        "name": "docker-fleet-agent",
+        "ephemeral_id": "884f01cb-3cd3-4f05-a4c3-c97c0eccb33d",
+        "id": "6f7e7793-7439-41de-9909-93ced0496b71",
+        "name": "elastic-agent-64848",
         "type": "filebeat",
-        "version": "8.10.1"
+        "version": "8.18.0"
     },
     "aws": {
         "s3": {
             "bucket": {
-                "arn": "arn:aws:s3:::elastic-package-aws-bucket-63461",
-                "name": "elastic-package-aws-bucket-63461"
+                "arn": "arn:aws:s3:::elastic-package-aws-bucket-51558",
+                "name": "elastic-package-aws-bucket-51558"
             },
             "object": {
                 "key": "extra-samples.log"
@@ -34,7 +34,7 @@
     },
     "data_stream": {
         "dataset": "aws.vpcflow",
-        "namespace": "ep",
+        "namespace": "18794",
         "type": "logs"
     },
     "destination": {
@@ -64,24 +64,26 @@
         "version": "8.11.0"
     },
     "elastic_agent": {
-        "id": "acba78ef-1401-4689-977c-d8c2e5d6a8fa",
-        "snapshot": false,
-        "version": "8.10.1"
+        "id": "6f7e7793-7439-41de-9909-93ced0496b71",
+        "snapshot": true,
+        "version": "8.18.0"
     },
     "event": {
+        "action": "ACCEPT",
         "agent_id_status": "verified",
         "category": [
             "network"
         ],
         "dataset": "aws.vpcflow",
         "end": "2014-12-14T04:07:50.000Z",
-        "ingested": "2023-11-07T14:31:30Z",
+        "ingested": "2025-03-13T10:01:27Z",
         "kind": "event",
         "original": "2 123456789010 eni-1235b8ca123456789 89.160.20.156 89.160.20.156 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK",
         "outcome": "success",
         "start": "2014-12-14T04:06:50.000Z",
         "type": [
-            "connection"
+            "connection",
+            "allowed"
         ]
     },
     "input": {
@@ -89,7 +91,7 @@
     },
     "log": {
         "file": {
-            "path": "https://elastic-package-aws-bucket-63461.s3.us-east-1.amazonaws.com/extra-samples.log"
+            "path": "https://elastic-package-aws-bucket-51558.s3.us-east-1.amazonaws.com/extra-samples.log"
         },
         "offset": 338
     },
@@ -137,4 +139,4 @@
         "forwarded",
         "aws-vpcflow"
     ]
-}
\ No newline at end of file
+}
diff --git a/packages/aws/docs/vpcflow.md b/packages/aws/docs/vpcflow.md
index 30abf788f63..fa36d7d1775 100644
--- a/packages/aws/docs/vpcflow.md
+++ b/packages/aws/docs/vpcflow.md
@@ -129,17 +129,17 @@ An example event for `vpcflow` looks as following:
 {
     "@timestamp": "2014-12-14T04:07:50.000Z",
     "agent": {
-        "ephemeral_id": "1ffa6099-2893-4e54-8f41-e84bf2244161",
-        "id": "acba78ef-1401-4689-977c-d8c2e5d6a8fa",
-        "name": "docker-fleet-agent",
+        "ephemeral_id": "884f01cb-3cd3-4f05-a4c3-c97c0eccb33d",
+        "id": "6f7e7793-7439-41de-9909-93ced0496b71",
+        "name": "elastic-agent-64848",
         "type": "filebeat",
-        "version": "8.10.1"
+        "version": "8.18.0"
     },
     "aws": {
         "s3": {
             "bucket": {
-                "arn": "arn:aws:s3:::elastic-package-aws-bucket-63461",
-                "name": "elastic-package-aws-bucket-63461"
+                "arn": "arn:aws:s3:::elastic-package-aws-bucket-51558",
+                "name": "elastic-package-aws-bucket-51558"
             },
             "object": {
                 "key": "extra-samples.log"
@@ -162,7 +162,7 @@ An example event for `vpcflow` looks as following:
     },
     "data_stream": {
         "dataset": "aws.vpcflow",
-        "namespace": "ep",
+        "namespace": "18794",
         "type": "logs"
     },
     "destination": {
@@ -192,24 +192,26 @@ An example event for `vpcflow` looks as following:
         "version": "8.11.0"
     },
     "elastic_agent": {
-        "id": "acba78ef-1401-4689-977c-d8c2e5d6a8fa",
-        "snapshot": false,
-        "version": "8.10.1"
+        "id": "6f7e7793-7439-41de-9909-93ced0496b71",
+        "snapshot": true,
+        "version": "8.18.0"
     },
     "event": {
+        "action": "ACCEPT",
         "agent_id_status": "verified",
         "category": [
             "network"
         ],
         "dataset": "aws.vpcflow",
         "end": "2014-12-14T04:07:50.000Z",
-        "ingested": "2023-11-07T14:31:30Z",
+        "ingested": "2025-03-13T10:01:27Z",
         "kind": "event",
         "original": "2 123456789010 eni-1235b8ca123456789 89.160.20.156 89.160.20.156 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK",
         "outcome": "success",
         "start": "2014-12-14T04:06:50.000Z",
         "type": [
-            "connection"
+            "connection",
+            "allowed"
         ]
     },
     "input": {
@@ -217,7 +219,7 @@ An example event for `vpcflow` looks as following:
     },
     "log": {
         "file": {
-            "path": "https://elastic-package-aws-bucket-63461.s3.us-east-1.amazonaws.com/extra-samples.log"
+            "path": "https://elastic-package-aws-bucket-51558.s3.us-east-1.amazonaws.com/extra-samples.log"
         },
         "offset": 338
     },
diff --git a/packages/aws/manifest.yml b/packages/aws/manifest.yml
index 30ac9ebd2b5..fbc60b055d2 100644
--- a/packages/aws/manifest.yml
+++ b/packages/aws/manifest.yml
@@ -1,7 +1,7 @@
 format_version: 3.3.1
 name: aws
 title: AWS
-version: 2.42.0
+version: 2.43.0
 description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.
 type: integration
 categories:


Reviews:
user: login: Kavindu-Dodan
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T22:09:35Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T23:17:28Z
user: login: kcreddy
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T08:32:09Z

Review comments:
diff_hunk: @@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.42.1"
path: packages/aws/changelog.yml
user: login: efd6
body: ```suggestion
- version: "2.43.0"
```
Enhancements bump minor.
created_at: 2025-03-13T23:16:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.42.1"
+  changes:
+    - description: Set event.type and event.action fields in vpcflow logs.
path: packages/aws/changelog.yml
user: login: efd6
body: Since we need to change the version, let's also
```suggestion
    - description: Set `event.type` and `event.action` fields in vpcflow logs.
```
created_at: 2025-03-13T23:17:24Z
author_association: CONTRIBUTOR

PR comments:
user: login: elasticmachine
created_at: 2025-03-13T10:27:49Z
body: Pinging @elastic/security-external-integrations (Team:Security-External Integrations)
author_association: NONE
user: login: elasticmachine
created_at: 2025-03-13T14:40:25Z
body: Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
author_association: NONE
user: login: elastic-vault-github-plugin-prod[bot]
created_at: 2025-03-14T05:04:47Z
body: ### :rocket: Benchmarks report


#### Package `aws` :&#43;1:(5) :green_heart:(3) :broken_heart:(11)
<details>
<summary>Expand to view</summary>

Data stream | Previous EPS | New EPS | Diff (%) | Result
----------- | ------------ | ------- | -------- | ------
`guardduty` | 1519.76 | 1011.12 | -508.64 (-33.47%) | :broken_heart:
`inspector` | 3086.42 | 1814.88 | -1271.54 (-41.2%) | :broken_heart:
`route53_public_logs` | 15151.52 | 12345.68 | -2805.84 (-18.52%) | :broken_heart:
`route53_resolver_logs` | 6289.31 | 5050.51 | -1238.8 (-19.7%) | :broken_heart:
`securityhub_findings` | 1834.86 | 1390.82 | -444.04 (-24.2%) | :broken_heart:
`securityhub_insights` | 1418.44 | 892.86 | -525.58 (-37.05%) | :broken_heart:
`apigateway_logs` | 13333.33 | 7042.25 | -6291.08 (-47.18%) | :broken_heart:
`cloudtrail` | 2070.39 | 1600 | -470.39 (-22.72%) | :broken_heart:
`cloudwatch_logs` | 500000 | 200000 | -300000 (-60%) | :broken_heart:
`ec2_logs` | 41666.67 | 29411.76 | -12254.91 (-29.41%) | :broken_heart:
`ec2_metrics` | 35714.29 | 28571.43 | -7142.86 (-20%) | :broken_heart:
</details>


To see the full report comment with `/test benchmark fullreport`
<!--COMMENT_GENERATED_WITH_ID_benchmark-report-->
author_association: NONE
user: login: elastic-sonarqube[bot]
created_at: 2025-03-14T05:05:26Z
body: ## [![Quality Gate passed](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonar.elastic.dev/dashboard?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099) **Quality Gate passed**  
Issues  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099&resolved=false) [0 New issues](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099&resolved=false)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/fixed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&fixedInPullRequest=13099) [0 Fixed issues](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&fixedInPullRequest=13099)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/accepted-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099&issueStatuses=ACCEPTED) [0 Accepted issues](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099&issueStatuses=ACCEPTED)

Measures  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/security_hotspots?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099) [0 Security Hotspots](https://sonar.elastic.dev/security_hotspots?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/component_measures?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099&metric=new_coverage&view=list) [100.0% Coverage on New Code](https://sonar.elastic.dev/component_measures?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099&metric=new_coverage&view=list)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/component_measures?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099&metric=new_duplicated_lines_density&view=list) [0.0% Duplication on New Code](https://sonar.elastic.dev/component_measures?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube](https://sonar.elastic.dev/dashboard?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13099)


author_association: NONE
user: login: elasticmachine
created_at: 2025-03-14T05:06:02Z
body: ## :green_heart: Build Succeeded
* [Buildkite Build](https://buildkite.com/elastic/integrations/builds/23384)
* Commit: 04d3f7879e8103a96d0de0eb00adebe81c879ebf

### History
* :green_heart: [Build #23346](https://buildkite.com/elastic/integrations/builds/23346) succeeded 475ca729c9617cf9eb38c33d606c9ba76bb64b52

cc @brijesh-elastic

<!--buildkite-pr-comment-integrations
{"builds":[{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/integrations/builds/23384","number":23384,"commit":"04d3f7879e8103a96d0de0eb00adebe81c879ebf"},{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/integrations/builds/23346","number":23346,"commit":"475ca729c9617cf9eb38c33d606c9ba76bb64b52"}],"number":23384}
buildkite-pr-comment-->
author_association: NONE
user: login: elastic-vault-github-plugin-prod[bot]
created_at: 2025-03-18T13:39:21Z
body: Package aws - 2.43.0 containing this change is available at https://epr.elastic.co/package/aws/2.43.0/
author_association: NONE