Repository: elastic/integrations
Pull request #13698: [Cloudflare] Use CEL input to add support for token authorization in Audit log data stream
Author: login: taniumalloy
Labels: name: Integration:cloudflare
description: Cloudflare (Community supported)
name: Team:Security-Service Integrations
description: Security Service Integrations team [elastic/security-service-integrations]
Created: 2025-04-28T00:10:12Z | Merged: 2025-05-08T02:39:53Z

… CEL and support auth tokens.

- Enhancement

## Proposed commit message

The previous version did not support token authorization and restricted users to key authorization only, which is a [legacy method](https://developers.cloudflare.com/fundamentals/api/get-started/keys/). Introducing support for token authorization enables admins to implement least privilege and better restrict the scope of the keys created. The legacy key authorization method issues global keys provisioned with the same access as the user/admin who creates them. 

To introduce support for tokens, the input type for the Audit data stream was converted from httpjson to CEL. I chose CEL due to its robustness and widespread use among many integrations, enabling me to implement best practices, including state tracking effectively. The Audit data stream manifest was updated to include a field for tokens, modeling the existing manifest for the logpull manifest. The CEL program tracks state, cursor, and redacts sensitive fields appropriately. Many options were included to enable administrator control over page size, polling intervals, and lookback periods in addition to the standard support for SSL, proxy, etc. 

The README.md file was updated to reflect the new capabilities in the Audit input.


<!-- Mandatory
Explain here the changes you made on the PR.

Please explain:

- WHAT: patterns used, algorithms implemented, design architecture, message processing, etc.
- WHY:  the rationale/motivation for the changes

This text will be pasted into the squash dialog when the change is committed and will be
a long term historical record of the change to help future contributors understand the
change, please help them by making it clear and comprehensive, they may be you.

If the commit title is adequate to describe both of these things, The text here may be omitted
or replaced with "See title". The title of the PR will be used as the commit message title when
the merge is made and the "See title" marker will be removed if present.

The text here and the PR title will be subject to the PR review process.
-->

## Checklist

- [x] I have reviewed [tips for building integrations](https://github.com/elastic/integrations/blob/main/docs/tips_for_building_integrations.md) and this pull request is aligned with them.
- [x] I have verified that all data streams collect metrics or logs.
- [x] I have added an entry to my package's `changelog.yml` file.
- [x] I have verified that Kibana version constraints are current according to [guidelines](https://github.com/elastic/elastic-package/blob/master/docs/howto/stack_version_support.md#when-to-update-the-condition).
- [x] I have verified that any added dashboard complies with Kibana's [Dashboard good practices](https://docs.elastic.dev/ux-guidelines/data-viz/dashboard-good-practices) 

## Author's Checklist

<!-- Recommended
Add a checklist of things that are required to be reviewed in order to have the PR approved
-->
- [x] Validate the end-user can switch between key and the token authorization 
- [x] Validate the program handles errors effectively and displays informative error messages to the user
- [x] Validate the integration redacts sensitive auth fields
- [x] Validate the paging, interval, and lookback functionality works
- [x] Verify spelling and grammar

## How to test this PR locally

Use elastic-package to construct a local Dockerized cluster and test with Cloudflare API. Test with API key and token if possible, use request tracing to view changes to authorization headers, test edge boundaries, and test integration with built-in elastic-package testing commands. 

<!-- Recommended
Explain here how this PR will be tested by the reviewer: commands, dependencies, steps, etc.
-->

## Related issues

<!-- Recommended
Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it.

- Closes #123
- Relates #123
- Requires #123
- Supersedes #123
-->
- N/a

## Screenshots

<!-- Optional
Add here screenshots presenting:
- Kibana UI forms presenting configuration options exposed by the integration
- dashboards with collected metrics or logs
-->



Diff:
diff --git a/packages/cloudflare/_dev/build/docs/README.md b/packages/cloudflare/_dev/build/docs/README.md
index 6587258ca48..af9954c19fd 100644
--- a/packages/cloudflare/_dev/build/docs/README.md
+++ b/packages/cloudflare/_dev/build/docs/README.md
@@ -1,9 +1,13 @@
 # Cloudflare Integration
 
-Cloudflare integration uses [Cloudflare's API](https://api.cloudflare.com/) to retrieve [audit logs](https://support.cloudflare.com/hc/en-us/articles/115002833612-Understanding-Cloudflare-Audit-Logs) and [traffic logs](https://developers.cloudflare.com/logs/logpull/understanding-the-basics/) from Cloudflare, for a particular zone, and ingest them into Elasticsearch. This allows you to search, observe and visualize the Cloudflare log events through Elasticsearch.
+Cloudflare integration uses [Cloudflare's API](https://api.cloudflare.com/) to retrieve [audit logs](https://support.cloudflare.com/hc/en-us/articles/115002833612-Understanding-Cloudflare-Audit-Logs) from the Cloudflare account and [Cloudflare's Logpull API](https://developers.cloudflare.com/logs/logpull/) to retrieve [traffic logs](https://developers.cloudflare.com/logs/logpull/understanding-the-basics/) from Cloudflare, for a particular zone, and ingest them into Elasticsearch. This allows you to search, observe and visualize the Cloudflare log events through Elasticsearch.
 
 Users of [Cloudflare](https://www.cloudflare.com/en-au/learning/what-is-cloudflare/) use Cloudflare services to increase the security and performance of their web sites and services. 
 
+> Note: Logpull is considered a legacy feature and it is recommended to use the [Cloudflare Logpush](https://www.elastic.co/docs/reference/integrations/cloudflare_logpush) integration for Cloudflare traffic logs for better performance and functionality. See [here](https://developers.cloudflare.com/logs/logpull/) for more details.
+
+>  Note: Authenticating with API Key (Auth Key) using `X-AUTH-EMAIL` and `X-AUTH-KEY` is considered to be a legacy feature with several limitations that makes it less secure than API token (Auth Token). See [here](https://developers.cloudflare.com/fundamentals/api/get-started/keys/#limitations) for more information on API Key limitations.
+
 ## Configuration
 
 ### Enabling the integration in Elastic
@@ -13,48 +17,72 @@ Users of [Cloudflare](https://www.cloudflare.com/en-au/learning/what-is-cloudfla
 3. Click on "Cloudflare" integration from the search results.
 4. Click on **Add Cloudflare** button to add Cloudflare integration.
 
-### Configure Cloudflare audit logs data stream
+### Configure Cloudflare Audit logs
+
+The integration can retrieve Cloudflare audit logs using -
+
+1. Auth Email and Auth Key
+2. Auth Token
+
+More information is available [here](https://developers.cloudflare.com/logs/logpull/requesting-logs/#required-authentication-headers).
+
+#### Configure using Auth Email and Auth Key
 
 Enter values "Auth Email", "Auth Key" and "Account ID".
 
 1. **Auth Email** is the email address associated with your account. 
-2. [**Auth Key**](https://developers.cloudflare.com/api/keys/) is the API key generated on the "My Account" page.
-3. **Account ID** can be found on the Cloudflare dashboard. Follow the navigation documentation from [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
+2. **Auth Key** is the Global API key generated on the "My Profile" > "API Tokens" page.
+3. **Account ID** can be found [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
 
-NOTE: See for `X-AUTH-EMAIL` and `X-AUTH-KEY` [here](https://api.cloudflare.com/#getting-started-requests) for more information on Auth Email and Auth Key.
+>  Note: See [here](https://developers.cloudflare.com/fundamentals/api/get-started/keys/) for more information on `X-AUTH-EMAIL` and `X-AUTH-KEY`.
 
-### Configure Cloudflare logs
+#### Configure using Auth Token
+
+Enter values "Auth Token" and "Account ID".
+
+For the Cloudflare integration to be able to successfully get logs, one of the following permissions must be granted to the API Token -
+
+- Account Settings Write, Account Settings Read
+
+1. **Auth Token** is the API Token generated on the "My Profile" > "API Tokens" or "Manage Account" > "Account API Tokens" page.
+2. **Account ID** can be found [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
+
+>  Note: See [here](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/) for more information on `API Token`.
+
+### Configure Cloudflare Logpull logs
 
 These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server. For more information see [here](https://developers.cloudflare.com/logs/logpull/).
 
 The integration can retrieve Cloudflare logs using -
 
 1. Auth Email and Auth Key
-2. API Token
+2. Auth Token
 
-More information is available [here](https://developers.cloudflare.com/logs/logpull/requesting-logs/#required-authentication-headers)
+More information is available [here](https://developers.cloudflare.com/logs/logpull/requesting-logs/#required-authentication-headers).
 
 #### Configure using Auth Email and Auth Key
 
 Enter values "Auth Email", "Auth Key" and "Zone ID".
 
 1. **Auth Email** is the email address associated with your account. 
-2. [**Auth Key**](https://developers.cloudflare.com/api/keys/) is the API key generated on the "My Account" page.
+2. **Auth Key** is the Global API key generated on the "My Profile" > "API Tokens" page.
 3. **Zone ID** can be found [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
 
->  Note: See for `X-AUTH-EMAIL` and `X-AUTH-KEY` [here](https://api.cloudflare.com/#getting-started-requests) for more information on Auth Email and Auth Key.
+>  Note: See [here](https://developers.cloudflare.com/fundamentals/api/get-started/keys/) for more information on `X-AUTH-EMAIL` and `X-AUTH-KEY`.
 
-#### Configure using API Token
+#### Configure using Auth Token
 
-Enter values "API Token" and "Zone ID".
+Enter values "Auth Token" and "Zone ID".
 
-For the Cloudflare integration to be able to successfully get logs the following permissions must be granted to the API token -
+For the Cloudflare integration to be able to successfully get logs the following permissions must be granted to the API Token -
 
 - Account.Access: Audit Logs: Read
 
-1. [**API Tokens**](https://developers.cloudflare.com/api/tokens/) allow for more granular permission settings. 
+1. **Auth Token** is the API Token generated on the "My Profile" > "API Tokens" or "Manage Account" > "Account API Tokens" page.
 2. **Zone ID** can be found [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
 
+>  Note: See [here](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/) for more information on `API Token`.
+
 ## Logs
 
 ### Audit
@@ -71,4 +99,4 @@ These logs contain data related to the connecting client, the request path throu
 
 {{fields "logpull"}}
 
-{{event "logpull"}}
+{{event "logpull"}}
\ No newline at end of file
diff --git a/packages/cloudflare/changelog.yml b/packages/cloudflare/changelog.yml
index 7e5e76dafb9..77f96394293 100644
--- a/packages/cloudflare/changelog.yml
+++ b/packages/cloudflare/changelog.yml
@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.31.0"
+  changes:
+    - description: Added CEL input to support API Tokens in Cloudflare audit data stream input.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/13698
 - version: "2.30.0"
   changes:
     - description: Update Kibana constraint to support 9.0.0.
diff --git a/packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs b/packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
new file mode 100755
index 00000000000..2a56a3fd585
--- /dev/null
+++ b/packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
@@ -0,0 +1,159 @@
+config_version: "2"
+interval: {{interval}}
+resource.tracer:
+  enabled: {{enable_request_tracer}}
+  filename: "../../logs/cel/http-request-trace-*.ndjson"
+  maxbackups: 5
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            resp.Body.decode_json().as(body, {
+                "events": body.result.map(e, {
+                    "message": e.encode_json(),
+                }),
+                "want_more": has(body.result) && size(body.result) == state.per_page,
+                "cursor": (
+                    (has(body.result) && size(body.result) == state.per_page ?
+                        {
+                            "page": int(state.cursor.page) + 1,
+                            "last_timestamp": state.cursor.last_timestamp,                           
+                        }
+                    :
+                        {
+                            "page": 1,
+                            "last_timestamp": (
+                                has(body.result) && size(body.result) > 0 ? 
+                                    (
+                                        (timestamp(body.result.map(e, e.when).map(t, timestamp(t)).max().as(batch_last_timestamp, 
+                                            (batch_last_timestamp > timestamp(state.cursor.last_timestamp) ?
+                                                batch_last_timestamp
+                                            :
+                                                timestamp(state.cursor.last_timestamp))
+                                        ))).format(time_layout.RFC3339Nano)
+                                    )
+                                :
+                                    state.cursor.last_timestamp
+                            )
+                        }
+                    )
+                ),
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": state.?auth_token,
+                ?"auth_email": state.?auth_email,
+                ?"auth_key": state.?auth_key,
+            })
+        :
+            {
+                "events": {
+                    "error": {
+                        "code": string(resp.StatusCode),
+                        "id": string(resp.Status),
+                        "message": "GET " + state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs: "+(
+                            size(resp.Body) != 0 ?
+                                string(resp.Body)
+                            :
+                                string(resp.Status) + ' (' + string(resp.StatusCode) + ')'
+                        ),
+                    },
+                },
+                "want_more": false,
+                "cursor": {
+                    "last_timestamp": state.cursor.last_timestamp,
+                    "page": state.cursor.page,
+                },
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": state.?auth_token,
+                ?"auth_email": state.?auth_email,
+                ?"auth_key": state.?auth_key,
+            }
+        )
+    )
+{{#if tags.length}}
+tags:
+{{else if preserve_original_event}}
+tags:
+{{/if}}
+{{#each tags as |tag i|}}
+  - {{tag}}
+{{/each}}
+{{#if preserve_original_event}}
+  - preserve_original_event
+{{/if}}
+{{#contains "forwarded" tags}}
+publisher_pipeline.disable_host: true
+{{/contains}}
+
+processors:
+- add_fields:
+    target: _config
+    fields:
+      account_id: {{account}}
+{{#if processors}}
+{{processors}}
+{{/if}}
diff --git a/packages/cloudflare/data_stream/audit/agent/stream/httpjson.yml.hbs b/packages/cloudflare/data_stream/audit/agent/stream/httpjson.yml.hbs
index 35a18f592eb..e7799b0232a 100644
--- a/packages/cloudflare/data_stream/audit/agent/stream/httpjson.yml.hbs
+++ b/packages/cloudflare/data_stream/audit/agent/stream/httpjson.yml.hbs
@@ -1,9 +1,9 @@
 config_version: "2"
 interval: {{interval}}
-{{#if enable_request_tracer}}
-request.tracer.filename: "../../logs/httpjson/http-request-trace-*.ndjson"
-request.tracer.maxbackups: 5
-{{/if}}
+request.tracer:
+  enabled: {{enable_request_tracer}}
+  filename: "../../logs/httpjson/http-request-trace-*.ndjson"
+  maxbackups: 5
 request.method: "GET"
 {{#if per_page}}
 request.url: {{api_url}}/client/v4/accounts/{{account}}/audit_logs?page=1&per_page={{per_page}}&direction=desc
diff --git a/packages/cloudflare/data_stream/audit/manifest.yml b/packages/cloudflare/data_stream/audit/manifest.yml
index 59a02983578..edadfb33275 100644
--- a/packages/cloudflare/data_stream/audit/manifest.yml
+++ b/packages/cloudflare/data_stream/audit/manifest.yml
@@ -73,4 +73,86 @@ streams:
         description: "Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. \nThis executes in the agent before the logs are parsed. \nSee [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.\n"
     template_path: httpjson.yml.hbs
     title: Cloudflare Audit logs
+    description: Deprecated. Please disable this and use the v2 Cloudflare Audit logs input instead.
+  - input: cel
+    enabled: false
+    vars:
+      - name: auth_email
+        type: text
+        title: Auth Email
+        description: The Auth Email. Needs to be used with an Auth Key. Do not fill if you are using an Auth Token.
+        multi: false
+        required: false
+        show_user: true
+      - name: auth_key
+        type: password
+        title: Auth Key
+        description: The Auth Key. Needs to be used with an Auth Email. Do not fill if you are using an Auth Token.
+        multi: false
+        required: false
+        show_user: true
+        secret: true
+      - name: auth_token
+        type: password
+        title: Auth Token
+        description: The Auth Token. If set, Auth Email and Auth Key will be ignored.
+        required: false
+        multi: false
+        show_user: true
+        secret: true
+      - name: account
+        type: text
+        title: Account ID
+        multi: false
+        required: true
+        show_user: true
+      - name: interval
+        type: text
+        title: Interval
+        multi: false
+        required: true
+        show_user: true
+        description: Interval at which the logs will be pulled. The value must be between 1s and 1h. Supported units for this parameter are h/m/s.
+        default: 5m
+      - name: initial_interval
+        type: text
+        title: Initial Interval
+        multi: false
+        required: true
+        show_user: false
+        description: Initial interval at which the logs will be pulled. Defaults to 30 days (720 hours). Max is 12960 hours (18 months). Supported units for this parameter are h/m/s.
+        default: 720h
+      - name: per_page
+        type: integer
+        title: Results per page
+        description: Sets the number of results to return per page. If set this must be greater or equal to 1 and less than or equal to 1000. If not set, the API default is used.
+        multi: false
+        required: false
+        show_user: false
+      - name: tags
+        type: text
+        title: Tags
+        multi: true
+        required: true
+        show_user: true
+        default:
+          - forwarded
+          - cloudflare-audit
+      - name: preserve_original_event
+        required: true
+        show_user: true
+        title: Preserve original event
+        description: Preserves a raw copy of the original event, added to the field `event.original`
+        type: bool
+        multi: false
+        default: false
+      - name: processors
+        type: yaml
+        title: Processors
+        multi: false
+        required: false
+        show_user: false
+        description: "Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. \nThis executes in the agent before the logs are parsed. \nSee [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.\n"
+    template_path: cel.yml.hbs
+    title: Cloudflare Audit logs
     description: Collect Cloudflare Audit logs via the API
diff --git a/packages/cloudflare/data_stream/logpull/agent/stream/httpjson.yml.hbs b/packages/cloudflare/data_stream/logpull/agent/stream/httpjson.yml.hbs
index f818574ff14..f900b56a1f9 100644
--- a/packages/cloudflare/data_stream/logpull/agent/stream/httpjson.yml.hbs
+++ b/packages/cloudflare/data_stream/logpull/agent/stream/httpjson.yml.hbs
@@ -1,9 +1,9 @@
 config_version: "2"
 interval: {{interval}}
-{{#if enable_request_tracer}}
-request.tracer.filename: "../../logs/httpjson/http-request-trace-*.ndjson"
-request.tracer.maxbackups: 5
-{{/if}}
+request.tracer:
+  enabled: {{enable_request_tracer}}
+  filename: "../../logs/httpjson/http-request-trace-*.ndjson"
+  maxbackups: 5
 request.method: "GET"
 request.url: {{api_url}}/client/v4/zones/{{zone_id}}/logs/received
 {{#if ssl}}
diff --git a/packages/cloudflare/docs/README.md b/packages/cloudflare/docs/README.md
index 5488afdd306..1a8a5267cd8 100644
--- a/packages/cloudflare/docs/README.md
+++ b/packages/cloudflare/docs/README.md
@@ -1,9 +1,13 @@
 # Cloudflare Integration
 
-Cloudflare integration uses [Cloudflare's API](https://api.cloudflare.com/) to retrieve [audit logs](https://support.cloudflare.com/hc/en-us/articles/115002833612-Understanding-Cloudflare-Audit-Logs) and [traffic logs](https://developers.cloudflare.com/logs/logpull/understanding-the-basics/) from Cloudflare, for a particular zone, and ingest them into Elasticsearch. This allows you to search, observe and visualize the Cloudflare log events through Elasticsearch.
+Cloudflare integration uses [Cloudflare's API](https://api.cloudflare.com/) to retrieve [audit logs](https://support.cloudflare.com/hc/en-us/articles/115002833612-Understanding-Cloudflare-Audit-Logs) from the Cloudflare account and [Cloudflare's Logpull API](https://developers.cloudflare.com/logs/logpull/) to retrieve [traffic logs](https://developers.cloudflare.com/logs/logpull/understanding-the-basics/) from Cloudflare, for a particular zone, and ingest them into Elasticsearch. This allows you to search, observe and visualize the Cloudflare log events through Elasticsearch.
 
 Users of [Cloudflare](https://www.cloudflare.com/en-au/learning/what-is-cloudflare/) use Cloudflare services to increase the security and performance of their web sites and services. 
 
+> Note: Logpull is considered a legacy feature and it is recommended to use the [Cloudflare Logpush](https://www.elastic.co/docs/reference/integrations/cloudflare_logpush) integration for Cloudflare traffic logs for better performance and functionality. See [here](https://developers.cloudflare.com/logs/logpull/) for more details.
+
+>  Note: Authenticating with API Key (Auth Key) using `X-AUTH-EMAIL` and `X-AUTH-KEY` is considered to be a legacy feature with several limitations that makes it less secure than API token (Auth Token). See [here](https://developers.cloudflare.com/fundamentals/api/get-started/keys/#limitations) for more information on API Key limitations.
+
 ## Configuration
 
 ### Enabling the integration in Elastic
@@ -13,48 +17,72 @@ Users of [Cloudflare](https://www.cloudflare.com/en-au/learning/what-is-cloudfla
 3. Click on "Cloudflare" integration from the search results.
 4. Click on **Add Cloudflare** button to add Cloudflare integration.
 
-### Configure Cloudflare audit logs data stream
+### Configure Cloudflare Audit logs
+
+The integration can retrieve Cloudflare audit logs using -
+
+1. Auth Email and Auth Key
+2. Auth Token
+
+More information is available [here](https://developers.cloudflare.com/logs/logpull/requesting-logs/#required-authentication-headers).
+
+#### Configure using Auth Email and Auth Key
 
 Enter values "Auth Email", "Auth Key" and "Account ID".
 
 1. **Auth Email** is the email address associated with your account. 
-2. [**Auth Key**](https://developers.cloudflare.com/api/keys/) is the API key generated on the "My Account" page.
-3. **Account ID** can be found on the Cloudflare dashboard. Follow the navigation documentation from [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
+2. **Auth Key** is the Global API key generated on the "My Profile" > "API Tokens" page.
+3. **Account ID** can be found [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
 
-NOTE: See for `X-AUTH-EMAIL` and `X-AUTH-KEY` [here](https://api.cloudflare.com/#getting-started-requests) for more information on Auth Email and Auth Key.
+>  Note: See [here](https://developers.cloudflare.com/fundamentals/api/get-started/keys/) for more information on `X-AUTH-EMAIL` and `X-AUTH-KEY`.
 
-### Configure Cloudflare logs
+#### Configure using Auth Token
+
+Enter values "Auth Token" and "Account ID".
+
+For the Cloudflare integration to be able to successfully get logs, one of the following permissions must be granted to the API Token -
+
+- Account Settings Write, Account Settings Read
+
+1. **Auth Token** is the API Token generated on the "My Profile" > "API Tokens" or "Manage Account" > "Account API Tokens" page.
+2. **Account ID** can be found [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
+
+>  Note: See [here](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/) for more information on `API Token`.
+
+### Configure Cloudflare Logpull logs
 
 These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server. For more information see [here](https://developers.cloudflare.com/logs/logpull/).
 
 The integration can retrieve Cloudflare logs using -
 
 1. Auth Email and Auth Key
-2. API Token
+2. Auth Token
 
-More information is available [here](https://developers.cloudflare.com/logs/logpull/requesting-logs/#required-authentication-headers)
+More information is available [here](https://developers.cloudflare.com/logs/logpull/requesting-logs/#required-authentication-headers).
 
 #### Configure using Auth Email and Auth Key
 
 Enter values "Auth Email", "Auth Key" and "Zone ID".
 
 1. **Auth Email** is the email address associated with your account. 
-2. [**Auth Key**](https://developers.cloudflare.com/api/keys/) is the API key generated on the "My Account" page.
+2. **Auth Key** is the Global API key generated on the "My Profile" > "API Tokens" page.
 3. **Zone ID** can be found [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
 
->  Note: See for `X-AUTH-EMAIL` and `X-AUTH-KEY` [here](https://api.cloudflare.com/#getting-started-requests) for more information on Auth Email and Auth Key.
+>  Note: See [here](https://developers.cloudflare.com/fundamentals/api/get-started/keys/) for more information on `X-AUTH-EMAIL` and `X-AUTH-KEY`.
 
-#### Configure using API Token
+#### Configure using Auth Token
 
-Enter values "API Token" and "Zone ID".
+Enter values "Auth Token" and "Zone ID".
 
-For the Cloudflare integration to be able to successfully get logs the following permissions must be granted to the API token -
+For the Cloudflare integration to be able to successfully get logs the following permissions must be granted to the API Token -
 
 - Account.Access: Audit Logs: Read
 
-1. [**API Tokens**](https://developers.cloudflare.com/api/tokens/) allow for more granular permission settings. 
+1. **Auth Token** is the API Token generated on the "My Profile" > "API Tokens" or "Manage Account" > "Account API Tokens" page.
 2. **Zone ID** can be found [here](https://developers.cloudflare.com/fundamentals/get-started/basic-tasks/find-account-and-zone-ids/).
 
+>  Note: See [here](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/) for more information on `API Token`.
+
 ## Logs
 
 ### Audit
@@ -443,4 +471,4 @@ An example event for `logpull` looks as following:
         "version": "5.2.2"
     }
 }
-```
+```
\ No newline at end of file
diff --git a/packages/cloudflare/manifest.yml b/packages/cloudflare/manifest.yml
index db119c07966..c5314fac813 100644
--- a/packages/cloudflare/manifest.yml
+++ b/packages/cloudflare/manifest.yml
@@ -1,13 +1,13 @@
 name: cloudflare
 title: Cloudflare
-version: "2.30.0"
+version: "2.31.0"
 description: Collect logs from Cloudflare with Elastic Agent.
 type: integration
 format_version: "3.0.2"
 categories: [security, network, cdn_security]
 conditions:
   kibana:
-    version: "^8.13.0 || ^9.0.0"
+    version: "^8.16.0 || ^9.0.0"
 icons:
   - src: /img/cf-logo-v.svg
     title: Cloudflare
@@ -31,8 +31,55 @@ policy_templates:
     title: Cloudflare logs
     description: Collect logs from Cloudflare
     inputs:
+      - type: cel
+        title: "Collect Cloudflare logs via API (v2)"
+        description: "Collecting logs from Cloudflare via API"
+        vars:
+          - name: api_url
+            type: text
+            title: API URL.
+            description: The API URL without the path.
+            multi: false
+            required: true
+            show_user: false
+            default: https://api.cloudflare.com
+          - name: enable_request_tracer
+            type: bool
+            title: Enable request tracing
+            multi: false
+            required: false
+            show_user: false
+            default: false
+            description: >-
+              The request tracer logs requests and responses to the agent's local file-system for debugging configurations.
+              Enabling this request tracing compromises security and should only be used for debugging. Disabling the request
+              tracer will delete any stored traces.
+              See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_enable)
+              for details.
+          - name: ssl
+            type: yaml
+            title: SSL
+            multi: false
+            required: false
+            show_user: false
+            description: SSL configuration options. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html#ssl-common-config) for details.
+          - name: proxy_url
+            type: text
+            title: Proxy URL
+            multi: false
+            required: false
+            show_user: false
+            description: URL to proxy connections in the form of http\[s\]://<user>:<password>@<server name/ip>:<port>
+          - name: http_client_timeout
+            type: text
+            title: HTTP Client Timeout
+            description: Duration before declaring that the HTTP client connection has timed out. Valid time units are ns, us, ms, s, m, h.
+            multi: false
+            required: false
+            show_user: true
+            default: 60s
       - type: httpjson
-        title: "Collect Cloudflare logs via API"
+        title: "Collect Cloudflare logs via API (Legacy Input)"
         description: "Collecting logs from Cloudflare via API"
         vars:
           - name: api_url
@@ -49,7 +96,13 @@ policy_templates:
             multi: false
             required: false
             show_user: false
-            description: The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#_request_tracer_filename) for details.
+            default: false
+            description: >-
+              The request tracer logs requests and responses to the agent's local file-system for debugging configurations.
+              Enabling this request tracing compromises security and should only be used for debugging. Disabling the request
+              tracer will delete any stored traces.
+              See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#_request_tracer_filename)
+              for details.
           - name: ssl
             type: yaml
             title: SSL


Reviews:
user: login: efd6
body: Has this been discussed? Is there an issue to track it?

This is a significant breaking change and I think we need to ensure that it is aligned before we go ahead with it.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T01:24:40Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T03:05:32Z
user: login: efd6
body: Nits only. Looking good. Thanks
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T00:06:57Z
user: login: efd6
body: Thanks
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T02:39:32Z

Review comments:
diff_hunk: @@ -0,0 +1,181 @@
+config_version: "2"
+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5
+{{/if}}
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: ```suggestion
```
created_at: 2025-05-01T02:39:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "3.0.0"
path: packages/cloudflare/changelog.yml
user: login: efd6
body: ```suggestion
- version: "2.31.0"
```
(this is no longer breaking)
created_at: 2025-05-01T02:41:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,181 @@
+config_version: "2"
+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5
+{{/if}}
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: If we bump the kibana.version to 8.16 or better, we can improve this behaviour by changing this to
```
resource.tracer:
  enabled: {{enable_request_tracer}}
  filename: "../../logs/cel/http-request-trace-*.ndjson"
  maxbackups: 5
```
This requires that we also set a false default for `enable_request_tracer` in the manifest.

A similar change can be made for the httpjson input.
created_at: 2025-05-01T02:44:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,181 @@
+config_version: "2"
+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5
+{{/if}}
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            bytes(resp.Body).decode_json().as(body, {
+                "events": body.result.map(e, {
+                    "message": e.encode_json(),
+                }),
+                "want_more": (has(body.result) && size(body.result) == state.per_page ? true : false),
+                "cursor": (
+                    (has(body.result) && size(body.result) == state.per_page ?
+                        {
+                            "page": int(state.cursor.page) + 1,
+                            "last_timestamp": state.cursor.last_timestamp,                           
+                        }
+                    :
+                    
+                        {
+                            "page": 1,
+                            "last_timestamp": (
+                                has(body.result) && size(body.result) > 0 ? 
+                                    (
+                                        (timestamp(body.result.map(e, e.when).map(t, timestamp(t)).max().as(batch_last_timestamp, 
+                                            (batch_last_timestamp > timestamp(state.cursor.last_timestamp) ?
+                                                batch_last_timestamp
+                                            :
+                                                timestamp(state.cursor.last_timestamp))
+                                        ))).format(time_layout.RFC3339Nano)
+                                    )
+                                :
+                                    state.cursor.last_timestamp
+                            )
+                        }
+                    )
+                ),
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": has(state.auth_token) ?
+                    optional.of(state.auth_token)
+                :
+                    optional.none(),
+                ?"auth_email": has(state.auth_email) ? 
+                    optional.of(state.auth_email)
+                :
+                    optional.none(),
+                ?"auth_key": has(state.auth_key) ? 
+                    optional.of(state.auth_key)
+                :
+                    optional.none(),
+            })
+
+        :
+            {
+                "events": {
+                    "error": {
+                        "code": string(resp.StatusCode),
+                        "id": string(resp.Status),
+                        "message": "GET:"+(
+                            size(resp.Body) != 0 ?
+                            string(resp.Body)
+                            :
+                            string(resp.Status) + ' (' + string(resp.StatusCode) + ')'
+                        ),
+                    },
+                },
+                "want_more": false,
+                "cursor": {
+                    "last_timestamp": state.cursor.last_timestamp,
+                    "page": state.cursor.page,
+                },
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": has(state.auth_token) ?
+                    optional.of(state.auth_token)
+                :
+                    optional.none(),
+                ?"auth_email": has(state.auth_email) ? 
+                    optional.of(state.auth_email)
+                :
+                    optional.none(),
+                ?"auth_key": has(state.auth_key) ? 
+                    optional.of(state.auth_key)
+                :
+                    optional.none(),
+            }
+        )
+    )
+
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: ```suggestion
```
created_at: 2025-05-01T02:44:42Z
author_association: CONTRIBUTOR
path: packages/cloudflare/data_stream/logpull/manifest.yml
user: login: efd6
body: Let's leave this alone and just have the PR focus on the audit data stream.
created_at: 2025-05-01T02:47:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -31,8 +31,49 @@ policy_templates:
     title: Cloudflare logs
     description: Collect logs from Cloudflare
     inputs:
+      - type: cel
+        title: "Collect Cloudflare logs via API (v2)"
+        description: "Collecting logs from Cloudflare via API"
+        vars:
+          - name: api_url
+            type: text
+            title: API URL.
+            description: The API URL without the path.
+            multi: false
+            required: true
+            show_user: false
+            default: https://api.cloudflare.com
+          - name: enable_request_tracer
+            type: bool
+            title: Enable request tracing
+            multi: false
+            required: false
+            show_user: false
+            description: The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#_request_tracer_filename) for details.
path: packages/cloudflare/manifest.yml
user: login: efd6
body: ```suggestion
            description: >-
              The request tracer logs requests and responses to the agent's local file-system for debugging configurations.
              Enabling this request tracing compromises security and should only be used for debugging. Disabling the request
              tracer will delete any stored traces.
              See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_enable)
              for details.
```
(partly for line length and partly for the correct link)
created_at: 2025-05-01T02:48:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -31,8 +31,49 @@ policy_templates:
     title: Cloudflare logs
     description: Collect logs from Cloudflare
     inputs:
+      - type: cel
+        title: "Collect Cloudflare logs via API (v2)"
+        description: "Collecting logs from Cloudflare via API"
+        vars:
+          - name: api_url
+            type: text
+            title: API URL.
+            description: The API URL without the path.
+            multi: false
+            required: true
+            show_user: false
+            default: https://api.cloudflare.com
+          - name: enable_request_tracer
path: packages/cloudflare/manifest.yml
user: login: efd6
body: Add a `default: false` if the comment above to change the behaviour of this feature is adopted.
created_at: 2025-05-01T02:50:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -31,8 +31,49 @@ policy_templates:
     title: Cloudflare logs
     description: Collect logs from Cloudflare
     inputs:
+      - type: cel
+        title: "Collect Cloudflare logs via API (v2)"
+        description: "Collecting logs from Cloudflare via API"
+        vars:
+          - name: api_url
+            type: text
+            title: API URL.
+            description: The API URL without the path.
+            multi: false
+            required: true
+            show_user: false
+            default: https://api.cloudflare.com
+          - name: enable_request_tracer
+            type: bool
+            title: Enable request tracing
+            multi: false
+            required: false
+            show_user: false
+            description: The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#_request_tracer_filename) for details.
+          - name: ssl
+            type: yaml
+            title: SSL
+            multi: false
+            required: false
+            show_user: false
+            description: SSL configuration options. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html#ssl-common-config) for details.
+          - name: proxy_url
+            type: text
+            title: Proxy URL
+            multi: false
+            required: false
+            show_user: false
+            description: URL to proxy connections in the form of http\[s\]://<user>:<password>@<server name/ip>:<port>
+          - name: http_client_timeout
+            type: text
+            title: HTTP Client Timeout
+            description: Duration before declaring that the HTTP client connection has timed out. Valid time units are ns, us, ms, s, m, h.
+            multi: false
+            required: false
+            show_user: true
+            default: 60s
       - type: httpjson
-        title: "Collect Cloudflare logs via API"
+        title: "Collect Cloudflare logs via API (v1)"
path: packages/cloudflare/manifest.yml
user: login: efd6
body: ```suggestion
        title: "Collect Cloudflare logs via API (Legacy Input)"
```
created_at: 2025-05-01T02:50:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,6 +1,6 @@
 name: cloudflare
 title: Cloudflare
-version: "2.30.0"
+version: "3.0.0"
path: packages/cloudflare/manifest.yml
user: login: efd6
body: ```suggestion
version: "2.31.0"
```
created_at: 2025-05-01T02:50:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,181 @@
+config_version: "2"
+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5
+{{/if}}
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            bytes(resp.Body).decode_json().as(body, {
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: If we bump to 8.16 or better, this can be
```suggestion
            resp.Body.decode_json().as(body, {
```
created_at: 2025-05-01T02:53:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,181 @@
+config_version: "2"
+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5
+{{/if}}
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            bytes(resp.Body).decode_json().as(body, {
+                "events": body.result.map(e, {
+                    "message": e.encode_json(),
+                }),
+                "want_more": (has(body.result) && size(body.result) == state.per_page ? true : false),
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: This can be simplified.
```suggestion
                "want_more": has(body.result) && size(body.result) == state.per_page,
```
otherwise what is stopping us from saying `(((has(body.result) && size(body.result) == state.per_page ? true : false) ? true : false) ? true : false) ? … )`? That way madness lies.
created_at: 2025-05-01T02:55:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,181 @@
+config_version: "2"
+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5
+{{/if}}
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            bytes(resp.Body).decode_json().as(body, {
+                "events": body.result.map(e, {
+                    "message": e.encode_json(),
+                }),
+                "want_more": (has(body.result) && size(body.result) == state.per_page ? true : false),
+                "cursor": (
+                    (has(body.result) && size(body.result) == state.per_page ?
+                        {
+                            "page": int(state.cursor.page) + 1,
+                            "last_timestamp": state.cursor.last_timestamp,                           
+                        }
+                    :
+                    
+                        {
+                            "page": 1,
+                            "last_timestamp": (
+                                has(body.result) && size(body.result) > 0 ? 
+                                    (
+                                        (timestamp(body.result.map(e, e.when).map(t, timestamp(t)).max().as(batch_last_timestamp, 
+                                            (batch_last_timestamp > timestamp(state.cursor.last_timestamp) ?
+                                                batch_last_timestamp
+                                            :
+                                                timestamp(state.cursor.last_timestamp))
+                                        ))).format(time_layout.RFC3339Nano)
+                                    )
+                                :
+                                    state.cursor.last_timestamp
+                            )
+                        }
+                    )
+                ),
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": has(state.auth_token) ?
+                    optional.of(state.auth_token)
+                :
+                    optional.none(),
+                ?"auth_email": has(state.auth_email) ? 
+                    optional.of(state.auth_email)
+                :
+                    optional.none(),
+                ?"auth_key": has(state.auth_key) ? 
+                    optional.of(state.auth_key)
+                :
+                    optional.none(),
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: ```suggestion
                ?"auth_token": state.?auth_token,
                ?"auth_email": state.?auth_email,
                ?"auth_key": state.?auth_key,
```
created_at: 2025-05-01T02:57:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,181 @@
+config_version: "2"
+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5
+{{/if}}
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            bytes(resp.Body).decode_json().as(body, {
+                "events": body.result.map(e, {
+                    "message": e.encode_json(),
+                }),
+                "want_more": (has(body.result) && size(body.result) == state.per_page ? true : false),
+                "cursor": (
+                    (has(body.result) && size(body.result) == state.per_page ?
+                        {
+                            "page": int(state.cursor.page) + 1,
+                            "last_timestamp": state.cursor.last_timestamp,                           
+                        }
+                    :
+                    
+                        {
+                            "page": 1,
+                            "last_timestamp": (
+                                has(body.result) && size(body.result) > 0 ? 
+                                    (
+                                        (timestamp(body.result.map(e, e.when).map(t, timestamp(t)).max().as(batch_last_timestamp, 
+                                            (batch_last_timestamp > timestamp(state.cursor.last_timestamp) ?
+                                                batch_last_timestamp
+                                            :
+                                                timestamp(state.cursor.last_timestamp))
+                                        ))).format(time_layout.RFC3339Nano)
+                                    )
+                                :
+                                    state.cursor.last_timestamp
+                            )
+                        }
+                    )
+                ),
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": has(state.auth_token) ?
+                    optional.of(state.auth_token)
+                :
+                    optional.none(),
+                ?"auth_email": has(state.auth_email) ? 
+                    optional.of(state.auth_email)
+                :
+                    optional.none(),
+                ?"auth_key": has(state.auth_key) ? 
+                    optional.of(state.auth_key)
+                :
+                    optional.none(),
+            })
+
+        :
+            {
+                "events": {
+                    "error": {
+                        "code": string(resp.StatusCode),
+                        "id": string(resp.Status),
+                        "message": "GET:"+(
+                            size(resp.Body) != 0 ?
+                            string(resp.Body)
+                            :
+                            string(resp.Status) + ' (' + string(resp.StatusCode) + ')'
+                        ),
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: ```suggestion
                        "message": "GET " + state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs:"+(
                            size(resp.Body) != 0 ?
                                string(resp.Body)
                            :
                                string(resp.Status) + ' (' + string(resp.StatusCode) + ')'
                        ),
```
created_at: 2025-05-01T02:59:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,181 @@
+config_version: "2"
+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5
+{{/if}}
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            bytes(resp.Body).decode_json().as(body, {
+                "events": body.result.map(e, {
+                    "message": e.encode_json(),
+                }),
+                "want_more": (has(body.result) && size(body.result) == state.per_page ? true : false),
+                "cursor": (
+                    (has(body.result) && size(body.result) == state.per_page ?
+                        {
+                            "page": int(state.cursor.page) + 1,
+                            "last_timestamp": state.cursor.last_timestamp,                           
+                        }
+                    :
+                    
+                        {
+                            "page": 1,
+                            "last_timestamp": (
+                                has(body.result) && size(body.result) > 0 ? 
+                                    (
+                                        (timestamp(body.result.map(e, e.when).map(t, timestamp(t)).max().as(batch_last_timestamp, 
+                                            (batch_last_timestamp > timestamp(state.cursor.last_timestamp) ?
+                                                batch_last_timestamp
+                                            :
+                                                timestamp(state.cursor.last_timestamp))
+                                        ))).format(time_layout.RFC3339Nano)
+                                    )
+                                :
+                                    state.cursor.last_timestamp
+                            )
+                        }
+                    )
+                ),
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": has(state.auth_token) ?
+                    optional.of(state.auth_token)
+                :
+                    optional.none(),
+                ?"auth_email": has(state.auth_email) ? 
+                    optional.of(state.auth_email)
+                :
+                    optional.none(),
+                ?"auth_key": has(state.auth_key) ? 
+                    optional.of(state.auth_key)
+                :
+                    optional.none(),
+            })
+
+        :
+            {
+                "events": {
+                    "error": {
+                        "code": string(resp.StatusCode),
+                        "id": string(resp.Status),
+                        "message": "GET:"+(
+                            size(resp.Body) != 0 ?
+                            string(resp.Body)
+                            :
+                            string(resp.Status) + ' (' + string(resp.StatusCode) + ')'
+                        ),
+                    },
+                },
+                "want_more": false,
+                "cursor": {
+                    "last_timestamp": state.cursor.last_timestamp,
+                    "page": state.cursor.page,
+                },
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": has(state.auth_token) ?
+                    optional.of(state.auth_token)
+                :
+                    optional.none(),
+                ?"auth_email": has(state.auth_email) ? 
+                    optional.of(state.auth_email)
+                :
+                    optional.none(),
+                ?"auth_key": has(state.auth_key) ? 
+                    optional.of(state.auth_key)
+                :
+                    optional.none(),
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: ```suggestion
                ?"auth_token": state.?auth_token,
                ?"auth_email": state.?auth_email,
                ?"auth_key": state.?auth_key,
```
created_at: 2025-05-01T03:00:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,161 @@
+config_version: "2"
+interval: {{interval}}
+resource.tracer:
+  enabled: {{enable_request_tracer}}
+  filename: "../../logs/cel/http-request-trace-*.ndjson"
+  maxbackups: 5
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            resp.Body.decode_json().as(body, {
+                "events": body.result.map(e, {
+                    "message": e.encode_json(),
+                }),
+                "want_more": has(body.result) && size(body.result) == state.per_page,
+                "cursor": (
+                    (has(body.result) && size(body.result) == state.per_page ?
+                        {
+                            "page": int(state.cursor.page) + 1,
+                            "last_timestamp": state.cursor.last_timestamp,                           
+                        }
+                    :
+                    
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: ```suggestion
```
created_at: 2025-05-08T00:04:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,161 @@
+config_version: "2"
+interval: {{interval}}
+resource.tracer:
+  enabled: {{enable_request_tracer}}
+  filename: "../../logs/cel/http-request-trace-*.ndjson"
+  maxbackups: 5
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            resp.Body.decode_json().as(body, {
+                "events": body.result.map(e, {
+                    "message": e.encode_json(),
+                }),
+                "want_more": has(body.result) && size(body.result) == state.per_page,
+                "cursor": (
+                    (has(body.result) && size(body.result) == state.per_page ?
+                        {
+                            "page": int(state.cursor.page) + 1,
+                            "last_timestamp": state.cursor.last_timestamp,                           
+                        }
+                    :
+                    
+                        {
+                            "page": 1,
+                            "last_timestamp": (
+                                has(body.result) && size(body.result) > 0 ? 
+                                    (
+                                        (timestamp(body.result.map(e, e.when).map(t, timestamp(t)).max().as(batch_last_timestamp, 
+                                            (batch_last_timestamp > timestamp(state.cursor.last_timestamp) ?
+                                                batch_last_timestamp
+                                            :
+                                                timestamp(state.cursor.last_timestamp))
+                                        ))).format(time_layout.RFC3339Nano)
+                                    )
+                                :
+                                    state.cursor.last_timestamp
+                            )
+                        }
+                    )
+                ),
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": state.?auth_token,
+                ?"auth_email": state.?auth_email,
+                ?"auth_key": state.?auth_key,
+            })
+
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: ```suggestion
```
created_at: 2025-05-08T00:04:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,161 @@
+config_version: "2"
+interval: {{interval}}
+resource.tracer:
+  enabled: {{enable_request_tracer}}
+  filename: "../../logs/cel/http-request-trace-*.ndjson"
+  maxbackups: 5
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{api_url}}
+state:
+    account_id: {{account}}
+{{#if per_page}}
+    per_page: {{per_page}}
+{{/if}}
+{{#if initial_interval}}
+    initial_interval: {{initial_interval}}
+{{/if}}
+{{#if auth_token}}
+    auth_token: {{auth_token}}
+{{/if}}
+{{#if auth_key}}
+{{#if auth_email}}
+    auth_key: {{auth_key}}
+    auth_email: {{auth_email}}
+{{/if}}
+{{/if}}
+redact:
+    fields:
+        - auth_token
+        - auth_key
+program: |
+    (
+        has(state.want_more) && state.want_more ?
+            state
+        :
+            state.with({
+                "cursor" : {
+                    "page": 1,
+                    "last_timestamp": state.?cursor.last_timestamp.orValue((now - duration(state.?initial_interval.orValue("720h"))).format(time_layout.RFC3339Nano)),
+                },
+                "per_page" : state.?per_page.orValue(100),
+            })
+    ).as(state, state.with(get_request(
+            state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs?" + {
+                "since": [state.cursor.last_timestamp],
+                "page": [string(state.cursor.page)], 
+                "per_page": [string(state.per_page)],           
+                "direction": ["asc"],
+            }.format_query()
+        )).with({
+            "Header": {
+                "Content-Type": ["application/json"],
+                ?"Authorization": has(state.auth_token) ?
+                    optional.of(["Bearer " + state.auth_token])
+                :
+                    optional.none(),
+                ?"X-Auth-Email": has(state.auth_email) && !has(state.auth_token) ?
+                    optional.of([state.auth_email])
+                :
+                    optional.none(),
+                ?"X-Auth-Key": has(state.auth_key) && !has(state.auth_token) ?
+                    optional.of([state.auth_key])
+                :
+                    optional.none(),
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            resp.Body.decode_json().as(body, {
+                "events": body.result.map(e, {
+                    "message": e.encode_json(),
+                }),
+                "want_more": has(body.result) && size(body.result) == state.per_page,
+                "cursor": (
+                    (has(body.result) && size(body.result) == state.per_page ?
+                        {
+                            "page": int(state.cursor.page) + 1,
+                            "last_timestamp": state.cursor.last_timestamp,                           
+                        }
+                    :
+                    
+                        {
+                            "page": 1,
+                            "last_timestamp": (
+                                has(body.result) && size(body.result) > 0 ? 
+                                    (
+                                        (timestamp(body.result.map(e, e.when).map(t, timestamp(t)).max().as(batch_last_timestamp, 
+                                            (batch_last_timestamp > timestamp(state.cursor.last_timestamp) ?
+                                                batch_last_timestamp
+                                            :
+                                                timestamp(state.cursor.last_timestamp))
+                                        ))).format(time_layout.RFC3339Nano)
+                                    )
+                                :
+                                    state.cursor.last_timestamp
+                            )
+                        }
+                    )
+                ),
+                "account_id": state.account_id,
+                "per_page": state.per_page,
+                ?"auth_token": state.?auth_token,
+                ?"auth_email": state.?auth_email,
+                ?"auth_key": state.?auth_key,
+            })
+
+        :
+            {
+                "events": {
+                    "error": {
+                        "code": string(resp.StatusCode),
+                        "id": string(resp.Status),
+                        "message": "GET " + state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs:"+(
path: packages/cloudflare/data_stream/audit/agent/stream/cel.yml.hbs
user: login: efd6
body: ```suggestion
                        "message": "GET " + state.url.trim_right("/") + "/client/v4/accounts/" + state.account_id + "/audit_logs: "+(
```
created_at: 2025-05-08T00:05:02Z
author_association: CONTRIBUTOR

PR comments:
user: login: cla-checker-service[bot]
created_at: 2025-04-28T00:10:16Z
body: <!-- CLA-CHECK:13698 -->
&#x1F49A; CLA has been signed
author_association: NONE
user: login: efd6
created_at: 2025-04-28T00:26:38Z
body: @taniumalloy Please sign the CLA so that we can take a look at this.
author_association: CONTRIBUTOR
user: login: taniumalloy
created_at: 2025-04-28T01:18:45Z
body: I just signed the agreement with my GitHub username. Please let me know if there are any issues. 
author_association: CONTRIBUTOR
user: login: efd6
created_at: 2025-04-28T01:24:49Z
body: /test
author_association: CONTRIBUTOR
user: login: taniumalloy
created_at: 2025-04-28T01:30:07Z
body: It has not been discussed, and there are no issues that I am aware of. I have not submitted an issue. I know this integration is supported by the community, so I was not sure how to approach getting these changes implemented other than submitting a PR. 

If you would like, we can close this PR, and I can proceed with submitting an issue. Please advise me of any other steps you would like me to take. 

Thank you
author_association: CONTRIBUTOR
user: login: efd6
created_at: 2025-04-28T01:46:39Z
body: I'll raise this internally. Until then, we can pause this. Don't close the PR.
author_association: CONTRIBUTOR
user: login: elastic-vault-github-plugin-prod[bot]
created_at: 2025-04-28T01:50:33Z
body: ### :rocket: Benchmarks report




To see the full report comment with `/test benchmark fullreport`
<!--COMMENT_GENERATED_WITH_ID_benchmark-report-->
author_association: NONE
user: login: efd6
created_at: 2025-04-28T01:58:58Z
body: /test
author_association: CONTRIBUTOR
user: login: elasticmachine
created_at: 2025-04-28T12:06:28Z
body: Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
author_association: NONE
user: login: efd6
created_at: 2025-04-29T22:58:13Z
body: @taniumalloy, thanks for your patience. We can't break people here, but we can add the new functionality using the CEL input. So please revert the deletion of the HTTP JSON input while retaining the new CEL input and mark the HTTP JSON input as deprecated and set the CEL input as `enabled: false`. Once that's done ping me and I'll review.
author_association: CONTRIBUTOR
user: login: taniumalloy
created_at: 2025-04-29T23:14:19Z
body: @efd6 Thank you for your assistance. I understand and think that is a good solution. Would you like me to move the PR to draft mode while I work on this? 
author_association: CONTRIBUTOR
user: login: efd6
created_at: 2025-04-30T00:28:48Z
body: Sure, move it to draft, and when you are ready, let me know.
author_association: CONTRIBUTOR
user: login: taniumalloy
created_at: 2025-05-01T01:48:01Z
body: @efd6 I am ready for you to take a look. Here is a summary of the changes I made:

- Updated cursor time tracking to use RFC3339Nano as some audit events, such as DNS changes, are logged with ms accuracy.
- Removed the +1s increment from the cursor time tracking. While keeping the same cursor timestamp for back-to-back batches with no new events will keep fetching the most recent event due to Cloudflare's inclusive "since" boundary, the pipeline assigns an _id value to each log based on a field set by Cloudflare that is unique for each event, so no duplicates are indexed. Based on what I see in other integrations, this seems to be common and acceptable. This also ensures the entire time range is covered to guarantee audit log delivery. 
- Added v1 tag to the parent httpjson input.
- Added v2 tag to the parent cel input.
- Disabled the cel input by default, ensuring current users will not be affected by the upgrade.
- Included a description for the deprecated HTTP JSON Audit logs input, informing users to migrate workload to the v2 input for Audit logs.
- Included a description for the v1 Cloudflare Logpull input, informing users to consider using Logpush. While Logull is a [legacy](https://developers.cloudflare.com/logs/logpull/) collection method, it is not yet deprecated by Cloudflare and is still supported. However, I feel this note is aligned well with the inherited v1 tag. 
- Made many modifications to the README.md to update broken links, improve phrasing, inform users of legacy methods, and organize the file to increase clarity and accuracy. 

I have conducted tests and see no functional issues. Upgrading the integration did not affect my established HTTP JSON Audit log input. I was able to move to the new v2 CEL Audit log input and change between Key and Token auth mechanisms without issue.

Let me know what you think. If you have any concerns or see any issues, please let me know. Thank you
author_association: CONTRIBUTOR
user: login: efd6
created_at: 2025-05-01T03:05:43Z
body: /test
author_association: CONTRIBUTOR
user: login: taniumalloy
created_at: 2025-05-02T02:51:41Z
body: @efd6 Thanks for the detailed review and suggestions. I resolved all of your suggestions and fixes. 

Summary:
- Bumped up minimum Kibana version for v2.31.0 to 8.16.0 to get the newest features
- Updated the request and resource tracing module configuration in each stream config
- Implemented recommended CEL program changes
- Focused on Audit log input
- Changed parent HTTP JSON module naming to "Collect Cloudflare logs via API (Legacy Input)"

I kept the README.md changes. Let me know if you have issues with these changes.

The new parent CEL module is still named as "Collect Cloudflare logs via API (v2)", so let me know if you have any issues or suggestions. The (v2) made sense with the corresponding (v1) parent, but I am open to ideas now. I can leave it or remove the (v2) tag altogether. 
author_association: CONTRIBUTOR
user: login: efd6
created_at: 2025-05-06T21:46:26Z
body: /test
author_association: CONTRIBUTOR
user: login: efd6
created_at: 2025-05-08T01:49:52Z
body: /test
author_association: CONTRIBUTOR
user: login: elastic-sonarqube[bot]
created_at: 2025-05-08T02:14:51Z
body: ## [![Quality Gate passed](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonar.elastic.dev/dashboard?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698) **Quality Gate passed**  
Issues  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698&resolved=false) [0 New issues](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698&resolved=false)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/fixed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&fixedInPullRequest=13698) [0 Fixed issues](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&fixedInPullRequest=13698)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/accepted-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698&issueStatuses=ACCEPTED) [0 Accepted issues](https://sonar.elastic.dev/project/issues?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698&issueStatuses=ACCEPTED)

Measures  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/security_hotspots?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698) [0 Security Hotspots](https://sonar.elastic.dev/security_hotspots?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/component_measures?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698&metric=new_coverage&view=list) [100.0% Coverage on New Code](https://sonar.elastic.dev/component_measures?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698&metric=new_coverage&view=list)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/component_measures?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698&metric=new_duplicated_lines_density&view=list) [0.0% Duplication on New Code](https://sonar.elastic.dev/component_measures?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube](https://sonar.elastic.dev/dashboard?id=elastic_integrations_AYu5LCaKQZlFqhqWIrk_&pullRequest=13698)


author_association: NONE
user: login: elasticmachine
created_at: 2025-05-08T02:15:07Z
body: ## :green_heart: Build Succeeded
* [Buildkite Build](https://buildkite.com/elastic/integrations/builds/25695)
* Commit: 69000be83ecc7e3761ee77e43106d4846bf69084

### History
* :green_heart: [Build #25595](https://buildkite.com/elastic/integrations/builds/25595) succeeded 49e0431f2f184fb327b6fa003e08580f8d441487
* :green_heart: [Build #25389](https://buildkite.com/elastic/integrations/builds/25389) succeeded e5220b460a0043a45680a982887c2653a05914f2
* :green_heart: [Build #25183](https://buildkite.com/elastic/integrations/builds/25183) succeeded 057f53a7cac4508ed74ebc203d545e31d488e3d7
* :green_heart: [Build #25180](https://buildkite.com/elastic/integrations/builds/25180) succeeded 9238c3dd572981f795ab381a1d2f1429e33d277d

<!--buildkite-pr-comment-integrations
{"builds":[{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/integrations/builds/25695","number":25695,"commit":"69000be83ecc7e3761ee77e43106d4846bf69084"},{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/integrations/builds/25595","number":25595,"commit":"49e0431f2f184fb327b6fa003e08580f8d441487"},{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/integrations/builds/25389","number":25389,"commit":"e5220b460a0043a45680a982887c2653a05914f2"},{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/integrations/builds/25183","number":25183,"commit":"057f53a7cac4508ed74ebc203d545e31d488e3d7"},{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/integrations/builds/25180","number":25180,"commit":"9238c3dd572981f795ab381a1d2f1429e33d277d"}],"number":25695}
buildkite-pr-comment-->
author_association: NONE
user: login: taniumalloy
created_at: 2025-05-08T02:24:52Z
body: Thank you for all your help. Please let me know if you need anything else.
author_association: CONTRIBUTOR
user: login: elastic-vault-github-plugin-prod[bot]
created_at: 2025-05-08T02:54:44Z
body: Package cloudflare - 2.31.0 containing this change is available at https://epr.elastic.co/package/cloudflare/2.31.0/
author_association: NONE