Repository: elastic/beats
Pull request #44204: [filebeat][streaming] - Added support for websocket keep_alive heartbeat, updated metrics & docs
Author: login: ShourieG
Labels: name: enhancement
name: docs
name: Filebeat
description: Filebeat
name: Team:Security-Service Integrations
description: Security Service Integrations Team
name: input:streaming
Created: 2025-05-05T12:03:50Z | Merged: 2025-05-14T11:48:37Z

## Type of change
- Enhancement
- Docs


## Proposed commit message
```
1. Added support for websocket keep_alive config option and
implemented keep_alive heartbeat logic in a separate go routine. 
2. Added support for new metrics introduced with heartbeats. 
3. Updated docs accordingly.
4. Added metrics test which was previously missing.
```

<!-- Mandatory
Explain here the changes you made on the PR.

Please explain:

- WHAT: patterns used, algorithms implemented, design architecture, message processing, etc.
- WHY:  the rationale/motivation for the changes

This text will be pasted into the squash dialog when the change is committed and will be
a long term historical record of the change to help future contributors understand the
change, please help them by making it clear and comprehensive, they may be you.

If the commit title is adequate to describe both of these things, The text here may be omitted
or replaced with "See title". The title of the PR will be used as the commit message title when
the merge is made and the "See title" marker will be removed if present.

The text here and the PR title will be subject to the PR review process.
-->

## Checklist

<!-- Mandatory
Add a checklist of things that are required to be reviewed in order to have the PR approved

List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. ~~Scratch this.~~)
-->

- [x] My code follows the style guidelines of this project
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] I have made corresponding change to the default configuration files
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] I have added an entry in `CHANGELOG.next.asciidoc` or `CHANGELOG-developer.next.asciidoc`.

## Disruptive User Impact

<!--
Will the changes introduced by this PR cause disruption to users in any way? If so, please describe what changes users
could make on their end to nullify or minimize this disruption. Consider impacts in related systems, not just directly
when using Beats.
-->

## Note

1. I do not see an efficient way to test this with our current testing setup (still exploring this, thus PR still kept as draft). I believe we will have to rely on manual testing for this.

2. Go channels were not used because of complexity related to channel lifecycles and instead a cancellable context with a manual `cancel()` call is used to avoid any channel related issues.

3. Inbuilt test was not possible for this scenario due to current limitations of the test framework and testing philosophy, hence manual testing was performed with a custom websocket server, for which a screen recording is attached below along with respective filebeat config.

## Author's Checklist

<!-- Recommended
Add a checklist of things that are required to be reviewed in order to have the PR approved
-->
- [ ]

## How to test this PR locally

<!-- Recommended
Explain here how this PR will be tested by the reviewer: commands, dependencies, steps, etc.
-->

## Related issues
- Closes  https://github.com/elastic/beats/issues/42277
- Relates https://github.com/elastic/integrations/issues/11816



## Use cases

Keeps low throughput WebSocket connections from idling out, which would otherwise result in frequent reconnects.

<!-- Recommended
Explain here the different behaviors that this PR introduces or modifies in this project, user roles, environment configuration, etc.

If you are familiar with Gherkin test scenarios, we recommend its usage: https://cucumber.io/docs/gherkin/reference/
-->

## Screenshots

### Screen Recording Of Manual Testing

**filebeat config used:**
```yml
filebeat.inputs:
- type: streaming
  url: ws://localhost:443/ws
  redact:
    fields: ~
  program: |
    bytes(state.response).decode_json().as(body,{
      "events": {
        "message":  body.encode_json(),
      }
    })
  keep_alive:
    enable: true
    interval: 5s
```
**Screen Recording**

**Test WS server is on the left, filebeat streaming input on the right:** 

https://github.com/user-attachments/assets/3b11b86f-5c68-4ae8-a9ff-8b613655db83


<!-- Optional
Add here screenshots about how the project will be changed after the PR is applied. They could be related to web pages, terminal, etc, or any other image you consider important to be shared with the team.
-->

## Test Server Gist
https://gist.github.com/ShourieG/322d38bdfc95cc7314e1729c3b892ae3

## Logs

### Manual Testing Debug Logs: 
```json
{
    "log.level": "debug",
    "@timestamp": "2025-05-13T15:45:31.012+0530",
    "log.logger": "input.streaming",
    "log.origin": {
        "function": "github.com/elastic/beats/v7/x-pack/filebeat/input/streaming.(*keepAlive).heartBeat.func1",
        "file.name": "streaming/websocket.go",
        "file.line": 76
    },
    "message": "received pong message from websocket server",
    "service.name": "filebeat",
    "id": "6E095AFCCE4611A6",
    "input_source": "ws://localhost:443/ws",
    "input_url": "ws://localhost:443/ws",
    "pong_data": "SENT PONG FRAME",
    "ecs.version": "1.6.0"
}
```
<!-- Recommended
Paste here output logs discovered while creating this PR, such as stack traces or integration logs, or any other output you consider important to be shared with the team.
-->



Diff:
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
index a973e8e40828..318eb1ae50a8 100644
--- a/CHANGELOG.next.asciidoc
+++ b/CHANGELOG.next.asciidoc
@@ -423,6 +423,7 @@ otherwise no tag is added. {issue}42208[42208] {pull}42403[42403]
 - Update CEL mito extensions to v1.18.0. {pull}43855[43855]
 - Added input metrics to Azure Blob Storage input. {issue}36641[36641] {pull}43954[43954]
 - Update CEL mito extensions to v1.19.0. {pull}44098[44098]
+- Added support for websocket keep_alive heartbeat in the streaming input. {issue}42277[42277] {pull}44204[44204]
 
 *Auditbeat*
 
diff --git a/docs/reference/filebeat/filebeat-input-streaming.md b/docs/reference/filebeat/filebeat-input-streaming.md
index b9eb9acc2b52..5cf0b2defcdc 100644
--- a/docs/reference/filebeat/filebeat-input-streaming.md
+++ b/docs/reference/filebeat/filebeat-input-streaming.md
@@ -207,6 +207,20 @@ filebeat.inputs:
   url: wss://localhost:443/_stream
 ```
 
+## Keep Alive configuration
+
+The `streaming` input currently supports keep-alive configuration options for streams of `type: websocket`. Use these configuration options to further optimize the stability
+of your WebSocket connections and prevent them from idling out.
+
+The `keep_alive` setting has the following configuration options:
+
+* `enable`: Indicates whether Keep-Alive is enabled. By default, this is set to `false`.
+* `interval`: Interval between Keep-Alive messages, expressed as a time duration value. The default value is `30s`.
+* `write_control_deadline`: Deadline for writing control frames, like `PING`, `PONG`, or `CLOSE`, on a WebSocket connection. The timeout, expressed as a time duration value, helps prevent indefinite blocking when the server or client is not responding to control frame requests. The default value is `10s`.
+   
+::::{note}
+Don't use the `blanket_retries` and `infinite_retries` configuration options together with the `keep_alive` settings. The purpose of `keep_alive` is to keep the connection open so you don't need to `retry` and reconnect all the time. In some scenarios `keep_alive` might not work if the host WebSocket server is not configured to handle `ping` frames.
+::::
 
 ## Input state [input-state-streaming]
 
@@ -428,8 +442,11 @@ This input exposes metrics under the [HTTP monitoring endpoint](/reference/fileb
 | `received_bytes_total` | Number of bytes received over the life cycle of the input. |
 | `events_received_total` | Number of events received. |
 | `events_published_total` | Number of events published. |
+| `write_control_errors` | Number of errors encountered for write control operations. |
 | `cel_processing_time` | Histogram of the elapsed successful CEL program processing times in nanoseconds. |
 | `batch_processing_time` | Histogram of the elapsed successful batch processing times in nanoseconds (time of receipt to time of ACK for non-empty batches). |
+| `ping_message_send_time` | Histogram of the elapsed successful ping message send times in nanoseconds. |
+| `pong_message_received_time` | Histogram of the elapsed successful pong message receive times in nanoseconds. |
 
 
 ## Developer tools [_developer_tools_2]
diff --git a/x-pack/filebeat/input/streaming/config.go b/x-pack/filebeat/input/streaming/config.go
index 753c36febf3a..63e08891963a 100644
--- a/x-pack/filebeat/input/streaming/config.go
+++ b/x-pack/filebeat/input/streaming/config.go
@@ -50,6 +50,8 @@ type config struct {
 	Retry *retry `config:"retry"`
 	// Transport is the common the transport config.
 	Transport httpcommon.HTTPTransportSettings `config:",inline"`
+	// KeepAlive is the configuration for keep-alive settings.
+	KeepAlive keepAliveConfig `config:"keep_alive"`
 	// CrowdstrikeAppID is the value used to set the
 	// appId request parameter in the FalconHose stream
 	// discovery request.
@@ -73,6 +75,20 @@ type retry struct {
 	InfiniteRetries bool          `config:"infinite_retries"`
 }
 
+// keepAliveConfig is the configuration for keep-alive settings.
+type keepAliveConfig struct {
+	// Enable indicates whether keep-alive is enabled.
+	Enable bool `config:"enable"`
+	// Interval is the interval between keep-alive messages.
+	// by default, this is set to 30 seconds.
+	Interval time.Duration `config:"interval"`
+	// WriteControlDeadline is the deadline for write control messages.
+	// by default, this is set to 10 seconds.
+	WriteControlDeadline time.Duration `config:"write_control_deadline"`
+	// readControlDeadline is the deadline for read control messages.
+	// this is internally set to 3x the value of the writeControlDeadline to account for network latency/jitter.
+	readControlDeadline time.Duration
+}
 type authConfig struct {
 	// Custom auth config to use for authentication.
 	CustomAuth *customAuthConfig `config:"custom"`
@@ -215,5 +231,9 @@ func defaultConfig() config {
 			WaitMin:     1 * time.Second,
 			WaitMax:     30 * time.Second,
 		},
+		KeepAlive: keepAliveConfig{
+			Interval:             30 * time.Second,
+			WriteControlDeadline: 10 * time.Second,
+		},
 	}
 }
diff --git a/x-pack/filebeat/input/streaming/crowdstrike.go b/x-pack/filebeat/input/streaming/crowdstrike.go
index 47421656bde7..5d3520490b28 100644
--- a/x-pack/filebeat/input/streaming/crowdstrike.go
+++ b/x-pack/filebeat/input/streaming/crowdstrike.go
@@ -50,7 +50,7 @@ func NewFalconHoseFollower(ctx context.Context, id string, cfg config, cursor ma
 			pub:     pub,
 			log:     log,
 			redact:  cfg.Redact,
-			metrics: newInputMetrics(id),
+			metrics: newInputMetrics(id, nil),
 		},
 		creds: &clientcredentials.Config{
 			ClientID:       cfg.Auth.OAuth2.ClientID,
diff --git a/x-pack/filebeat/input/streaming/input_manager.go b/x-pack/filebeat/input/streaming/input_manager.go
index 7d2d60a7834a..88f2c3b7fd82 100644
--- a/x-pack/filebeat/input/streaming/input_manager.go
+++ b/x-pack/filebeat/input/streaming/input_manager.go
@@ -35,7 +35,10 @@ func NewInputManager(log *logp.Logger, store statestore.States) InputManager {
 }
 
 func cursorConfigure(cfg *conf.C) ([]inputcursor.Source, inputcursor.Input, error) {
-	src := &source{cfg: defaultConfig()}
+	dc := defaultConfig()
+	// set readControlDeadline to 3x the writeControlDeadline
+	dc.KeepAlive.readControlDeadline = 3 * dc.KeepAlive.WriteControlDeadline
+	src := &source{cfg: dc}
 	if err := cfg.Unpack(&src.cfg); err != nil {
 		return nil, nil, err
 	}
diff --git a/x-pack/filebeat/input/streaming/metrics.go b/x-pack/filebeat/input/streaming/metrics.go
index 6f94b5e888b3..dcad49c655b3 100644
--- a/x-pack/filebeat/input/streaming/metrics.go
+++ b/x-pack/filebeat/input/streaming/metrics.go
@@ -14,38 +14,48 @@ import (
 
 // inputMetrics handles the input's metric reporting.
 type inputMetrics struct {
-	unregister          func()
-	url                 *monitoring.String // URL of the input resource
-	celEvalErrors       *monitoring.Uint   // number of errors encountered during cel program evaluation
-	batchesReceived     *monitoring.Uint   // number of event arrays received
-	errorsTotal         *monitoring.Uint   // number of errors encountered
-	receivedBytesTotal  *monitoring.Uint   // number of bytes received
-	eventsReceived      *monitoring.Uint   // number of events received
-	batchesPublished    *monitoring.Uint   // number of event arrays published
-	eventsPublished     *monitoring.Uint   // number of events published
-	celProcessingTime   metrics.Sample     // histogram of the elapsed successful cel program processing times in nanoseconds
-	batchProcessingTime metrics.Sample     // histogram of the elapsed successful batch processing times in nanoseconds (time of receipt to time of ACK for non-empty batches).
+	unregister              func()
+	url                     *monitoring.String // URL of the input resource
+	celEvalErrors           *monitoring.Uint   // number of errors encountered during cel program evaluation
+	batchesReceived         *monitoring.Uint   // number of event arrays received
+	errorsTotal             *monitoring.Uint   // number of errors encountered
+	receivedBytesTotal      *monitoring.Uint   // number of bytes received
+	eventsReceived          *monitoring.Uint   // number of events received
+	batchesPublished        *monitoring.Uint   // number of event arrays published
+	eventsPublished         *monitoring.Uint   // number of events published
+	writeControlErrors      *monitoring.Uint   // number of errors encountered while sending write control messages like ping
+	celProcessingTime       metrics.Sample     // histogram of the elapsed successful cel program processing times in nanoseconds
+	batchProcessingTime     metrics.Sample     // histogram of the elapsed successful batch processing times in nanoseconds (time of receipt to time of ACK for non-empty batches).
+	pingMessageSendTime     metrics.Sample     // histogram of the elapsed successful ping message send times in nanoseconds
+	pongMessageReceivedTime metrics.Sample     // histogram of the elapsed successful pong message receive times in nanoseconds
 }
 
-func newInputMetrics(id string) *inputMetrics {
-	reg, unreg := inputmon.NewInputRegistry(inputName, id, nil)
+func newInputMetrics(id string, optionalParent *monitoring.Registry) *inputMetrics {
+	reg, unreg := inputmon.NewInputRegistry(inputName, id, optionalParent)
 	out := &inputMetrics{
-		unregister:          unreg,
-		url:                 monitoring.NewString(reg, "url"),
-		celEvalErrors:       monitoring.NewUint(reg, "cel_eval_errors"),
-		batchesReceived:     monitoring.NewUint(reg, "batches_received_total"),
-		errorsTotal:         monitoring.NewUint(reg, "errors_total"),
-		receivedBytesTotal:  monitoring.NewUint(reg, "received_bytes_total"),
-		eventsReceived:      monitoring.NewUint(reg, "events_received_total"),
-		batchesPublished:    monitoring.NewUint(reg, "batches_published_total"),
-		eventsPublished:     monitoring.NewUint(reg, "events_published_total"),
-		celProcessingTime:   metrics.NewUniformSample(1024),
-		batchProcessingTime: metrics.NewUniformSample(1024),
+		unregister:              unreg,
+		url:                     monitoring.NewString(reg, "url"),
+		celEvalErrors:           monitoring.NewUint(reg, "cel_eval_errors"),
+		batchesReceived:         monitoring.NewUint(reg, "batches_received_total"),
+		errorsTotal:             monitoring.NewUint(reg, "errors_total"),
+		receivedBytesTotal:      monitoring.NewUint(reg, "received_bytes_total"),
+		eventsReceived:          monitoring.NewUint(reg, "events_received_total"),
+		batchesPublished:        monitoring.NewUint(reg, "batches_published_total"),
+		eventsPublished:         monitoring.NewUint(reg, "events_published_total"),
+		writeControlErrors:      monitoring.NewUint(reg, "write_control_errors"),
+		celProcessingTime:       metrics.NewUniformSample(1024),
+		batchProcessingTime:     metrics.NewUniformSample(1024),
+		pingMessageSendTime:     metrics.NewUniformSample(1024),
+		pongMessageReceivedTime: metrics.NewUniformSample(1024),
 	}
 	_ = adapter.NewGoMetrics(reg, "cel_processing_time", adapter.Accept).
 		Register("histogram", metrics.NewHistogram(out.celProcessingTime))
 	_ = adapter.NewGoMetrics(reg, "batch_processing_time", adapter.Accept).
 		Register("histogram", metrics.NewHistogram(out.batchProcessingTime))
+	_ = adapter.NewGoMetrics(reg, "ping_message_send_time", adapter.Accept).
+		Register("histogram", metrics.NewHistogram(out.pingMessageSendTime))
+	_ = adapter.NewGoMetrics(reg, "pong_message_received_time", adapter.Accept).
+		Register("histogram", metrics.NewHistogram(out.pongMessageReceivedTime))
 
 	return out
 }
diff --git a/x-pack/filebeat/input/streaming/metrics_test.go b/x-pack/filebeat/input/streaming/metrics_test.go
new file mode 100644
index 000000000000..bbbaf0768ac3
--- /dev/null
+++ b/x-pack/filebeat/input/streaming/metrics_test.go
@@ -0,0 +1,64 @@
+// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+// or more contributor license agreements. Licensed under the Elastic License;
+// you may not use this file except in compliance with the Elastic License.
+
+package streaming
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+
+	"github.com/elastic/elastic-agent-libs/monitoring"
+)
+
+// TestInputMetricsClose asserts that metrics registered by this input are
+// removed after Close() is called. This is important because an input with
+// the same ID could be re-registered, and that ID cannot exist in the
+// monitoring registry.
+func TestInputMetricsClose(t *testing.T) {
+	reg := monitoring.NewRegistry()
+
+	metrics := newInputMetrics("streaming-8b312b5f-9f99-492c-b035-3dff354a1f01", reg)
+	metrics.Close()
+
+	reg.Do(monitoring.Full, func(s string, _ interface{}) {
+		t.Errorf("registry should be empty, but found %v", s)
+	})
+}
+
+// TestNewInputMetricsInstance asserts that all the metrics are initialized
+// when a newInputMetrics method is invoked. This avoids nil hit panics when
+// a getter is invoked on any uninitialized metric.
+func TestNewInputMetricsInstance(t *testing.T) {
+	reg := monitoring.NewRegistry()
+	metrics := newInputMetrics("streaming-metric-test", reg)
+
+	assert.NotNil(t,
+		metrics.errorsTotal,
+		metrics.celEvalErrors,
+		metrics.batchesReceived,
+		metrics.receivedBytesTotal,
+		metrics.eventsReceived,
+		metrics.batchesPublished,
+		metrics.eventsPublished,
+		metrics.writeControlErrors,
+		metrics.celProcessingTime,
+		metrics.batchProcessingTime,
+		metrics.pingMessageSendTime,
+		metrics.pongMessageReceivedTime,
+	)
+
+	assert.Equal(t, uint64(0x0), metrics.errorsTotal.Get())
+	assert.Equal(t, uint64(0x0), metrics.celEvalErrors.Get())
+	assert.Equal(t, uint64(0x0), metrics.batchesReceived.Get())
+	assert.Equal(t, uint64(0x0), metrics.receivedBytesTotal.Get())
+	assert.Equal(t, uint64(0x0), metrics.eventsReceived.Get())
+	assert.Equal(t, uint64(0x0), metrics.batchesPublished.Get())
+	assert.Equal(t, uint64(0x0), metrics.eventsPublished.Get())
+	assert.Equal(t, uint64(0x0), metrics.writeControlErrors.Get())
+	assert.Equal(t, int64(0), metrics.celProcessingTime.Count())
+	assert.Equal(t, int64(0), metrics.batchProcessingTime.Count())
+	assert.Equal(t, int64(0), metrics.pingMessageSendTime.Count())
+	assert.Equal(t, int64(0), metrics.pongMessageReceivedTime.Count())
+}
diff --git a/x-pack/filebeat/input/streaming/websocket.go b/x-pack/filebeat/input/streaming/websocket.go
index 6c1970e08488..5d41ccfaaa76 100644
--- a/x-pack/filebeat/input/streaming/websocket.go
+++ b/x-pack/filebeat/input/streaming/websocket.go
@@ -39,6 +39,7 @@ type websocketStream struct {
 	tokenSource oauth2.TokenSource
 	tokenExpiry <-chan time.Time
 	time        func() time.Time
+	keepAlive   *keepAlive
 }
 
 type loggingRoundTripper struct {
@@ -46,6 +47,13 @@ type loggingRoundTripper struct {
 	log *logp.Logger
 }
 
+// keepAlive manages the configuration and metrics for WebSocket keep-alive functionality
+type keepAlive struct {
+	cfg     keepAliveConfig
+	metrics *inputMetrics
+	log     *logp.Logger
+}
+
 func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
 	resp, err := l.rt.RoundTrip(req)
 	// avoided logging request and and response body as it may contain sensitive information and can be huge
@@ -58,6 +66,45 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(pongData string) error {
+		k.log.Debugw("received pong message from websocket server", "pong_data", pongData)
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ctx.Done():
+				k.log.Debugw("heartbeat stopped")
+				return
+			case now := <-ticker.C:
+				err := conn.WriteControl(websocket.PingMessage, nil, now.Add(k.cfg.WriteControlDeadline))
+				if err != nil {
+					k.log.Debugw("error sending ping control frame to websocket server", "error", err)
+					k.metrics.writeControlErrors.Inc()
+					k.metrics.errorsTotal.Inc()
+				} else {
+					k.log.Debugw("sent ping control frame to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				}
+			}
+		}
+	}()
+
+	return cancel
+}
+
 // NewWebsocketFollower performs environment construction including CEL
 // program and regexp compilation, and input metrics set-up for a websocket
 // stream follower.
@@ -71,7 +118,7 @@ func NewWebsocketFollower(ctx context.Context, id string, cfg config, cursor map
 			pub:     pub,
 			log:     log,
 			redact:  cfg.Redact,
-			metrics: newInputMetrics(id),
+			metrics: newInputMetrics(id, nil),
 		},
 		// the token expiry handler will never trigger unless a valid expiry time is assigned
 		tokenExpiry: nil,
@@ -108,6 +155,17 @@ func NewWebsocketFollower(ctx context.Context, id string, cfg config, cursor map
 		s.tokenExpiry = time.After(time.Until(token.Expiry) - cfg.Auth.OAuth2.TokenExpiryBuffer)
 	}
 
+	// set and assign KeepAlive if enabled
+	if cfg.KeepAlive.Enable {
+		// create a new keepAlive instance
+		k := &keepAlive{
+			cfg:     cfg.KeepAlive,
+			metrics: s.metrics,
+			log:     log,
+		}
+		s.keepAlive = k
+	}
+
 	patterns, err := regexpsFromConfig(cfg)
 	if err != nil {
 		s.metrics.errorsTotal.Inc()
@@ -128,7 +186,9 @@ func NewWebsocketFollower(ctx context.Context, id string, cfg config, cursor map
 // FollowStream receives, processes and publishes events from the subscribed
 // websocket stream.
 func (s *websocketStream) FollowStream(ctx context.Context) error {
+	var heartBeatCancel context.CancelFunc
 	state := s.cfg.State
+
 	if state == nil {
 		state = make(map[string]any)
 	}
@@ -151,6 +211,12 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 		s.log.Errorw("failed to establish websocket connection", "error", err)
 		return err
 	}
+	// Start the keep-alive routine if enabled and the connection is established successfully
+	// this is for the initial connection only, the keep-alive will be restarted during the reconnect
+	// logic/token refresh.
+	if s.keepAlive != nil {
+		heartBeatCancel = s.keepAlive.heartBeat(ctx, c, s.now().In(time.UTC))
+	}
 
 	// ensures this is the last connection closed when the function returns
 	defer func() {
@@ -164,11 +230,18 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 
 	for {
 		select {
+		// If the keep-alive is enabled, heartbeat will be automatically cancelled when
+		// the parent context is done().
 		case <-ctx.Done():
 			s.log.Debugw("context cancelled, closing websocket connection")
 			return ctx.Err()
 		// s.tokenExpiry channel will only trigger if oauth2 is enabled and the token is about to expire
 		case <-s.tokenExpiry:
+			// Cancel the keep-alive routine if enabled since we need to establish a new
+			// connection instance with the refreshed token.
+			if s.keepAlive != nil {
+				heartBeatCancel()
+			}
 			// get the new token
 			token, err := s.tokenSource.Token()
 			if err != nil {
@@ -177,9 +250,11 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 				return err
 			}
 			// gracefully close current connection
-			if err := c.Close(); err != nil {
-				s.metrics.errorsTotal.Inc()
-				s.log.Errorw("encountered an error while closing the existing websocket connection during token refresh", "error", err)
+			if c != nil {
+				if err := c.Close(); err != nil {
+					s.metrics.errorsTotal.Inc()
+					s.log.Errorw("encountered an error while closing the existing websocket connection during token refresh", "error", err)
+				}
 			}
 			// set the new token in the config
 			s.cfg.Auth.OAuth2.accessToken = token.AccessToken
@@ -193,21 +268,33 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 				s.log.Errorw("failed to establish a new websocket connection on token refresh", "error", err)
 				return err
 			}
+			// Restart the keep-alive routine on a token refresh if enabled and the
+			// connection is established successfully.
+			if s.keepAlive != nil {
+				heartBeatCancel = s.keepAlive.heartBeat(ctx, c, s.now().In(time.UTC))
+			}
 		default:
 			_, message, err := c.ReadMessage()
 			if err != nil {
 				s.metrics.errorsTotal.Inc()
+				// Cancel the keep-alive routine if enabled since we need to establish a
+				// new connection via our reconnect logic.
+				if s.keepAlive != nil {
+					heartBeatCancel()
+				}
 				if !s.cfg.Retry.BlanketRetries && !isRetryableError(err) {
 					s.log.Errorw("failed to read websocket data", "error", err)
 					return err
 				}
 				s.log.Debugw("websocket connection encountered an error, attempting to reconnect...", "error", err)
-				// close the old connection and reconnect
-				if err := c.Close(); err != nil {
-					s.metrics.errorsTotal.Inc()
-					s.log.Errorw("encountered an error while closing the websocket connection", "error", err)
+				if c != nil {
+					if err := c.Close(); err != nil {
+						s.metrics.errorsTotal.Inc()
+						s.log.Errorw("encountered an error while closing the websocket connection", "error", err)
+					}
 				}
-				// since c is already a pointer, we can reassign it to the new connection and the defer func will still handle it
+				// Since c is already a pointer, we can reassign it to the new connection
+				// and the defer func will still handle it.
 				c, resp, err = connectWebSocket(ctx, s.cfg, url, s.log)
 				handleConnectionResponse(resp, s.metrics, s.log)
 				if err != nil {
@@ -215,8 +302,14 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 					s.log.Errorw("failed to reconnect websocket connection", "error", err)
 					return err
 				}
+				// Restart the keep-alive routine if enabled after a successful reconnection.
+				if s.keepAlive != nil {
+					heartBeatCancel = s.keepAlive.heartBeat(ctx, c, s.now().In(time.UTC))
+				}
+
 				continue
 			}
+
 			s.metrics.receivedBytesTotal.Add(uint64(len(message)))
 			state["response"] = message
 			s.log.Debugw("received websocket message", logp.Namespace(s.ns), "msg", string(message))


Reviews:
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T12:08:11Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:05:32Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T05:21:42Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T05:35:32Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T05:43:59Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T05:51:29Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T06:03:56Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T23:10:42Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T23:12:18Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T01:28:42Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T01:50:30Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T01:54:01Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T05:59:24Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T08:40:31Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T10:50:37Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T00:33:16Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T05:04:33Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T05:08:41Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T05:12:57Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T05:14:19Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T07:41:57Z
user: login: efd6
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T21:37:47Z
user: login: theletterf
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-14T07:47:15Z
user: login: theletterf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-14T07:57:36Z
user: login: theletterf
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-14T10:09:16Z

Review comments:
diff_hunk: @@ -35,7 +35,10 @@ func NewInputManager(log *logp.Logger, store statestore.States) InputManager {
 }
 
 func cursorConfigure(cfg *conf.C) ([]inputcursor.Source, inputcursor.Input, error) {
-	src := &source{cfg: defaultConfig()}
+	dc := defaultConfig()
+	// set readControlDeadline to 3x the writeControlDeadline
path: x-pack/filebeat/input/streaming/input_manager.go
user: login: ShourieG
body: this is a standard convention
created_at: 2025-05-05T12:08:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,7 +35,10 @@ func NewInputManager(log *logp.Logger, store statestore.States) InputManager {
 }
 
 func cursorConfigure(cfg *conf.C) ([]inputcursor.Source, inputcursor.Input, error) {
-	src := &source{cfg: defaultConfig()}
+	dc := defaultConfig()
+	// set readControlDeadline to 3x the writeControlDeadline
path: x-pack/filebeat/input/streaming/input_manager.go
user: login: efd6
body: Link?
created_at: 2025-05-06T00:57:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
+					}
+					k.log.Debugw("sent ping message to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				} else {
+					k.log.Debugw("websocket connection is nil but heartbeat routine is still running!")
+				}
+			case <-ctx.Done():
+				k.log.Debugw("heartbeat stopped")
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: Move this to the bottom of the function so that it picks up all stops?
created_at: 2025-05-06T01:00:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: Do we want to return here? Is there anything else that we need to do other than just taking our bat and ball and going home?
created_at: 2025-05-06T01:01:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
+					}
+					k.log.Debugw("sent ping message to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				} else {
+					k.log.Debugw("websocket connection is nil but heartbeat routine is still running!")
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: Does this happen? If it does, should we not return since `conn` is never going to be non-nil after this.
created_at: 2025-05-06T01:03:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
			case now := <-ticker.C:
				if conn != nil {
					err := conn.WriteControl(websocket.PingMessage, nil, now.Add(k.cfg.KeepAlive.WriteControlDeadline))
```
created_at: 2025-05-06T01:04:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,7 +35,10 @@ func NewInputManager(log *logp.Logger, store statestore.States) InputManager {
 }
 
 func cursorConfigure(cfg *conf.C) ([]inputcursor.Source, inputcursor.Input, error) {
-	src := &source{cfg: defaultConfig()}
+	dc := defaultConfig()
+	// set readControlDeadline to 3x the writeControlDeadline
path: x-pack/filebeat/input/streaming/input_manager.go
user: login: ShourieG
body: I should reframe the statement here for better clarity, it is generally the case that `readControlDeadline > writeControlDeadline` since we need to give the ws server some time to respond with a `pong control frame`. The `3x` amount was my choice since it seems like a nice buffer accounting for network jitter. 
created_at: 2025-05-07T05:21:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: No this `return` should be removed. We want the heartbeat routine to continue irrespective of write_control errors and let our retry logic decide when to quit. If such a scenario occurs when the infinite & blanket retries are also simultaneously enabled - (this is clearly not recommended in the doc), these debug logs will indicate the server is clearly unresponsive.
created_at: 2025-05-07T05:35:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
+					}
+					k.log.Debugw("sent ping message to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				} else {
+					k.log.Debugw("websocket connection is nil but heartbeat routine is still running!")
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: This should ideally never happen, but I've been recently burned by gorilla/websocket's weird behaviour of [returning nil response objects](https://github.com/elastic/beats/pull/44068) due to server crash/sudden disconnects, hence put this here for any such unexpected events.
created_at: 2025-05-07T05:43:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
+					}
+					k.log.Debugw("sent ping message to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				} else {
+					k.log.Debugw("websocket connection is nil but heartbeat routine is still running!")
+				}
+			case <-ctx.Done():
+				k.log.Debugw("heartbeat stopped")
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: It's already at the bottom of the select block (though that should not matter), do you want the `debug log` to be present at the bottom outside the select ? 
created_at: 2025-05-07T05:51:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,48 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case now := <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, nil, now.Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping control frame to websocket server:", err)
+						k.metrics.writeControlErrors.Inc()
+						k.metrics.errorsTotal.Inc()
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: @efd6, Would it be the correct option to increase the total errors here ? This could increase error metrics bloat due to various network issues but according to the design principle we currently have, this seems like the correct thing to do.

created_at: 2025-05-07T06:03:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
+					}
+					k.log.Debugw("sent ping message to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				} else {
+					k.log.Debugw("websocket connection is nil but heartbeat routine is still running!")
+				}
+			case <-ctx.Done():
+				k.log.Debugw("heartbeat stopped")
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: > do you want the `debug log` to be present at the bottom outside the select ?

In the previous state of the code it was necessary to have it outside the select due to the return in the other case. Since that `return` has been removed, it's fine to leave it here now.
created_at: 2025-05-07T23:10:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,48 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case now := <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, nil, now.Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping control frame to websocket server:", err)
+						k.metrics.writeControlErrors.Inc()
+						k.metrics.errorsTotal.Inc()
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: Maybe add a ping errors metric? That would be possible to then be used to calculate the non-ping errors from the total.
created_at: 2025-05-07T23:12:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,48 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case now := <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, nil, now.Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping control frame to websocket server:", err)
+						k.metrics.writeControlErrors.Inc()
+						k.metrics.errorsTotal.Inc()
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: Yes already added it as write_control errors
created_at: 2025-05-08T01:28:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,48 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case now := <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, nil, now.Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping control frame to websocket server:", err)
+						k.metrics.writeControlErrors.Inc()
+						k.metrics.errorsTotal.Inc()
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: In that case, I think this is fine.
created_at: 2025-05-08T01:50:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
+					}
+					k.log.Debugw("sent ping message to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				} else {
+					k.log.Debugw("websocket connection is nil but heartbeat routine is still running!")
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: OK, so we should return then. Once `conn` is nil, there is no opportunity for it to become non-nil. In fact, the if should be hoisted out to the preamble for `(*keepAlive).heartBeat` since it is not possible for it to change its nilness from that point.
created_at: 2025-05-08T01:54:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
+					}
+					k.log.Debugw("sent ping message to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				} else {
+					k.log.Debugw("websocket connection is nil but heartbeat routine is still running!")
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: Yes that could be done, but that would only provide safeguard against the initial trigger. What if the `conn` becomes `nil` when the routine is running because of some unexpected reason? This would cause the `conn.WriteControl` to panic. This was my thinking behind putting this check inside the select block.

In this scenario the the parent(main) routine will cancel the child routine context if there's an appropriate error, but there might be a small gap between the signals to already cause the panic to happen.
created_at: 2025-05-08T05:59:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
+					}
+					k.log.Debugw("sent ping message to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				} else {
+					k.log.Debugw("websocket connection is nil but heartbeat routine is still running!")
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: > What if the conn becomes nil when the routine is running because of some unexpected reason?

Can you show me a mechanism whereby that can happen that does not involve memory corruption?
created_at: 2025-05-08T08:40:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,47 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.KeepAlive.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.KeepAlive.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ticker.C:
+				if conn != nil {
+					err := conn.WriteControl(websocket.PingMessage, []byte{}, time.Now().Add(k.cfg.KeepAlive.WriteControlDeadline))
+					if err != nil {
+						k.log.Debugw("error sending ping message to websocket server:", err)
+						return
+					}
+					k.log.Debugw("sent ping message to websocket server")
+					k.metrics.pingMessageSendTime.Update(time.Since(start).Nanoseconds())
+				} else {
+					k.log.Debugw("websocket connection is nil but heartbeat routine is still running!")
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: So I went through the whole control flow properly and yes, it seems this is unlikely to happen besides a in the case of a memory corruption. 

1. We only ever call `keepAlive.heartBeat()` if a connection is successfully established.
2. We always call `heartBeatCancel()` to gracefully stop this routine in all error/context cancellation/token  expiry scenarios.

So I'll remove this check and we don't even require a preamble check to exist, since we only trigger this on a successful connection.
created_at: 2025-05-08T10:50:37Z
author_association: CONTRIBUTOR
path: docs/reference/filebeat/filebeat-input-streaming.md
user: login: efd6
body: Add metric docs for the change here.
created_at: 2025-05-13T00:17:35Z
author_association: CONTRIBUTOR
path: x-pack/filebeat/input/streaming/metrics_test.go
user: login: efd6
body: Are these not testing other packages? Do we need to do this?
created_at: 2025-05-13T00:19:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,13 +39,20 @@ type websocketStream struct {
 	tokenSource oauth2.TokenSource
 	tokenExpiry <-chan time.Time
 	time        func() time.Time
+	keepAlive   *keepAlive
 }
 
 type loggingRoundTripper struct {
 	rt  http.RoundTripper
 	log *logp.Logger
 }
 
+type keepAlive struct {
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: Godoc for this please.
created_at: 2025-05-13T00:20:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,13 +39,20 @@ type websocketStream struct {
 	tokenSource oauth2.TokenSource
 	tokenExpiry <-chan time.Time
 	time        func() time.Time
+	keepAlive   *keepAlive
 }
 
 type loggingRoundTripper struct {
 	rt  http.RoundTripper
 	log *logp.Logger
 }
 
+type keepAlive struct {
+	cfg     config
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
	cfg     KeepAlive
```
with associated `s/.KeepAlive//g`.
created_at: 2025-05-13T00:24:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -73,6 +75,20 @@ type retry struct {
 	InfiniteRetries bool          `config:"infinite_retries"`
 }
 
+// KeepAlive is the configuration for keep-alive settings.
+type KeepAlive struct {
path: x-pack/filebeat/input/streaming/config.go
user: login: efd6
body: Is there a reason this needs to be exported? If it's just to avoid name collision, rename this to `keepAliveConfig`.
created_at: 2025-05-13T00:25:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -151,6 +210,11 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 		s.log.Errorw("failed to establish websocket connection", "error", err)
 		return err
 	}
+	// start the keep-alive routine if enabled and the connection is established successfully
+	// this is for the initial connection only, the keep-alive will be restarted during the reconnect logic/ token refresh
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
	// Start the keep-alive routine if enabled and the connection is established successfully
	// this is for the initial connection only, the keep-alive will be restarted during the reconnect
	// logic/token refresh.
```
created_at: 2025-05-13T00:28:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -164,11 +228,16 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 
 	for {
 		select {
+		// if the keep-alive is enabled, heartbeat will be automatically cancelled when the parent context is done()
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
		// If the keep-alive is enabled, heartbeat will be automatically cancelled when
		// the parent context is done().
```
created_at: 2025-05-13T00:28:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -164,11 +228,16 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 
 	for {
 		select {
+		// if the keep-alive is enabled, heartbeat will be automatically cancelled when the parent context is done()
 		case <-ctx.Done():
 			s.log.Debugw("context cancelled, closing websocket connection")
 			return ctx.Err()
 		// s.tokenExpiry channel will only trigger if oauth2 is enabled and the token is about to expire
 		case <-s.tokenExpiry:
+			// cancel the keep-alive routine if enabled since we need to establish a new connection instance with the refreshed token
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
			// Cancel the keep-alive routine if enabled since we need to establish a new
			// connection instance with the refreshed token.
```
created_at: 2025-05-13T00:28:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -193,19 +264,29 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 				s.log.Errorw("failed to establish a new websocket connection on token refresh", "error", err)
 				return err
 			}
+			// restart the keep-alive routine on a token refresh if enabled and the connection is established successfully
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
			// Restart the keep-alive routine on a token refresh if enabled and the
			// connection is established successfully.
```
created_at: 2025-05-13T00:29:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -193,19 +264,29 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 				s.log.Errorw("failed to establish a new websocket connection on token refresh", "error", err)
 				return err
 			}
+			// restart the keep-alive routine on a token refresh if enabled and the connection is established successfully
+			if s.keepAlive != nil {
+				heartBeatCancel = s.keepAlive.heartBeat(ctx, c, s.now().In(time.UTC))
+			}
 		default:
 			_, message, err := c.ReadMessage()
 			if err != nil {
 				s.metrics.errorsTotal.Inc()
+				// cancel the keep-alive routine if enabled since we need to establish a new connection via our reconnect logic
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
				// Cancel the keep-alive routine if enabled since we need to establish a
				// new connection via our reconnect logic.
```
created_at: 2025-05-13T00:29:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -193,19 +264,29 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 				s.log.Errorw("failed to establish a new websocket connection on token refresh", "error", err)
 				return err
 			}
+			// restart the keep-alive routine on a token refresh if enabled and the connection is established successfully
+			if s.keepAlive != nil {
+				heartBeatCancel = s.keepAlive.heartBeat(ctx, c, s.now().In(time.UTC))
+			}
 		default:
 			_, message, err := c.ReadMessage()
 			if err != nil {
 				s.metrics.errorsTotal.Inc()
+				// cancel the keep-alive routine if enabled since we need to establish a new connection via our reconnect logic
+				if s.keepAlive != nil {
+					heartBeatCancel()
+				}
 				if !s.cfg.Retry.BlanketRetries && !isRetryableError(err) {
 					s.log.Errorw("failed to read websocket data", "error", err)
 					return err
 				}
 				s.log.Debugw("websocket connection encountered an error, attempting to reconnect...", "error", err)
-				// close the old connection and reconnect
-				if err := c.Close(); err != nil {
-					s.metrics.errorsTotal.Inc()
-					s.log.Errorw("encountered an error while closing the websocket connection", "error", err)
+				// close the old connection gracefully and reconnect
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
				// Close the old connection gracefully and reconnect.
```
(but do we need this comment — see [Pike Style](http://doc.cat-v.org/bell_labs/pikestyle) in the Comments section)
created_at: 2025-05-13T00:30:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -193,19 +264,29 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 				s.log.Errorw("failed to establish a new websocket connection on token refresh", "error", err)
 				return err
 			}
+			// restart the keep-alive routine on a token refresh if enabled and the connection is established successfully
+			if s.keepAlive != nil {
+				heartBeatCancel = s.keepAlive.heartBeat(ctx, c, s.now().In(time.UTC))
+			}
 		default:
 			_, message, err := c.ReadMessage()
 			if err != nil {
 				s.metrics.errorsTotal.Inc()
+				// cancel the keep-alive routine if enabled since we need to establish a new connection via our reconnect logic
+				if s.keepAlive != nil {
+					heartBeatCancel()
+				}
 				if !s.cfg.Retry.BlanketRetries && !isRetryableError(err) {
 					s.log.Errorw("failed to read websocket data", "error", err)
 					return err
 				}
 				s.log.Debugw("websocket connection encountered an error, attempting to reconnect...", "error", err)
-				// close the old connection and reconnect
-				if err := c.Close(); err != nil {
-					s.metrics.errorsTotal.Inc()
-					s.log.Errorw("encountered an error while closing the websocket connection", "error", err)
+				// close the old connection gracefully and reconnect
+				if c != nil {
+					if err := c.Close(); err != nil {
+						s.metrics.errorsTotal.Inc()
+						s.log.Errorw("encountered an error while closing the websocket connection", "error", err)
+					}
 				}
 				// since c is already a pointer, we can reassign it to the new connection and the defer func will still handle it
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
				// Since c is already a pointer, we can reassign it to the new connection
				// and the defer func will still handle it.
```
created_at: 2025-05-13T00:31:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -215,8 +296,14 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 					s.log.Errorw("failed to reconnect websocket connection", "error", err)
 					return err
 				}
+				// restart the keep-alive routine if enabled after a successful reconnection
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
				// Restart the keep-alive routine if enabled after a successful reconnection.
```
created_at: 2025-05-13T00:31:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,45 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: Is there a reason that this needs to be a method on a type, rather than a pure func or a closure?
created_at: 2025-05-13T00:32:41Z
author_association: CONTRIBUTOR
path: x-pack/filebeat/input/streaming/metrics_test.go
user: login: ShourieG
body: This seems to be a standard in the other inputs like s3, gcs and abs, so figured I'd include here as well
created_at: 2025-05-13T05:04:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -193,19 +264,29 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 				s.log.Errorw("failed to establish a new websocket connection on token refresh", "error", err)
 				return err
 			}
+			// restart the keep-alive routine on a token refresh if enabled and the connection is established successfully
+			if s.keepAlive != nil {
+				heartBeatCancel = s.keepAlive.heartBeat(ctx, c, s.now().In(time.UTC))
+			}
 		default:
 			_, message, err := c.ReadMessage()
 			if err != nil {
 				s.metrics.errorsTotal.Inc()
+				// cancel the keep-alive routine if enabled since we need to establish a new connection via our reconnect logic
+				if s.keepAlive != nil {
+					heartBeatCancel()
+				}
 				if !s.cfg.Retry.BlanketRetries && !isRetryableError(err) {
 					s.log.Errorw("failed to read websocket data", "error", err)
 					return err
 				}
 				s.log.Debugw("websocket connection encountered an error, attempting to reconnect...", "error", err)
-				// close the old connection and reconnect
-				if err := c.Close(); err != nil {
-					s.metrics.errorsTotal.Inc()
-					s.log.Errorw("encountered an error while closing the websocket connection", "error", err)
+				// close the old connection gracefully and reconnect
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: this is not a requirement, the code is self explanatory, will remove such sporadic comments
created_at: 2025-05-13T05:08:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,45 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: No particular hard bound reason, just helps in keeping the method signature short and more specific to the consequential parameters being passed while providing the `Initi of keepAlive`  in  `NewWebsocketFollower` and not having to pass it as an argument param always. Not a justifiable reason at this point but leaves room for easy extension in future if we need some more functionality/parameters.
created_at: 2025-05-13T05:12:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +65,45 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: Yeah, fair enough.
created_at: 2025-05-13T05:14:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +66,45 @@ func (l *loggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, erro
 	return resp, err
 }
 
+// heartBeat sends a ping message to the websocket server at regular intervals
+func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start time.Time) context.CancelFunc {
+	ctx, cancel := context.WithCancel(ctx)
+	// set initial read deadline
+	conn.SetReadDeadline(time.Now().Add(k.cfg.readControlDeadline))
+	// set pong handler to update read deadline
+	conn.SetPongHandler(func(string) error {
+		k.log.Debugw("received pong message from websocket server")
+		k.metrics.pongMessageReceivedTime.Update(time.Since(start).Nanoseconds())
+		return conn.SetReadDeadline(time.Now().Add(k.cfg.readControlDeadline))
+	})
+
+	// set heartbeat ping routine
+	go func() {
+		ticker := time.NewTicker(k.cfg.Interval)
+		defer ticker.Stop()
+
+		for {
+			select {
+			case <-ctx.Done():
+				k.log.Debugw("heartbeat stopped")
+				return
+			case now := <-ticker.C:
+				err := conn.WriteControl(websocket.PingMessage, nil, now.Add(k.cfg.WriteControlDeadline))
+				if err != nil {
+					k.log.Debugw("error sending ping control frame to websocket server:", err)
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: ```suggestion
					k.log.Debugw("error sending ping control frame to websocket server", "error", err)
```
created_at: 2025-05-13T07:36:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -207,6 +207,20 @@ filebeat.inputs:
   url: wss://localhost:443/_stream
 ```
 
+## Keep Alive Configuration
path: docs/reference/filebeat/filebeat-input-streaming.md
user: login: theletterf
body: ```suggestion
## Keep-Alive configuration
```
created_at: 2025-05-14T07:38:28Z
author_association: MEMBER
diff_hunk: @@ -207,6 +207,20 @@ filebeat.inputs:
   url: wss://localhost:443/_stream
 ```
 
+## Keep Alive Configuration
path: docs/reference/filebeat/filebeat-input-streaming.md
user: login: theletterf
body: We use sentence-style capitalization for headings.
created_at: 2025-05-14T07:39:07Z
author_association: MEMBER
diff_hunk: @@ -207,6 +207,20 @@ filebeat.inputs:
   url: wss://localhost:443/_stream
 ```
 
+## Keep Alive Configuration
+
+The `streaming` input currently supports keep-alive configuration options for streams of `type: websocket`. Using these configuration options we can further optimize the stability
+of our websocket connections and prevent them from idling out. 
path: docs/reference/filebeat/filebeat-input-streaming.md
user: login: theletterf
body: ```suggestion
The `streaming` input currently supports keep-alive configuration options for streams of `type: websocket`. Use these configuration options to further optimize the stability
of your WebSocket connections and prevent them from idling out. 
```
created_at: 2025-05-14T07:39:41Z
author_association: MEMBER
diff_hunk: @@ -207,6 +207,20 @@ filebeat.inputs:
   url: wss://localhost:443/_stream
 ```
 
+## Keep Alive Configuration
+
+The `streaming` input currently supports keep-alive configuration options for streams of `type: websocket`. Using these configuration options we can further optimize the stability
+of our websocket connections and prevent them from idling out. 
+
+The `keep_alive` setting has the following configuration options:
+
+* `enable`: This indicates whether keep-alive is enabled. By default, this is set to `false`.
+* `interval`: This is the interval between keep-alive messages. This is a time duration value and by default, this is set to `30s`.
+* `write_control_deadline`: This is the deadline for writing control frames (like `PING`, `PONG`, or `CLOSE`) on a websocket connection. This timeout helps prevent indefinite blocking when the server or client is not responding to control frame requests. This is a time duration value and by default, this is set to `10s`.
path: docs/reference/filebeat/filebeat-input-streaming.md
user: login: theletterf
body: ```suggestion
* `enable`: Indicates whether Keep-Alive is enabled. By default, this is set to `false`.
* `interval`: Interval between Keep-Alive messages, expressed as a time duration value. The default value is `30s`.
* `write_control_deadline`: Deadline for writing control frames, like `PING`, `PONG`, or `CLOSE`, on a WebSocket connection. The timeout, expressed as a time duration value, helps prevent indefinite blocking when the server or client is not responding to control frame requests. The default value is `10s`.
```
created_at: 2025-05-14T07:41:54Z
author_association: MEMBER
diff_hunk: @@ -207,6 +207,20 @@ filebeat.inputs:
   url: wss://localhost:443/_stream
 ```
 
+## Keep Alive Configuration
+
+The `streaming` input currently supports keep-alive configuration options for streams of `type: websocket`. Using these configuration options we can further optimize the stability
+of our websocket connections and prevent them from idling out. 
+
+The `keep_alive` setting has the following configuration options:
+
+* `enable`: This indicates whether keep-alive is enabled. By default, this is set to `false`.
+* `interval`: This is the interval between keep-alive messages. This is a time duration value and by default, this is set to `30s`.
+* `write_control_deadline`: This is the deadline for writing control frames (like `PING`, `PONG`, or `CLOSE`) on a websocket connection. This timeout helps prevent indefinite blocking when the server or client is not responding to control frame requests. This is a time duration value and by default, this is set to `10s`.
+   
+::::{note}
+It is recommended to not use `blanket_retries` and `infinite_retries` configuration options together with the `keep_alive` settings. The purpose of `keep_alive` is to keep the connection open so we do not need to `retry` and reconnect all the time. In some scenarios `keep_alive` might not work if the host websocket server is not configured to handle `ping` frames.
+::::
path: docs/reference/filebeat/filebeat-input-streaming.md
user: login: theletterf
body: ```suggestion
::::{note}
Don't use the `blanket_retries` and `infinite_retries` configuration options together with the `keep_alive` settings. The purpose of `keep_alive` is to keep the connection open so you don't need to `retry` and reconnect all the time. In some scenarios `keep_alive` might not work if the host WebSocket server is not configured to handle `ping` frames.
::::
```
created_at: 2025-05-14T07:46:42Z
author_association: MEMBER
diff_hunk: @@ -207,6 +207,20 @@ filebeat.inputs:
   url: wss://localhost:443/_stream
 ```
 
+## Keep Alive Configuration
+
+The `streaming` input currently supports keep-alive configuration options for streams of `type: websocket`. Using these configuration options we can further optimize the stability
+of our websocket connections and prevent them from idling out. 
+
+The `keep_alive` setting has the following configuration options:
+
+* `enable`: This indicates whether keep-alive is enabled. By default, this is set to `false`.
+* `interval`: This is the interval between keep-alive messages. This is a time duration value and by default, this is set to `30s`.
+* `write_control_deadline`: This is the deadline for writing control frames (like `PING`, `PONG`, or `CLOSE`) on a websocket connection. This timeout helps prevent indefinite blocking when the server or client is not responding to control frame requests. This is a time duration value and by default, this is set to `10s`.
path: docs/reference/filebeat/filebeat-input-streaming.md
user: login: theletterf
body: `WebSocket` and `Keep-Alive` must be capitalized because they're proper nouns for their respective techs.
created_at: 2025-05-14T07:57:36Z
author_association: MEMBER

PR comments:
user: login: mergify[bot]
created_at: 2025-05-05T12:04:29Z
body: This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:
* The upcoming major version of the Elastic Stack
* The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:
* `backport-8./d` is the label to automatically backport to the `8./d` branch. `/d` is the digit
* `backport-active-all` is the label that automatically backports to all active branches.
* `backport-active-8` is the label that automatically backports to all active minor branches for the 8 major.
* `backport-active-9` is the label that automatically backports to all active minor branches for the 9 major.
author_association: CONTRIBUTOR
user: login: elasticmachine
created_at: 2025-05-12T08:14:28Z
body: Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
author_association: CONTRIBUTOR
user: login: ShourieG
created_at: 2025-05-13T06:43:48Z
body: @efd6, addressed all the suggestions.
author_association: CONTRIBUTOR
user: login: ShourieG
created_at: 2025-05-13T07:46:09Z
body: /test
author_association: CONTRIBUTOR
user: login: ShourieG
created_at: 2025-05-13T11:11:58Z
body: @efd6, I've resolved all the comments and put up a screen recording showing manual testing for this new feature along with a server restart. Sorry for the quality, res is 720p because of GitHub upload size limits.
author_association: CONTRIBUTOR
user: login: ShourieG
created_at: 2025-05-13T15:05:29Z
body: /test
author_association: CONTRIBUTOR
user: login: efd6
created_at: 2025-05-13T21:37:32Z
body: Can you put the test WS server code up somewhere so that we can use it for testing in future?
author_association: CONTRIBUTOR
user: login: ShourieG
created_at: 2025-05-14T06:32:24Z
body: > Can you put the test WS server code up somewhere so that we can use it for testing in future?

@efd6, added the following [Gist](https://gist.github.com/ShourieG/322d38bdfc95cc7314e1729c3b892ae3) in the PR description
author_association: CONTRIBUTOR
user: login: ShourieG
created_at: 2025-05-14T09:45:49Z
body: @theletterf, resolved all the suggestions.
author_association: CONTRIBUTOR
user: login: ShourieG
created_at: 2025-06-30T09:22:54Z
body: @Mergifyio backport 8.19 9.0
author_association: CONTRIBUTOR
user: login: mergify[bot]
created_at: 2025-06-30T09:23:03Z
body: > backport 8.19 9.0

#### ✅ Backports have been created

<details>

* [#45110 [8.19](backport #44204) [filebeat][streaming] - Added support for websocket keep_alive heartbeat, updated metrics & docs](https://github.com/elastic/beats/pull/45110) has been created for branch `8.19` but encountered conflicts
* [#45111 [9.0](backport #44204) [filebeat][streaming] - Added support for websocket keep_alive heartbeat, updated metrics & docs](https://github.com/elastic/beats/pull/45111) has been created for branch `9.0` but encountered conflicts

</details>


<!---
DO NOT EDIT
-*- Mergify Payload -*-
{"command": "backport 8.19 9.0", "conclusion": "success", "action_is_running": true}
-*- Mergify Payload End -*-
-->
author_association: CONTRIBUTOR