Repository: elastic/beats
Pull request #44340: x-pack/filebeat/input/streaming: add fleet status update functionality
Author: login: efd6
Labels: name: enhancement
name: Filebeat
description: Filebeat
name: Team:Security-Service Integrations
description: Security Service Integrations Team
name: backport-9.0
description: Automated backport to the 9.0 branch
name: backport-8.19
description: Automated backport to the 8.19 branch
Created: 2025-05-15T04:11:06Z | Merged: 2025-05-27T06:42:49Z

<!-- Type of change
Please label this PR with one of the following labels, depending on the scope of your change:
- Bug
- Enhancement
- Breaking change
- Deprecation
- Cleanup
- Docs
-->

## Proposed commit message

See title.

<!-- Mandatory
Explain here the changes you made on the PR.

Please explain:

- WHAT: patterns used, algorithms implemented, design architecture, message processing, etc.
- WHY:  the rationale/motivation for the changes

This text will be pasted into the squash dialog when the change is committed and will be
a long term historical record of the change to help future contributors understand the
change, please help them by making it clear and comprehensive, they may be you.

If the commit title is adequate to describe both of these things, The text here may be omitted
or replaced with "See title". The title of the PR will be used as the commit message title when
the merge is made and the "See title" marker will be removed if present.

The text here and the PR title will be subject to the PR review process.
-->

## Checklist

<!-- Mandatory
Add a checklist of things that are required to be reviewed in order to have the PR approved

List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. ~~Scratch this.~~)
-->

- [ ] My code follows the style guidelines of this project
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] I have made corresponding change to the default configuration files
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added an entry in `CHANGELOG.next.asciidoc` or `CHANGELOG-developer.next.asciidoc`.

## Disruptive User Impact

<!--
Will the changes introduced by this PR cause disruption to users in any way? If so, please describe what changes users
could make on their end to nullify or minimize this disruption. Consider impacts in related systems, not just directly
when using Beats.
-->

## Author's Checklist

<!-- Recommended
Add a checklist of things that are required to be reviewed in order to have the PR approved
-->
- [ ]

## How to test this PR locally

<!-- Recommended
Explain here how this PR will be tested by the reviewer: commands, dependencies, steps, etc.
-->

## Related issues

<!-- Recommended
Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it.

- Closes #123
- Relates #123
- Requires #123
- Superseds #123
-->
- Closes #44284

## Use cases

<!-- Recommended
Explain here the different behaviors that this PR introduces or modifies in this project, user roles, environment configuration, etc.

If you are familiar with Gherkin test scenarios, we recommend its usage: https://cucumber.io/docs/gherkin/reference/
-->

## Screenshots

<!-- Optional
Add here screenshots about how the project will be changed after the PR is applied. They could be related to web pages, terminal, etc, or any other image you consider important to be shared with the team.
-->

## Logs

<!-- Recommended
Paste here output logs discovered while creating this PR, such as stack traces or integration logs, or any other output you consider important to be shared with the team.
-->


Linked issue #123: need help to debug why Packetbeat not capturing postgresql
Can anyone help me how to debug why packetbeat not capturing postgresql packets. 

I tried running packetbeat -e -d "pgsql" and it keep on giving below logs
"pgsql.go:472: WARN Postgresql Message too short (length=1). Wait for more.
pgsql.go:630: DBG  Message not complete, but sending to the next layer
pgsql.go:723: WARN Response from unknown transaction. Ignoring."

Somewhere before I read that this is because of preparedstatements, but even commands which I ran in psql are not begin captured by packetbeat. Is there any way to debug what is the issue?

Thanks

Linked issue comments: user: login: harshamadala
created_at: 2015-04-27T18:57:53Z
body: Configuration using:
device = "any"
[protocols.pgsql]
  ports = [5432]
[procs.monitored.pgsql]
  cmdline_grep = "postgres"

author_association: NONE
user: login: tsg
created_at: 2015-04-27T19:05:12Z
body: You might have encryption on? See https://github.com/packetbeat/packetbeat/issues/90

author_association: CONTRIBUTOR
user: login: tsg
created_at: 2015-04-27T19:06:00Z
body: Sorry, this is the ticket I wanted to link to: https://github.com/packetbeat/packetbeat/issues/46

author_association: CONTRIBUTOR
user: login: harshamadala
created_at: 2015-04-28T09:02:20Z
body: Thanks for quick response.

We have encryption off. And I am getting 
"pgsql.go:723: WARN Response from unknown transaction. Ignoring." continuously. 
Is this because of prepared statements? How to debug this?

author_association: NONE
user: login: tsg
created_at: 2015-04-28T09:05:02Z
body: Best would be to create a small trace, like this one:

```
tcpdump -i any -s0 -w"trace.pcap" "port 5432"
```

And send it to us at info@packetbeat.com. We can than figure it out if it's because of prepared statements or something else. 

author_association: CONTRIBUTOR
user: login: tsg
created_at: 2015-04-28T10:56:19Z
body: Thanks for the trace, indeed, it's using "unnamed prepared statements". You can see them showing up in wireshark as "T/D/C/Z".

Closing this one as a duplicate of #78. We'll try to figure out a solution.

author_association: CONTRIBUTOR

Diff:
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
index 07d5ae1ebe45..8478a77bc48e 100644
--- a/CHANGELOG.next.asciidoc
+++ b/CHANGELOG.next.asciidoc
@@ -433,6 +433,7 @@ otherwise no tag is added. {issue}42208[42208] {pull}42403[42403]
 - Add support for OPTIONS request to HTTP Endpoint input. {issue}43930[43930] {pull}44387[44387]
 - Add Fleet status update functionality to lumberjack input. {issue}44283[44283] {pull}44339[44339]
 - Add Fleet status updating to HTTP Endpoint input. {issue}44281[44281] {pull}44310[44310]
+- Add Fleet status updating to streaming input. {issue}44284[44284] {pull}44340[44340]
 
 *Auditbeat*
 
diff --git a/x-pack/filebeat/input/streaming/crowdstrike.go b/x-pack/filebeat/input/streaming/crowdstrike.go
index 5d3520490b28..eaf157729006 100644
--- a/x-pack/filebeat/input/streaming/crowdstrike.go
+++ b/x-pack/filebeat/input/streaming/crowdstrike.go
@@ -18,6 +18,7 @@ import (
 	"golang.org/x/oauth2/clientcredentials"
 
 	inputcursor "github.com/elastic/beats/v7/filebeat/input/v2/input-cursor"
+	"github.com/elastic/beats/v7/libbeat/management/status"
 	"github.com/elastic/elastic-agent-libs/logp"
 	"github.com/elastic/elastic-agent-libs/mapstr"
 	"github.com/elastic/elastic-agent-libs/transport/httpcommon"
@@ -30,6 +31,8 @@ type falconHoseStream struct {
 	cfg    config
 	cursor map[string]any
 
+	status status.StatusReporter
+
 	creds       *clientcredentials.Config
 	discoverURL string
 	plainClient *http.Client
@@ -40,11 +43,16 @@ type falconHoseStream struct {
 // NewFalconHoseFollower performs environment construction including CEL
 // program and regexp compilation, and input metrics set-up for a Crowdstrike
 // FalconHose stream follower.
-func NewFalconHoseFollower(ctx context.Context, id string, cfg config, cursor map[string]any, pub inputcursor.Publisher, log *logp.Logger, now func() time.Time) (StreamFollower, error) {
+func NewFalconHoseFollower(ctx context.Context, id string, cfg config, cursor map[string]any, pub inputcursor.Publisher, stat status.StatusReporter, log *logp.Logger, now func() time.Time) (StreamFollower, error) {
+	if stat == nil {
+		stat = noopReporter{}
+	}
+	stat.UpdateStatus(status.Configuring, "")
 	s := falconHoseStream{
 		id:     id,
 		cfg:    cfg,
 		cursor: cursor,
+		status: stat,
 		processor: processor{
 			ns:      "falcon_hose",
 			pub:     pub,
@@ -66,6 +74,7 @@ func NewFalconHoseFollower(ctx context.Context, id string, cfg config, cursor ma
 	patterns, err := regexpsFromConfig(cfg)
 	if err != nil {
 		s.metrics.errorsTotal.Inc()
+		stat.UpdateStatus(status.Failed, "invalid regular expression: "+err.Error())
 		s.Close()
 		return nil, err
 	}
@@ -73,13 +82,16 @@ func NewFalconHoseFollower(ctx context.Context, id string, cfg config, cursor ma
 	s.prg, s.ast, err = newProgram(ctx, cfg.Program, root, patterns, log)
 	if err != nil {
 		s.metrics.errorsTotal.Inc()
+		stat.UpdateStatus(status.Failed, err.Error())
 		s.Close()
 		return nil, err
 	}
 
 	u, err := url.Parse(s.cfg.URL.String())
 	if err != nil {
-		return nil, fmt.Errorf("failed parse url: %w", err)
+		err = fmt.Errorf("failed to parse url: %w", err)
+		stat.UpdateStatus(status.Failed, err.Error())
+		return nil, err
 	}
 	query := url.Values{"appId": []string{cfg.CrowdstrikeAppID}}
 	u.RawQuery = query.Encode()
@@ -87,6 +99,7 @@ func NewFalconHoseFollower(ctx context.Context, id string, cfg config, cursor ma
 
 	s.plainClient, err = cfg.Transport.Client(httpcommon.WithAPMHTTPInstrumentation())
 	if err != nil {
+		stat.UpdateStatus(status.Failed, "failed to configure client: "+err.Error())
 		return nil, err
 	}
 
@@ -115,25 +128,34 @@ func (s *falconHoseStream) FollowStream(ctx context.Context) error {
 		if err != nil {
 			if !errors.Is(err, Warning{}) {
 				if errors.Is(err, context.Canceled) {
+					s.status.UpdateStatus(status.Stopping, "")
 					return nil
 				}
 				s.metrics.errorsTotal.Inc()
+				// Status for failures is handled within followSession.
 				return err
 			}
 			s.metrics.errorsTotal.Inc()
+			s.status.UpdateStatus(status.Degraded, err.Error())
 			s.log.Warnw("session warning", "error", err)
+			continue
 		}
+		s.status.UpdateStatus(status.Running, "")
 	}
 }
 
 func (s *falconHoseStream) followSession(ctx context.Context, cli *http.Client, state map[string]any) (map[string]any, error) {
 	req, err := http.NewRequestWithContext(ctx, http.MethodGet, s.discoverURL, nil)
 	if err != nil {
-		return nil, fmt.Errorf("failed to prepare discover stream request: %w", err)
+		err = fmt.Errorf("failed to prepare discover stream request: %w", err)
+		s.status.UpdateStatus(status.Degraded, err.Error())
+		return nil, err
 	}
 	resp, err := cli.Do(req)
 	if err != nil {
-		return nil, fmt.Errorf("failed GET to discover stream: %w", err)
+		err = fmt.Errorf("failed GET to discover stream: %w", err)
+		s.status.UpdateStatus(status.Degraded, err.Error())
+		return nil, err
 	}
 	defer resp.Body.Close()
 
@@ -181,6 +203,7 @@ func (s *falconHoseStream) followSession(ctx context.Context, cli *http.Client,
 					req, err := http.NewRequestWithContext(ctx, http.MethodPost, r.RefreshURL, nil)
 					if err != nil {
 						s.metrics.errorsTotal.Inc()
+						s.status.UpdateStatus(status.Failed, "failed to prepare refresh stream request: "+err.Error())
 						s.log.Errorw("failed to prepare refresh stream request", "error", err)
 						return
 					}
@@ -188,12 +211,14 @@ func (s *falconHoseStream) followSession(ctx context.Context, cli *http.Client,
 					resp, err := cli.Do(req)
 					if err != nil {
 						s.metrics.errorsTotal.Inc()
+						s.status.UpdateStatus(status.Failed, "failed to refresh stream connection: "+err.Error())
 						s.log.Errorw("failed to refresh stream connection", "error", err)
 						return
 					}
 					err = resp.Body.Close()
 					if err != nil {
 						s.metrics.errorsTotal.Inc()
+						s.status.UpdateStatus(status.Failed, "failed to close refresh response body: "+err.Error())
 						s.log.Warnw("failed to close refresh response body", "error", err)
 					}
 				}
@@ -248,6 +273,7 @@ func (s *falconHoseStream) followSession(ctx context.Context, cli *http.Client,
 			err = s.process(ctx, state, s.cursor, s.now().In(time.UTC))
 			if err != nil {
 				s.log.Errorw("failed to process and publish data", "error", err)
+				s.status.UpdateStatus(status.Failed, "failed to process and publish data: "+err.Error())
 				return nil, err
 			}
 		}
diff --git a/x-pack/filebeat/input/streaming/crowdstrike_test.go b/x-pack/filebeat/input/streaming/crowdstrike_test.go
index 775cb4a88cd8..b9a064ed0760 100644
--- a/x-pack/filebeat/input/streaming/crowdstrike_test.go
+++ b/x-pack/filebeat/input/streaming/crowdstrike_test.go
@@ -83,7 +83,7 @@ func TestCrowdstrikeFalconHose(t *testing.T) {
 	if *offset >= 0 {
 		cursor = map[string]any{"offset": *offset}
 	}
-	s, err := NewFalconHoseFollower(ctx, "crowdstrike_testing", cfg, cursor, &testPublisher{logger}, logger, time.Now)
+	s, err := NewFalconHoseFollower(ctx, "crowdstrike_testing", cfg, cursor, &testPublisher{logger}, nil, logger, time.Now)
 	if err != nil {
 		t.Fatalf("unexpected error constructing follower: %v", err)
 	}
diff --git a/x-pack/filebeat/input/streaming/input.go b/x-pack/filebeat/input/streaming/input.go
index 9e53f9c6956a..4a66ea6effaf 100644
--- a/x-pack/filebeat/input/streaming/input.go
+++ b/x-pack/filebeat/input/streaming/input.go
@@ -20,6 +20,7 @@ import (
 	inputcursor "github.com/elastic/beats/v7/filebeat/input/v2/input-cursor"
 	"github.com/elastic/beats/v7/libbeat/beat"
 	"github.com/elastic/beats/v7/libbeat/feature"
+	"github.com/elastic/beats/v7/libbeat/management/status"
 	"github.com/elastic/beats/v7/libbeat/statestore"
 	"github.com/elastic/elastic-agent-libs/logp"
 	"github.com/elastic/go-concert/ctxtool"
@@ -74,10 +75,13 @@ func (input) Test(src inputcursor.Source, _ v2.TestContext) error {
 // Run starts the input and blocks as long as websocket connections are alive. It will return on
 // context cancellation or type invalidity errors, any other error will be retried.
 func (input) Run(env v2.Context, src inputcursor.Source, crsr inputcursor.Cursor, pub inputcursor.Publisher) error {
+	env.UpdateStatus(status.Starting, "")
 	var cursor map[string]interface{}
 	if !crsr.IsNew() { // Allow the user to bootstrap the program if needed.
+		env.UpdateStatus(status.Configuring, "")
 		err := crsr.Unpack(&cursor)
 		if err != nil {
+			env.UpdateStatus(status.Failed, "failed to unpack cursor: "+err.Error())
 			return err
 		}
 	}
@@ -97,18 +101,24 @@ func (i input) run(env v2.Context, src *source, cursor map[string]any, pub input
 	// want to be a registry. Until then, let's keep this simple.
 	switch cfg.Type {
 	case "", "websocket":
-		s, err = NewWebsocketFollower(ctx, env.ID, cfg, cursor, pub, log, i.time)
+		s, err = NewWebsocketFollower(ctx, env.ID, cfg, cursor, pub, env.StatusReporter, log, i.time)
 	case "crowdstrike":
-		s, err = NewFalconHoseFollower(ctx, env.ID, cfg, cursor, pub, log, i.time)
+		s, err = NewFalconHoseFollower(ctx, env.ID, cfg, cursor, pub, env.StatusReporter, log, i.time)
 	}
 	if err != nil {
 		return err
 	}
 	defer s.Close()
 
-	return s.FollowStream(ctx)
+	err = s.FollowStream(ctx)
+	env.UpdateStatus(status.Stopped, "")
+	return err
 }
 
+type noopReporter struct{}
+
+func (noopReporter) UpdateStatus(status.Status, string) {}
+
 // getURL initializes the input URL with the help of the url_program.
 func getURL(ctx context.Context, name, src, url string, state map[string]any, redaction *redact, log *logp.Logger, now func() time.Time) (string, error) {
 	if src == "" {
diff --git a/x-pack/filebeat/input/streaming/websocket.go b/x-pack/filebeat/input/streaming/websocket.go
index 94471fca6ae7..6a04659302d0 100644
--- a/x-pack/filebeat/input/streaming/websocket.go
+++ b/x-pack/filebeat/input/streaming/websocket.go
@@ -25,6 +25,7 @@ import (
 	"golang.org/x/oauth2/clientcredentials"
 
 	inputcursor "github.com/elastic/beats/v7/filebeat/input/v2/input-cursor"
+	"github.com/elastic/beats/v7/libbeat/management/status"
 	"github.com/elastic/elastic-agent-libs/logp"
 	"github.com/elastic/elastic-agent-libs/transport/httpcommon"
 	"github.com/elastic/elastic-agent-libs/transport/tlscommon"
@@ -43,6 +44,7 @@ type websocketStream struct {
 	id          string
 	cfg         config
 	cursor      map[string]any
+	status      status.StatusReporter
 	tokenSource oauth2.TokenSource
 	tokenExpiry <-chan time.Time
 	time        func() time.Time
@@ -115,11 +117,16 @@ func (k *keepAlive) heartBeat(ctx context.Context, conn *websocket.Conn, start t
 // NewWebsocketFollower performs environment construction including CEL
 // program and regexp compilation, and input metrics set-up for a websocket
 // stream follower.
-func NewWebsocketFollower(ctx context.Context, id string, cfg config, cursor map[string]any, pub inputcursor.Publisher, log *logp.Logger, now func() time.Time) (StreamFollower, error) {
+func NewWebsocketFollower(ctx context.Context, id string, cfg config, cursor map[string]any, pub inputcursor.Publisher, stat status.StatusReporter, log *logp.Logger, now func() time.Time) (StreamFollower, error) {
+	if stat == nil {
+		stat = noopReporter{}
+	}
+	stat.UpdateStatus(status.Configuring, "")
 	s := websocketStream{
 		id:     id,
 		cfg:    cfg,
 		cursor: cursor,
+		status: stat,
 		processor: processor{
 			ns:      "websocket",
 			pub:     pub,
@@ -153,7 +160,9 @@ func NewWebsocketFollower(ctx context.Context, id string, cfg config, cursor map
 		if err != nil {
 			s.metrics.errorsTotal.Inc()
 			s.Close()
-			return nil, fmt.Errorf("failed to obtain oauth2 token: %w", err)
+			err = fmt.Errorf("failed to obtain oauth2 token: %w", err)
+			stat.UpdateStatus(status.Failed, err.Error())
+			return nil, err
 		}
 		// set the initial token in the config if oauth2 is enabled
 		// this allows seamless header creation in formHeader() for the initial connection
@@ -176,6 +185,7 @@ func NewWebsocketFollower(ctx context.Context, id string, cfg config, cursor map
 	patterns, err := regexpsFromConfig(cfg)
 	if err != nil {
 		s.metrics.errorsTotal.Inc()
+		stat.UpdateStatus(status.Failed, "invalid regular expression: "+err.Error())
 		s.Close()
 		return nil, err
 	}
@@ -183,6 +193,7 @@ func NewWebsocketFollower(ctx context.Context, id string, cfg config, cursor map
 	s.prg, s.ast, err = newProgram(ctx, cfg.Program, root, patterns, log)
 	if err != nil {
 		s.metrics.errorsTotal.Inc()
+		stat.UpdateStatus(status.Failed, err.Error())
 		s.Close()
 		return nil, err
 	}
@@ -207,15 +218,17 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 	url, err := getURL(ctx, "websocket", s.cfg.URLProgram, s.cfg.URL.String(), state, s.cfg.Redact, s.log, s.now)
 	if err != nil {
 		s.metrics.errorsTotal.Inc()
+		s.status.UpdateStatus(status.Failed, "failed to get url: "+err.Error())
 		return err
 	}
 
 	// websocket client
-	c, resp, err := connectWebSocket(ctx, s.cfg, url, s.log)
+	c, resp, err := connectWebSocket(ctx, s.cfg, url, s.status, s.log)
 	handleConnectionResponse(resp, s.metrics, s.log)
 	if err != nil {
 		s.metrics.errorsTotal.Inc()
 		s.log.Errorw("failed to establish websocket connection", "error", err)
+		s.status.UpdateStatus(status.Failed, "failed to establish websocket connection: "+err.Error())
 		return err
 	}
 	// Start the keep-alive routine if enabled and the connection is established successfully
@@ -241,6 +254,7 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 		// the parent context is done().
 		case <-ctx.Done():
 			s.log.Debugw("context cancelled, closing websocket connection")
+			s.status.UpdateStatus(status.Stopping, "")
 			return ctx.Err()
 		// s.tokenExpiry channel will only trigger if oauth2 is enabled and the token is about to expire
 		case <-s.tokenExpiry:
@@ -254,6 +268,7 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 			if err != nil {
 				s.metrics.errorsTotal.Inc()
 				s.log.Errorw("failed to obtain oauth2 token during token refresh", "error", err)
+				s.status.UpdateStatus(status.Failed, "failed to obtain oauth2 token during token refresh: "+err.Error())
 				return err
 			}
 			// gracefully close current connection
@@ -268,10 +283,11 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 			// set the new token expiry channel with 2 mins buffer
 			s.tokenExpiry = time.After(time.Until(token.Expiry) - s.cfg.Auth.OAuth2.TokenExpiryBuffer)
 			// establish a new connection with the new token
-			c, resp, err = connectWebSocket(ctx, s.cfg, url, s.log)
+			c, resp, err = connectWebSocket(ctx, s.cfg, url, s.status, s.log)
 			handleConnectionResponse(resp, s.metrics, s.log)
 			if err != nil {
 				s.metrics.errorsTotal.Inc()
+				s.status.UpdateStatus(status.Failed, "failed to establish a new websocket connection on token refresh: "+err.Error())
 				s.log.Errorw("failed to establish a new websocket connection on token refresh", "error", err)
 				return err
 			}
@@ -290,10 +306,12 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 					heartBeatCancel()
 				}
 				if !s.cfg.Retry.BlanketRetries && !isRetryableError(err) {
+					s.status.UpdateStatus(status.Failed, "failed to read websocket data: "+err.Error())
 					s.log.Errorw("failed to read websocket data", "error", err)
 					return err
 				}
 				s.log.Debugw("websocket connection encountered an error, attempting to reconnect...", "error", err)
+				s.status.UpdateStatus(status.Degraded, "websocket connection encountered an error: "+err.Error())
 				if c != nil {
 					if err := c.Close(); err != nil {
 						s.metrics.errorsTotal.Inc()
@@ -302,10 +320,11 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 				}
 				// Since c is already a pointer, we can reassign it to the new connection
 				// and the defer func will still handle it.
-				c, resp, err = connectWebSocket(ctx, s.cfg, url, s.log)
+				c, resp, err = connectWebSocket(ctx, s.cfg, url, s.status, s.log)
 				handleConnectionResponse(resp, s.metrics, s.log)
 				if err != nil {
 					s.metrics.errorsTotal.Inc()
+					s.status.UpdateStatus(status.Failed, "failed to reconnect websocket connection: "+err.Error())
 					s.log.Errorw("failed to reconnect websocket connection", "error", err)
 					return err
 				}
@@ -323,10 +342,12 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 			err = s.process(ctx, state, s.cursor, s.now().In(time.UTC))
 			if err != nil {
 				s.metrics.errorsTotal.Inc()
+				s.status.UpdateStatus(status.Failed, "failed to process and publish data: "+err.Error())
 				s.log.Errorw("failed to process and publish data", "error", err)
 				return err
 			}
 		}
+		s.status.UpdateStatus(status.Running, "")
 	}
 }
 
@@ -401,7 +422,7 @@ func handleConnectionResponse(resp *http.Response, metrics *inputMetrics, log *l
 }
 
 // connectWebSocket attempts to connect to the websocket server with exponential backoff if retry config is available else it connects without retry.
-func connectWebSocket(ctx context.Context, cfg config, url string, log *logp.Logger) (*websocket.Conn, *http.Response, error) {
+func connectWebSocket(ctx context.Context, cfg config, url string, stat status.StatusReporter, log *logp.Logger) (*websocket.Conn, *http.Response, error) {
 	var conn *websocket.Conn
 	var response *http.Response
 	var err error
@@ -416,8 +437,10 @@ func connectWebSocket(ctx context.Context, cfg config, url string, log *logp.Log
 			for attempt := 1; attempt <= retryConfig.MaxAttempts; attempt++ {
 				conn, response, err = dialer.DialContext(ctx, url, headers)
 				if err == nil {
+					stat.UpdateStatus(status.Running, "")
 					return conn, response, nil
 				}
+				stat.UpdateStatus(status.Degraded, "attempting to reconnect websocket")
 				// in case of sudden network errors or server crashes the response will be nil and logging should be done gracefully
 				if response != nil {
 					//nolint:errorlint // it will never be a wrapped error at this point
@@ -440,8 +463,10 @@ func connectWebSocket(ctx context.Context, cfg config, url string, log *logp.Log
 			for attempt := 1; ; attempt++ {
 				conn, response, err = dialer.DialContext(ctx, url, headers)
 				if err == nil {
+					stat.UpdateStatus(status.Running, "")
 					return conn, response, nil
 				}
+				stat.UpdateStatus(status.Degraded, "attempting to reconnect websocket")
 				// in case of sudden network errors or server crashes the response will be nil and logging should be done gracefully
 				if response != nil {
 					//nolint:errorlint // it will never be a wrapped error at this point


Reviews:
user: login: ShourieG
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-26T11:00:22Z
user: login: ShourieG
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-26T11:49:57Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-26T21:26:28Z
user: login: efd6
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-26T21:29:38Z

Review comments:
diff_hunk: @@ -283,10 +299,12 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 					heartBeatCancel()
 				}
 				if !s.cfg.Retry.BlanketRetries && !isRetryableError(err) {
+					s.status.UpdateStatus(status.Failed, "failed to read websocket data: "+err.Error())
 					s.log.Errorw("failed to read websocket data", "error", err)
 					return err
 				}
 				s.log.Debugw("websocket connection encountered an error, attempting to reconnect...", "error", err)
+				s.status.UpdateStatus(status.Degraded, "websocket connection encountered an error: "+err.Error())
path: x-pack/filebeat/input/streaming/websocket.go
user: login: ShourieG
body: So one question here, should we not wait for a certain number of failures to occur before we mark the status as Degraded ? The current philosophy seems to take a heavy handed approach as in 1st error -> update status as "degraded" -> if it reconnects then update status to "running".
created_at: 2025-05-26T11:49:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -283,10 +299,12 @@ func (s *websocketStream) FollowStream(ctx context.Context) error {
 					heartBeatCancel()
 				}
 				if !s.cfg.Retry.BlanketRetries && !isRetryableError(err) {
+					s.status.UpdateStatus(status.Failed, "failed to read websocket data: "+err.Error())
 					s.log.Errorw("failed to read websocket data", "error", err)
 					return err
 				}
 				s.log.Debugw("websocket connection encountered an error, attempting to reconnect...", "error", err)
+				s.status.UpdateStatus(status.Degraded, "websocket connection encountered an error: "+err.Error())
path: x-pack/filebeat/input/streaming/websocket.go
user: login: efd6
body: I think I'm OK with the current approach. We can revisit this if it is something that leads to UI/log churn. If we have a certain number other than one (a very certain number :smile:), we need to decide what it is, or allow a user configuration.
created_at: 2025-05-26T21:26:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -430,6 +430,7 @@ otherwise no tag is added. {issue}42208[42208] {pull}42403[42403]
 - Add support to the Active Directory entity analytics provider for device entities. {pull}44309[44309]
 - Add support for OPTIONS request to HTTP Endpoint input. {issue}43930[43930] {pull}44387[44387]
 
path: CHANGELOG.next.asciidoc
user: login: efd6
body: ```suggestion
```
created_at: 2025-05-26T21:29:38Z
author_association: CONTRIBUTOR

PR comments:
user: login: elasticmachine
created_at: 2025-05-15T08:52:45Z
body: Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
author_association: CONTRIBUTOR
user: login: mergify[bot]
created_at: 2025-05-20T19:28:02Z
body: This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/
```
git fetch upstream
git checkout -b 44284-streaming upstream/44284-streaming
git merge upstream/main
git push upstream 44284-streaming
```
author_association: CONTRIBUTOR
user: login: mergify[bot]
created_at: 2025-05-21T06:43:27Z
body: This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/
```
git fetch upstream
git checkout -b 44284-streaming upstream/44284-streaming
git merge upstream/main
git push upstream 44284-streaming
```
author_association: CONTRIBUTOR
user: login: mergify[bot]
created_at: 2025-05-22T21:02:15Z
body: This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/
```
git fetch upstream
git checkout -b 44284-streaming upstream/44284-streaming
git merge upstream/main
git push upstream 44284-streaming
```
author_association: CONTRIBUTOR
user: login: ShourieG
created_at: 2025-05-26T11:09:31Z
body: Just one query should the asciidoc update for this be ideally one in `CHANGELOG-developer.next.asciidoc` ?
author_association: CONTRIBUTOR
user: login: efd6
created_at: 2025-05-26T21:29:30Z
body: There is an entry in CHANGELOG.next.asciidoc. Why do we need one in developer as well?
author_association: CONTRIBUTOR
user: login: mergify[bot]
created_at: 2025-05-26T23:41:39Z
body: This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/
```
git fetch upstream
git checkout -b 44284-streaming upstream/44284-streaming
git merge upstream/main
git push upstream 44284-streaming
```
author_association: CONTRIBUTOR
user: login: ShourieG
created_at: 2025-05-27T02:26:36Z
body: > There is an entry in CHANGELOG.next.asciidoc. Why do we need one in developer as well?

No it's fine, initially I felt this was more of an internal change and only needed to be inside the Dev asciidoc. But the end user will see these status effects so all good.
author_association: CONTRIBUTOR
user: login: mergify[bot]
created_at: 2025-05-27T04:31:05Z
body: This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/
```
git fetch upstream
git checkout -b 44284-streaming upstream/44284-streaming
git merge upstream/main
git push upstream 44284-streaming
```
author_association: CONTRIBUTOR