Repository: influxdata/telegraf
Pull request #16646: feat(outputs.health): Add max time between metrics check
Author: login: tulku
Labels: name: feat
description: Improvement on an existing feature such as adding a new setting/mode to an existing plugin
name: ready for final review
description: This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
name: plugin/output
description: 1. Request for new output plugins 2. Issues/PRs that are related to out plugins
Created: 2025-03-17T09:34:32Z | Merged: 2025-04-14T16:03:22Z

## Summary
This adds a new check to the `health` output plugin that can detect moments when no metrics are flowing through `telegraf`. For example, when impacted by issues like https://github.com/influxdata/telegraf/issues/16070

This is disabled by default, and can be enabled using the new configuration options:

```
[[outputs.health]]
  service_address = "http://:8080"
  max_time_between_metrics = "1s"
  unhealthy_if_metrics_are_delayed = true
```

Besides the included unittests, this change was validated running `telegraf` with the following configuration:

```
[[inputs.internal]]
  interval = "20s"

[[outputs.health]]
  service_address = "http://:8080"
  max_time_between_metrics = "1s"
  unhealthy_if_metrics_are_delayed = true
```

And querying the health status using: 
```
while true; do curl -X GET localhost:8080; sleep 0.01; done
```

1. Telegraf starts healthy.
2. Once 1s passes after the first internal metrics are written, the status changes to unhealthy
3. After that, every 20s, it reports OK for 1s and then goes back to unhealthy.

This PR replaces the previous implementation attempt done in https://github.com/influxdata/telegraf/pull/16212
For more context, refer to [this slack thread](https://influxcommunity.slack.com/archives/CH99HUH8V/p1736782025589019?thread_ts=1731947234.950749&cid=CH99HUH8V)

## Checklist
<!-- Mandatory
Please confirm the following by replacing the space with an "x" between the []:
-->

- [x] No AI generated code was used in this PR

## Related issues
<!-- Mandatory
All PRs should resolve an issue, if one does not exist, please open one.
-->
resolves https://github.com/influxdata/telegraf/issues/16645



Diff:
diff --git a/plugins/outputs/health/README.md b/plugins/outputs/health/README.md
index ad0a5db917b37..284eec26675da 100644
--- a/plugins/outputs/health/README.md
+++ b/plugins/outputs/health/README.md
@@ -46,6 +46,12 @@ See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
   # tls_cert = "/etc/telegraf/cert.pem"
   # tls_key = "/etc/telegraf/key.pem"
 
+  ## Maximum expected time between metrics being written
+  ## Enforces an unhealthy state if there was no new metric seen for at least
+  ## the specified time. The check is disabled by default and only used if a
+  ## positive time is specified.
+  # max_time_between_metrics = "0s"
+
   ## NOTE: Due to the way TOML is parsed, tables must be at the END of the
   ## plugin definition, otherwise additional config options are read as part of
   ## the table
@@ -67,6 +73,19 @@ See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
   ##   field = "buffer_size"
 ```
 
+### Maximum time between metrics
+
+The health plugin can assert that metrics are being delivered to it at an
+expected rate when setting `max_time_between_metrics` to a positive number.
+The check measures the time between consecutive writes to the plugin and
+compares it to the defined `max_time_between_metrics`. When the time
+elapsed between writes is greater than the configured maximum time, the plugin
+will report an unhealthy status. As soon as metrics are written again to the
+plugin, the health status will reset to healthy.
+
+Note that the metric timestamps are not taken into account, rather the time they
+are written to the plugin.
+
 ### compares
 
 The `compares` check is used to assert basic mathematical relationships.  Use
diff --git a/plugins/outputs/health/health.go b/plugins/outputs/health/health.go
index e99da404452ee..216535bcad757 100644
--- a/plugins/outputs/health/health.go
+++ b/plugins/outputs/health/health.go
@@ -41,17 +41,20 @@ type Health struct {
 	BasicPassword  string          `toml:"basic_password"`
 	common_tls.ServerConfig
 
-	Compares []*Compares     `toml:"compares"`
-	Contains []*Contains     `toml:"contains"`
+	Compares              []*Compares     `toml:"compares"`
+	Contains              []*Contains     `toml:"contains"`
+	MaxTimeBetweenMetrics config.Duration `toml:"max_time_between_metrics"`
+
 	Log      telegraf.Logger `toml:"-"`
 	checkers []Checker
 
-	wg      sync.WaitGroup
-	server  *http.Server
-	origin  string
-	network string
-	address string
-	tlsConf *tls.Config
+	wg             sync.WaitGroup
+	server         *http.Server
+	origin         string
+	network        string
+	address        string
+	tlsConf        *tls.Config
+	lastMetricTime time.Time
 
 	mu      sync.Mutex
 	healthy bool
@@ -117,7 +120,9 @@ func (h *Health) Connect() error {
 	h.origin = h.getOrigin(listener)
 
 	h.Log.Infof("Listening on %s", h.origin)
-
+	// Initialize lastMetricTime here to fail if no metrics are received
+	// before the configured max timeout.
+	h.lastMetricTime = time.Now()
 	h.wg.Add(1)
 	go func() {
 		defer h.wg.Done()
@@ -143,7 +148,13 @@ func (h *Health) listen() (net.Listener, error) {
 
 func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 	var code = http.StatusOK
-	if !h.isHealthy() {
+
+	healthy := h.isHealthy()
+	if h.MaxTimeBetweenMetrics > 0 {
+		healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)
+	}
+
+	if !healthy {
 		code = http.StatusServiceUnavailable
 	}
 
@@ -153,6 +164,7 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.lastMetricTime = time.Now()
 	healthy := true
 	for _, checker := range h.checkers {
 		success := checker.Check(metrics)
@@ -160,7 +172,9 @@ func (h *Health) Write(metrics []telegraf.Metric) error {
 			healthy = false
 		}
 	}
-
+	// healthy only represents the result of the configured checkers and not
+	// the MaxTimeBetweenMetrics validation. The timeout check is done when
+	// serving the HTTP response.
 	h.setHealthy(healthy)
 	return nil
 }
diff --git a/plugins/outputs/health/health_test.go b/plugins/outputs/health/health_test.go
index e155a6cba0929..e070e5f539961 100644
--- a/plugins/outputs/health/health_test.go
+++ b/plugins/outputs/health/health_test.go
@@ -9,6 +9,7 @@ import (
 	"github.com/stretchr/testify/require"
 
 	"github.com/influxdata/telegraf"
+	"github.com/influxdata/telegraf/config"
 	"github.com/influxdata/telegraf/plugins/outputs/health"
 	"github.com/influxdata/telegraf/testutil"
 )
@@ -214,3 +215,118 @@ func TestInitServiceAddress(t *testing.T) {
 		})
 	}
 }
+
+func TestTimeBetweenMetrics(t *testing.T) {
+	arbitraryTime := time.Time{}.AddDate(2002, 0, 0)
+	tests := []struct {
+		name                  string
+		maxTimeBetweenMetrics config.Duration
+		metrics               []telegraf.Metric
+		delay                 time.Duration
+		expectedCode          int
+	}{
+		{
+			name:                  "healthy enabled no metrics before timeout",
+			maxTimeBetweenMetrics: config.Duration(1 * time.Second),
+			metrics:               nil,
+			delay:                 0 * time.Second,
+			expectedCode:          200,
+		},
+		{
+			name:                  "unhealthy enabled no metrics after timeout",
+			maxTimeBetweenMetrics: config.Duration(5 * time.Millisecond),
+			metrics:               nil,
+			delay:                 5 * time.Millisecond,
+			expectedCode:          503,
+		},
+		{
+			name:                  "healthy when disabled and old metric",
+			maxTimeBetweenMetrics: config.Duration(0),
+			metrics: []telegraf.Metric{
+				testutil.MustMetric(
+					"cpu",
+					map[string]string{},
+					map[string]any{
+						"time_idle": 42,
+					},
+					arbitraryTime),
+				testutil.MustMetric(
+					"cpu",
+					map[string]string{},
+					map[string]any{
+						"time_idle": 64,
+					},
+					arbitraryTime),
+			},
+			delay:        10 * time.Millisecond,
+			expectedCode: 200,
+		},
+		{
+			name:                  "healthy when enabled and recent metric",
+			maxTimeBetweenMetrics: config.Duration(5 * time.Second),
+			metrics: []telegraf.Metric{
+				testutil.MustMetric(
+					"cpu",
+					map[string]string{},
+					map[string]any{
+						"time_idle": 42,
+					},
+					arbitraryTime),
+			},
+			delay:        0 * time.Second,
+			expectedCode: 200,
+		},
+		{
+			name:                  "unhealthy when enabled and old metric",
+			maxTimeBetweenMetrics: config.Duration(5 * time.Millisecond),
+			metrics: []telegraf.Metric{
+				testutil.MustMetric(
+					"cpu",
+					map[string]string{},
+					map[string]any{
+						"time_idle": 42,
+					},
+					arbitraryTime),
+				testutil.MustMetric(
+					"cpu",
+					map[string]string{},
+					map[string]any{
+						"time_idle": 64,
+					},
+					arbitraryTime),
+			},
+			delay:        10 * time.Millisecond,
+			expectedCode: 503,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			dut := health.NewHealth()
+			dut.ServiceAddress = "tcp://127.0.0.1:0"
+			dut.Log = testutil.Logger{}
+			dut.MaxTimeBetweenMetrics = tt.maxTimeBetweenMetrics
+
+			err := dut.Init()
+			require.NoError(t, err)
+
+			err = dut.Connect()
+			require.NoError(t, err)
+
+			err = dut.Write(tt.metrics)
+			require.NoError(t, err)
+
+			time.Sleep(tt.delay)
+			resp, err := http.Get(dut.Origin())
+			require.NoError(t, err)
+			defer resp.Body.Close()
+			require.Equal(t, tt.expectedCode, resp.StatusCode)
+
+			_, err = io.ReadAll(resp.Body)
+			require.NoError(t, err)
+
+			err = dut.Close()
+			require.NoError(t, err)
+		})
+	}
+}
diff --git a/plugins/outputs/health/sample.conf b/plugins/outputs/health/sample.conf
index 07d5a4b3d35e4..3e7207242d56f 100644
--- a/plugins/outputs/health/sample.conf
+++ b/plugins/outputs/health/sample.conf
@@ -21,6 +21,12 @@
   # tls_cert = "/etc/telegraf/cert.pem"
   # tls_key = "/etc/telegraf/key.pem"
 
+  ## Maximum expected time between metrics being written
+  ## Enforces an unhealthy state if there was no new metric seen for at least
+  ## the specified time. The check is disabled by default and only used if a
+  ## positive time is specified.
+  # max_time_between_metrics = "0s"
+
   ## NOTE: Due to the way TOML is parsed, tables must be at the END of the
   ## plugin definition, otherwise additional config options are read as part of
   ## the table


Reviews:
user: login: srebhan
body: Thanks @tulku for your contribution! I think the idea is nice. Some suggestions for simplification in the code. The tests need to be adapted accordingly... 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T15:32:27Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:16:37Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:57:33Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:57:41Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:58:14Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:59:08Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:59:14Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:59:55Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T17:00:45Z
user: login: srebhan
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-09T14:08:20Z
user: login: srebhan
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-09T14:11:46Z
user: login: srebhan
body: Thanks @tulku! One more comment... Please also check my responses to the unresolved previous comments...
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-09T14:14:04Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T14:43:27Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T08:58:27Z
user: login: srebhan
body: Thanks @tulku! Please set the initial `h.lastMetricTime` in `Connect()` to avoid cases where no metric arrives at all and the plugin is in healthy state.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-10T13:36:52Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T14:27:26Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T14:27:40Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T14:27:54Z
user: login: tulku
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T14:28:00Z
user: login: srebhan
body: Thanks a lot @tulku!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-14T12:06:00Z
user: login: skartikey
body: @tulku Thanks for your contribution!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T16:03:08Z

Review comments:
diff_hunk: @@ -46,6 +46,12 @@ See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
   # tls_cert = "/etc/telegraf/cert.pem"
   # tls_key = "/etc/telegraf/key.pem"
 
+  ## Enables the max_time_between_metrics. Disabled by default
+  # unhealthy_if_metrics_are_delayed = false
+  ## Sets the maximum expected time between metrics being written
+  ## to the plugin. Ignored when unhealthy_if_metrics_are_delayed is false.
+  # max_time_between_metrics = "1s"
path: plugins/outputs/health/README.md
user: login: srebhan
body: How about
```suggestion
  ## Maximum expected time between metrics being written
  ## Enforces an unhealthy state if there was no new metric seen for at least the
  ## specified time. The check is disabled by default and only used if a positive
  ## time is specified.
  # max_time_between_metrics = "0s"
```
created_at: 2025-03-20T15:26:19Z
author_association: MEMBER
diff_hunk: @@ -46,12 +46,17 @@ type Health struct {
 	Log      telegraf.Logger `toml:"-"`
 	checkers []Checker
 
-	wg      sync.WaitGroup
-	server  *http.Server
-	origin  string
-	network string
-	address string
-	tlsConf *tls.Config
+	UnhealthyIfMetricsAreDelayed bool            `toml:"unhealthy_if_metrics_are_delayed"`
+	MaxTimeBetweenMetrics        config.Duration `toml:"max_time_between_metrics"`
path: plugins/outputs/health/health.go
user: login: srebhan
body: With the change in the `sample.conf` you can do
```suggestion
	MaxTimeBetweenMetrics        config.Duration `toml:"max_time_between_metrics"`
```
Please move this up to before the `Log` member...
created_at: 2025-03-20T15:27:35Z
author_association: MEMBER
diff_hunk: @@ -143,7 +148,13 @@ func (h *Health) listen() (net.Listener, error) {
 
 func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 	var code = http.StatusOK
-	if !h.isHealthy() {
+
+	healthy := h.isHealthy()
+	if h.UnhealthyIfMetricsAreDelayed && h.firstMetricReceived {
+		healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)
+	}
path: plugins/outputs/health/health.go
user: login: srebhan
body: ```suggestion
	if h.MaxTimeBetweenMetrics > 0 {
		healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)
	}
```
created_at: 2025-03-20T15:28:22Z
author_association: MEMBER
diff_hunk: @@ -153,14 +164,15 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.firstMetricReceived = true
+	h.lastMetricTime = time.Now()
path: plugins/outputs/health/health.go
user: login: srebhan
body: How about initializing the `lastMetricTime` in `Connect`?
created_at: 2025-03-20T15:29:23Z
author_association: MEMBER
diff_hunk: @@ -153,14 +164,15 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.firstMetricReceived = true
+	h.lastMetricTime = time.Now()
 	healthy := true
 	for _, checker := range h.checkers {
 		success := checker.Check(metrics)
 		if !success {
 			healthy = false
 		}
 	}
-
path: plugins/outputs/health/health.go
user: login: srebhan
body: Please restore the newline.
created_at: 2025-03-20T15:29:37Z
author_association: MEMBER
diff_hunk: @@ -219,10 +231,12 @@ func (h *Health) isHealthy() bool {
 
 func NewHealth() *Health {
 	return &Health{
-		ServiceAddress: defaultServiceAddress,
-		ReadTimeout:    config.Duration(defaultReadTimeout),
-		WriteTimeout:   config.Duration(defaultWriteTimeout),
-		healthy:        true,
+		ServiceAddress:               defaultServiceAddress,
+		ReadTimeout:                  config.Duration(defaultReadTimeout),
+		WriteTimeout:                 config.Duration(defaultWriteTimeout),
+		UnhealthyIfMetricsAreDelayed: false,
+		healthy:                      true,
+		firstMetricReceived:          false,
path: plugins/outputs/health/health.go
user: login: srebhan
body: This change is unnecessary if we constrain the check to a positive value (i.e. > 0) for the time check.
created_at: 2025-03-20T15:30:35Z
author_association: MEMBER
diff_hunk: @@ -21,6 +21,12 @@
   # tls_cert = "/etc/telegraf/cert.pem"
   # tls_key = "/etc/telegraf/key.pem"
 
+  ## Enables the max_time_between_metrics. Disabled by default
+  # unhealthy_if_metrics_are_delayed = false
+  ## Sets the maximum expected time between metrics being written
+  ## to the plugin. Ignored when unhealthy_if_metrics_are_delayed is false.
+  # max_time_between_metrics = "1s"
+
path: plugins/outputs/health/sample.conf
user: login: srebhan
body: How about
```suggestion
  ## Maximum expected time between metrics being written
  ## Enforces an unhealthy state if there was no new metric seen for at least the
  ## specified time. The check is disabled by default and only used if a positive
  ## time is specified.
  # max_time_between_metrics = "0s"
```
created_at: 2025-03-20T15:31:22Z
author_association: MEMBER
diff_hunk: @@ -153,14 +164,15 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.firstMetricReceived = true
+	h.lastMetricTime = time.Now()
path: plugins/outputs/health/health.go
user: login: tulku
body: This is because `Connect` is guaranteed to be called before the first `Write`, correct?
created_at: 2025-03-27T16:16:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -46,6 +46,12 @@ See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
   # tls_cert = "/etc/telegraf/cert.pem"
   # tls_key = "/etc/telegraf/key.pem"
 
+  ## Enables the max_time_between_metrics. Disabled by default
+  # unhealthy_if_metrics_are_delayed = false
+  ## Sets the maximum expected time between metrics being written
+  ## to the plugin. Ignored when unhealthy_if_metrics_are_delayed is false.
+  # max_time_between_metrics = "1s"
path: plugins/outputs/health/README.md
user: login: tulku
body: Updated
created_at: 2025-03-27T16:57:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -46,12 +46,17 @@ type Health struct {
 	Log      telegraf.Logger `toml:"-"`
 	checkers []Checker
 
-	wg      sync.WaitGroup
-	server  *http.Server
-	origin  string
-	network string
-	address string
-	tlsConf *tls.Config
+	UnhealthyIfMetricsAreDelayed bool            `toml:"unhealthy_if_metrics_are_delayed"`
+	MaxTimeBetweenMetrics        config.Duration `toml:"max_time_between_metrics"`
path: plugins/outputs/health/health.go
user: login: tulku
body: Moved
created_at: 2025-03-27T16:57:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -143,7 +148,13 @@ func (h *Health) listen() (net.Listener, error) {
 
 func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 	var code = http.StatusOK
-	if !h.isHealthy() {
+
+	healthy := h.isHealthy()
+	if h.UnhealthyIfMetricsAreDelayed && h.firstMetricReceived {
+		healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)
+	}
path: plugins/outputs/health/health.go
user: login: tulku
body: Here I added a check for the `lastMetricTime` being non zero also, so the status is healthy before the first metrics are written.
created_at: 2025-03-27T16:58:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -153,14 +164,15 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.firstMetricReceived = true
+	h.lastMetricTime = time.Now()
path: plugins/outputs/health/health.go
user: login: tulku
body: I left the initialization in `NewHealth` (which is called from `init`). If `Connect` is preferred, I can move it there.
created_at: 2025-03-27T16:59:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -153,14 +164,15 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.firstMetricReceived = true
+	h.lastMetricTime = time.Now()
 	healthy := true
 	for _, checker := range h.checkers {
 		success := checker.Check(metrics)
 		if !success {
 			healthy = false
 		}
 	}
-
path: plugins/outputs/health/health.go
user: login: tulku
body: restored
created_at: 2025-03-27T16:59:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -219,10 +231,12 @@ func (h *Health) isHealthy() bool {
 
 func NewHealth() *Health {
 	return &Health{
-		ServiceAddress: defaultServiceAddress,
-		ReadTimeout:    config.Duration(defaultReadTimeout),
-		WriteTimeout:   config.Duration(defaultWriteTimeout),
-		healthy:        true,
+		ServiceAddress:               defaultServiceAddress,
+		ReadTimeout:                  config.Duration(defaultReadTimeout),
+		WriteTimeout:                 config.Duration(defaultWriteTimeout),
+		UnhealthyIfMetricsAreDelayed: false,
+		healthy:                      true,
+		firstMetricReceived:          false,
path: plugins/outputs/health/health.go
user: login: tulku
body: removed both boolean flags `UnhealthyIfMetricsAreDelayed` and `firstMetricReceived`. Used only the max delay and the last metric time.
created_at: 2025-03-27T16:59:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,6 +21,12 @@
   # tls_cert = "/etc/telegraf/cert.pem"
   # tls_key = "/etc/telegraf/key.pem"
 
+  ## Enables the max_time_between_metrics. Disabled by default
+  # unhealthy_if_metrics_are_delayed = false
+  ## Sets the maximum expected time between metrics being written
+  ## to the plugin. Ignored when unhealthy_if_metrics_are_delayed is false.
+  # max_time_between_metrics = "1s"
+
path: plugins/outputs/health/sample.conf
user: login: tulku
body: changed to the proposed text.
created_at: 2025-03-27T17:00:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -143,7 +148,13 @@ func (h *Health) listen() (net.Listener, error) {
 
 func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 	var code = http.StatusOK
-	if !h.isHealthy() {
+
+	healthy := h.isHealthy()
+	if h.UnhealthyIfMetricsAreDelayed && h.firstMetricReceived {
+		healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)
+	}
path: plugins/outputs/health/health.go
user: login: srebhan
body: Hmmm shouldn't we rather set the `h.lastMetricTime` to `now` in `Connect` as otherwise the output stays healthy if no metric arrives at all...
created_at: 2025-04-09T14:08:20Z
author_association: MEMBER
diff_hunk: @@ -153,14 +164,15 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.firstMetricReceived = true
+	h.lastMetricTime = time.Now()
path: plugins/outputs/health/health.go
user: login: srebhan
body: Yes, `Connect` would be better. The `init` function is called when importing this plugin so you might get timeouts if setting up Telegraf takes long. `Connect` is called _after_ `Init` but before the first `Write` and is the better place I think.
created_at: 2025-04-09T14:11:46Z
author_association: MEMBER
diff_hunk: @@ -219,10 +229,12 @@ func (h *Health) isHealthy() bool {
 
 func NewHealth() *Health {
 	return &Health{
-		ServiceAddress: defaultServiceAddress,
-		ReadTimeout:    config.Duration(defaultReadTimeout),
-		WriteTimeout:   config.Duration(defaultWriteTimeout),
-		healthy:        true,
+		ServiceAddress:        defaultServiceAddress,
+		ReadTimeout:           config.Duration(defaultReadTimeout),
+		WriteTimeout:          config.Duration(defaultWriteTimeout),
+		MaxTimeBetweenMetrics: 0,
+		healthy:               true,
+		lastMetricTime:        time.Time{},
path: plugins/outputs/health/health.go
user: login: srebhan
body: You don't need to initialize the two fields as Golang always uses the `nil` value when creating new instances. So go with
```suggestion
		healthy:               true,
```
created_at: 2025-04-09T14:13:17Z
author_association: MEMBER
diff_hunk: @@ -143,7 +148,13 @@ func (h *Health) listen() (net.Listener, error) {
 
 func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 	var code = http.StatusOK
-	if !h.isHealthy() {
+
+	healthy := h.isHealthy()
+	if h.UnhealthyIfMetricsAreDelayed && h.firstMetricReceived {
+		healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)
+	}
path: plugins/outputs/health/health.go
user: login: tulku
body: I want the output to be healthy if no metric is received. This is because I want to avoid the case where telegraf is restarted (because of being unhealthy) on startup.
created_at: 2025-04-09T14:43:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -153,14 +164,15 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.firstMetricReceived = true
+	h.lastMetricTime = time.Now()
path: plugins/outputs/health/health.go
user: login: tulku
body: With the latest change, `lastMetricTime` is initialized as 0 and it is only updated in `Write`.  Whe can add an initialization to `now` on `Connect`. This will help catch cases where telegraf starts but no metric is written before the max time elapses.
The current behaviour waits for the first metric to be written to start checking if the time between metrics is longer than expected. This was by design, to prevent the health output to fail during startup. However, if `Connect` is called 'close enough' to when everything should already be working, we can set `lastMetricTime = now` in connect also.
created_at: 2025-04-10T08:58:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -143,7 +146,13 @@ func (h *Health) listen() (net.Listener, error) {
 
 func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 	var code = http.StatusOK
-	if !h.isHealthy() {
+
+	healthy := h.isHealthy()
+	if h.MaxTimeBetweenMetrics > 0 && !h.lastMetricTime.IsZero() {
path: plugins/outputs/health/health.go
user: login: srebhan
body: Please do
```suggestion
	if h.MaxTimeBetweenMetrics > 0  {
```

and set 

```golang

	h.lastMetricTime = time.Now()
```

in `Connect()` around line 123.
created_at: 2025-04-10T13:33:52Z
author_association: MEMBER
diff_hunk: @@ -153,6 +162,7 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.lastMetricTime = time.Now()
path: plugins/outputs/health/health.go
user: login: srebhan
body: Would be nice to add a comment below (line 174) stating why we don't need to check the metric timeout before calling `h.setHealthy()`...
created_at: 2025-04-10T13:35:50Z
author_association: MEMBER
diff_hunk: @@ -143,7 +148,13 @@ func (h *Health) listen() (net.Listener, error) {
 
 func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 	var code = http.StatusOK
-	if !h.isHealthy() {
+
+	healthy := h.isHealthy()
+	if h.UnhealthyIfMetricsAreDelayed && h.firstMetricReceived {
+		healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)
+	}
path: plugins/outputs/health/health.go
user: login: tulku
body: with the last change, `lastMetricTime` is set to `now` in `Connect`.
created_at: 2025-04-10T14:27:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -153,14 +164,15 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.firstMetricReceived = true
+	h.lastMetricTime = time.Now()
path: plugins/outputs/health/health.go
user: login: tulku
body: with the last change, `lastMetricTime` is set to `now` in `Connect`.
created_at: 2025-04-10T14:27:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -143,7 +146,13 @@ func (h *Health) listen() (net.Listener, error) {
 
 func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 	var code = http.StatusOK
-	if !h.isHealthy() {
+
+	healthy := h.isHealthy()
+	if h.MaxTimeBetweenMetrics > 0 && !h.lastMetricTime.IsZero() {
path: plugins/outputs/health/health.go
user: login: tulku
body: added
created_at: 2025-04-10T14:27:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -153,6 +162,7 @@ func (h *Health) ServeHTTP(rw http.ResponseWriter, _ *http.Request) {
 
 // Write runs all checks over the metric batch and adjust health state.
 func (h *Health) Write(metrics []telegraf.Metric) error {
+	h.lastMetricTime = time.Now()
path: plugins/outputs/health/health.go
user: login: tulku
body: added
created_at: 2025-04-10T14:27:59Z
author_association: CONTRIBUTOR

PR comments:
user: login: tulku
created_at: 2025-03-27T17:02:41Z
body: Thanks for the review @srebhan ! The code is now simpler, removing the two boolean flags. I took the liberty to initialize `lastMetricTime` in `NewHealth` instead of `Connect`. Let me know if this is a problem.
author_association: CONTRIBUTOR
user: login: srebhan
created_at: 2025-04-09T14:14:59Z
body: @tulku please also run `make lint` on your machine and fix the linter issues...
author_association: MEMBER
user: login: telegraf-tiger[bot]
created_at: 2025-04-10T15:09:37Z
body: Download PR build artifacts for [linux_amd64.tar.gz](https://output.circle-artifacts.com/output/job/123a5bbf-196f-448a-827d-251991894508/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_amd64.tar.gz), [darwin_arm64.tar.gz](https://output.circle-artifacts.com/output/job/ca330a32-8f19-4336-beb5-bcef5fba55c2/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_darwin_arm64.tar.gz), and [windows_amd64.zip](https://output.circle-artifacts.com/output/job/d87a09e1-21ff-4d5b-beaf-140e525d49a2/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_windows_amd64.zip).
Downloads for additional architectures and packages are available below.

:relaxed: This pull request doesn't significantly change the Telegraf binary size (less than 1%) 

<details>
<summary> :package: Click here to get additional PR build artifacts</summary>

## Artifact URLs

|                                                                               DEB                                                                                |                                                                               RPM                                                                                |                                                                                      TAR GZ                                                                                      |                                                                                    ZIP                                                                                     |
|------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [amd64.deb](https://output.circle-artifacts.com/output/job/123a5bbf-196f-448a-827d-251991894508/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_amd64.deb)     | [aarch64.rpm](https://output.circle-artifacts.com/output/job/649bce98-b065-46cb-8bf4-fe3d369104c2/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48-0.aarch64.rpm) | [darwin_amd64.tar.gz](https://output.circle-artifacts.com/output/job/c89648b7-c96e-48b9-ac87-aa7556a936a2/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_darwin_amd64.tar.gz)   | [windows_amd64.zip](https://output.circle-artifacts.com/output/job/d87a09e1-21ff-4d5b-beaf-140e525d49a2/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_windows_amd64.zip) |
| [arm64.deb](https://output.circle-artifacts.com/output/job/649bce98-b065-46cb-8bf4-fe3d369104c2/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_arm64.deb)     | [armel.rpm](https://output.circle-artifacts.com/output/job/9a8259ac-0ad6-4c10-bda1-faeee2e9e804/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48-0.armel.rpm)     | [darwin_arm64.tar.gz](https://output.circle-artifacts.com/output/job/ca330a32-8f19-4336-beb5-bcef5fba55c2/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_darwin_arm64.tar.gz)   | [windows_arm64.zip](https://output.circle-artifacts.com/output/job/d87a09e1-21ff-4d5b-beaf-140e525d49a2/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_windows_arm64.zip) |
| [armel.deb](https://output.circle-artifacts.com/output/job/9a8259ac-0ad6-4c10-bda1-faeee2e9e804/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_armel.deb)     | [armv6hl.rpm](https://output.circle-artifacts.com/output/job/9d68da30-31f4-4589-8f40-3bd138f3c1a7/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48-0.armv6hl.rpm) | [freebsd_amd64.tar.gz](https://output.circle-artifacts.com/output/job/123a5bbf-196f-448a-827d-251991894508/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_freebsd_amd64.tar.gz) | [windows_i386.zip](https://output.circle-artifacts.com/output/job/d87a09e1-21ff-4d5b-beaf-140e525d49a2/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_windows_i386.zip)   |
| [armhf.deb](https://output.circle-artifacts.com/output/job/9d68da30-31f4-4589-8f40-3bd138f3c1a7/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_armhf.deb)     | [i386.rpm](https://output.circle-artifacts.com/output/job/5a5d7d38-40ee-49c4-9b29-6978ef076fe4/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48-0.i386.rpm)       | [freebsd_armv7.tar.gz](https://output.circle-artifacts.com/output/job/9d68da30-31f4-4589-8f40-3bd138f3c1a7/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_freebsd_armv7.tar.gz) |                                                                                                                                                                            |
| [i386.deb](https://output.circle-artifacts.com/output/job/5a5d7d38-40ee-49c4-9b29-6978ef076fe4/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_i386.deb)       | [ppc64le.rpm](https://output.circle-artifacts.com/output/job/082924f1-75ca-4d35-8d16-ef975b8f6f6b/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48-0.ppc64le.rpm) | [freebsd_i386.tar.gz](https://output.circle-artifacts.com/output/job/5a5d7d38-40ee-49c4-9b29-6978ef076fe4/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_freebsd_i386.tar.gz)   |                                                                                                                                                                            |
| [mips.deb](https://output.circle-artifacts.com/output/job/92793ec6-b870-49fd-934d-593f3e5b437f/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_mips.deb)       | [riscv64.rpm](https://output.circle-artifacts.com/output/job/50f2d79e-29b2-47c1-bd55-77f25a1e5ea7/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48-0.riscv64.rpm) | [linux_amd64.tar.gz](https://output.circle-artifacts.com/output/job/123a5bbf-196f-448a-827d-251991894508/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_amd64.tar.gz)     |                                                                                                                                                                            |
| [mipsel.deb](https://output.circle-artifacts.com/output/job/0c6df7ad-e740-461e-a59d-34ea9182dfa3/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_mipsel.deb)   | [s390x.rpm](https://output.circle-artifacts.com/output/job/b1fd9c91-66f8-4971-b34c-ee35c4ca429d/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48-0.s390x.rpm)     | [linux_arm64.tar.gz](https://output.circle-artifacts.com/output/job/649bce98-b065-46cb-8bf4-fe3d369104c2/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_arm64.tar.gz)     |                                                                                                                                                                            |
| [ppc64el.deb](https://output.circle-artifacts.com/output/job/082924f1-75ca-4d35-8d16-ef975b8f6f6b/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_ppc64el.deb) | [x86_64.rpm](https://output.circle-artifacts.com/output/job/123a5bbf-196f-448a-827d-251991894508/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48-0.x86_64.rpm)   | [linux_armel.tar.gz](https://output.circle-artifacts.com/output/job/9a8259ac-0ad6-4c10-bda1-faeee2e9e804/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_armel.tar.gz)     |                                                                                                                                                                            |
| [riscv64.deb](https://output.circle-artifacts.com/output/job/50f2d79e-29b2-47c1-bd55-77f25a1e5ea7/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_riscv64.deb) |                                                                                                                                                                  | [linux_armhf.tar.gz](https://output.circle-artifacts.com/output/job/9d68da30-31f4-4589-8f40-3bd138f3c1a7/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_armhf.tar.gz)     |                                                                                                                                                                            |
| [s390x.deb](https://output.circle-artifacts.com/output/job/b1fd9c91-66f8-4971-b34c-ee35c4ca429d/artifacts/0/build/dist/telegraf_1.35.0~8d25ea48-0_s390x.deb)     |                                                                                                                                                                  | [linux_i386.tar.gz](https://output.circle-artifacts.com/output/job/5a5d7d38-40ee-49c4-9b29-6978ef076fe4/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_i386.tar.gz)       |                                                                                                                                                                            |
|                                                                                                                                                                  |                                                                                                                                                                  | [linux_mips.tar.gz](https://output.circle-artifacts.com/output/job/92793ec6-b870-49fd-934d-593f3e5b437f/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_mips.tar.gz)       |                                                                                                                                                                            |
|                                                                                                                                                                  |                                                                                                                                                                  | [linux_mipsel.tar.gz](https://output.circle-artifacts.com/output/job/0c6df7ad-e740-461e-a59d-34ea9182dfa3/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_mipsel.tar.gz)   |                                                                                                                                                                            |
|                                                                                                                                                                  |                                                                                                                                                                  | [linux_ppc64le.tar.gz](https://output.circle-artifacts.com/output/job/082924f1-75ca-4d35-8d16-ef975b8f6f6b/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_ppc64le.tar.gz) |                                                                                                                                                                            |
|                                                                                                                                                                  |                                                                                                                                                                  | [linux_riscv64.tar.gz](https://output.circle-artifacts.com/output/job/50f2d79e-29b2-47c1-bd55-77f25a1e5ea7/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_riscv64.tar.gz) |                                                                                                                                                                            |
|                                                                                                                                                                  |                                                                                                                                                                  | [linux_s390x.tar.gz](https://output.circle-artifacts.com/output/job/b1fd9c91-66f8-4971-b34c-ee35c4ca429d/artifacts/0/build/dist/telegraf-1.35.0~8d25ea48_linux_s390x.tar.gz)     |                                                                                                                                                                            |
</details>
author_association: CONTRIBUTOR