Repository: influxdata/telegraf (Go)

Issue:
influxDb spamming retries
When influxDb is having connection problems (for example the host if the output doesnt exist, or network is down) and the metrics request with errors surpasses the metric_batch_size it creates a new batch that spams retries in the machine.

we got to a point where it was spaming more than 1 log per ms and surpassed the memory of our machine, shutting down the whole system 

<img width="1691" height="972" alt="Image" src="https://github.com/user-attachments/assets/288c2be5-3c4d-4405-b009-3ca1beac7ebe" />

_Originally posted by @andrev10 in 
https://github.com/influxdata/telegraf/issues/17246#issuecomment-3114508981_
            
[influxdb_v2] flush interval when connection interrupted with batch size exceeded
### Relevant telegraf.conf

```toml
[[outputs.influxdb_v2]]
  alias = "test influx"
  
  urls = ["http://localhost:8086"]
  token = "xxx"
  organization = "Development"
  bucket_tag = "bucket"
  exclude_bucket_tag  = true
  content_encoding = "gzip"
  
  flush_interval = "10s"
  metric_batch_size = 2
  metric_buffer_limit = 100000000
```

### Logs from Telegraf

```text
With completely disconnected server

2025-08-01T13:56:56Z E! [outputs.influxdb_v2::test influx] Post "http://10.0.0.2:8086/api/v2/write?bucket=test&org=Development": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2025-08-01T13:56:56Z E! [outputs.influxdb_v2::test influx] When writing to [http://10.0.0.2:8086/api/v2/write]: Post "http://10.0.0.2:8086/api/v2/write?bucket=test&org=Development": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2025-08-01T13:56:56Z D! [outputs.influxdb_v2::test influx] Buffer fullness: 10 metrics
2025-08-01T13:56:56Z E! [agent] Error writing to outputs.influxdb_v2::test influx: Post "http://10.0.0.2:8086/api/v2/write?bucket=test&org=Development": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

With connected server without influxdb running
2025-08-01T14:01:43Z E! [outputs.influxdb_v2::test influx] Post "http://localhost:8086/api/v2/write?bucket=test&org=Development": dial tcp [::1]:8086: connectex: No connection could be made because the target machine actively refused it.
2025-08-01T14:01:43Z E! [outputs.influxdb_v2::test influx] When writing to [http://localhost:8086/api/v2/write]: Post "http://localhost:8086/api/v2/write?bucket=test&org=Development": dial tcp [::1]:8086: connectex: No connection could be made because the target machine actively refused it.
2025-08-01T14:01:43Z D! [outputs.influxdb_v2::test influx] Buffer fullness: 10 metrics
2025-08-01T14:01:43Z E! [agent] Error writing to outputs.influxdb_v2::test influx: Post "http://localhost:8086/api/v2/write?bucket=test&org=Development": dial tcp [::1]:8086: connectex: No connection could be made because the target machine actively refused it.
```

### System info

 Telegraf 1.35.3 WinServer2022

### Docker

_No response_

### Steps to reproduce

1. Create a config that has a small output batch size
2. Overflow the batchsize
3. Direct output to an unreachable server/reachable server without influx running.
4. 
...

### Expected behavior

Only retry to flush the buffer on the flush intervals. If the connection is down just wait until the next interval.

Or make this configurable with `retries` and `retries_wait`, this way you can for example configure the output to retry 5 times with an interval of 1000ms and then try again the next flush interval.

### Actual behavior

When the database is completely disconnected the output tries to flush its full buffer at a rate of 2 times (could be coincidence) the flush interval.
When the database is not running but the server is reachable, the output retries to flush its buffer as fast as it can.

This only happens when the batchsize is exceeded. So it seems like it wants to flush instantly when the batchsize is exceeded.

### Additional info

I am testing out the capability of telegraf to store and forward metrics. That's why I exceed the batchsize when my database is down and noticed this behavior.

Or am I missing some existing configuration?

Seems like the same issue as #17378 


Discussion:
Same with outputs.opentelemetry

Log snippet:

```
Jul 30 16:40:11 tycho telegraf[1269]: 2025-07-30T20:40:11Z E! [agent] Error writing to outputs.opentelemetry: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.0.1.3:4317: connect: connection refused"
```

Count of just connection refused (16:40:11 - 16:44:58):
```
# journalctl -b0 -u telegraf.service | grep -c "connection refused"
150019
```

CPU at 100%, and overwhelming log volume once OTEL was available again

<img width="688" height="230" alt="Image" src="https://github.com/user-attachments/assets/dbb79e44-1f09-411c-8366-c2dd35333bba" />
I encountered the same issue when testing out the buffer strategy in the event of a database outtage.
Can confirmed that I encountered this recently. 
@andrev10, @chennin, @JeroenVanHoye and @jmickey please test the binary in PR #17566, available as soon as CI finished the tests, and let me know if this fixes the issue!
The fix works on my setup!
We discovered this issue in a multi-tenant cloud environment via a spike in data transfer costs. We're unable to easily test the PR binary, but are very appreciative for the attention this is receiving!

@JeroenVanHoye please test the binary in PR #17566, available as soon as CI finished the tests, and let me know if this fixes the issue!
Looks good to me. 
Had a config that generated 12 metrics at startup, had the batchsize set to 2 and it only tried to write once every flush interval (10s). After enabling influxdb the plugin waited until the next interval and flushed the buffer in 6 writes.
Also rechecked the behaviour of v1.35.3 with this config and watched the spamming go wild.

Thanks @srebhan 



Patch:
diff --git a/models/running_output.go b/models/running_output.go
index 38ca15f386c3b..12cf6769ccb6c 100644
--- a/models/running_output.go
+++ b/models/running_output.go
@@ -406,8 +406,10 @@ func (r *RunningOutput) updateTransaction(tx *Transaction, err error) {
 		return
 	}
 
-	// Transfer the accepted and rejected indices based on the write error values
-	r.lastWriteFailed.Store(false)
+	// Transfer the accepted and rejected indices based on the write error
+	// values. Only allow to retrigger before the flush interval if at least
+	// one metric was accepted in order to avoid
+	r.lastWriteFailed.Store(len(writeErr.MetricsAccept) == 0)
 	tx.Accept = writeErr.MetricsAccept
 	tx.Reject = writeErr.MetricsReject
 }


Test patch:
diff --git a/models/running_output_test.go b/models/running_output_test.go
index 9d5a78752f51b..a7ea56697b845 100644
--- a/models/running_output_test.go
+++ b/models/running_output_test.go
@@ -637,7 +637,7 @@ func TestRunningOutputNoRetriggerOnError(t *testing.T) {
 	require.Equal(t, totalMetrics, ro.buffer.Len())
 }
 
-func TestRunningOutputNoRetriggerOnPartialWriteError(t *testing.T) {
+func TestRunningOutputNoRetriggerOnSuccessfulPartialWriteError(t *testing.T) {
 	// Setup output with a post-write hook to be able to block write until
 	// we added more metrics
 	conf := &OutputConfig{
@@ -680,6 +680,7 @@ func TestRunningOutputNoRetriggerOnPartialWriteError(t *testing.T) {
 	ctx, cancel := context.WithCancel(t.Context())
 	defer cancel()
 
+	var triggerCount atomic.Uint32
 	var errCount atomic.Uint32
 	var wg sync.WaitGroup
 	wg.Add(1)
@@ -690,6 +691,7 @@ func TestRunningOutputNoRetriggerOnPartialWriteError(t *testing.T) {
 			case <-cctx.Done():
 				return
 			case <-ro.BatchReady:
+				triggerCount.Add(1)
 				if err := ro.Write(); err != nil {
 					errCount.Add(1)
 				}
@@ -706,9 +708,75 @@ func TestRunningOutputNoRetriggerOnPartialWriteError(t *testing.T) {
 	// Check for writing errors and make sure all metrics were written,
 	// including the ones added while writing took place
 	require.Equal(t, errCount.Load(), plugin.writes.Load())
+	require.Equal(t, triggerCount.Load(), plugin.writes.Load())
 	require.Equal(t, batchCount, int(plugin.writes.Load()))
 }
 
+func TestRunningOutputNoRetriggerOnUnsuccessfulPartialWriteError(t *testing.T) {
+	// Setup output with a post-write hook to be able to block write until
+	// we added more metrics
+	conf := &OutputConfig{
+		Filter: Filter{},
+	}
+
+	plugin := &mockOutput{
+		batchAcceptSize: 0,
+		preWriteHook: func([]telegraf.Metric) error {
+			return &internal.PartialWriteError{
+				Err: errors.New("unable to connect"),
+			}
+		},
+	}
+	const batchSize = 5
+	ro := NewRunningOutput(plugin, conf, batchSize, 100)
+
+	// Create a multiple of batch size many metrics beyond the batch size
+	const batchCount = 10
+	const totalMetrics = batchCount * batchSize
+	inputs := make([]telegraf.Metric, 0, totalMetrics)
+	for i := range totalMetrics {
+		inputs = append(inputs, testutil.TestMetric(i, "test"))
+	}
+
+	// Add the metrics
+	for _, m := range inputs {
+		ro.AddMetric(m)
+	}
+
+	// Setup a event based writing loop similar to what the agent code does.
+	// Remember the first write will block to allow us adding more metrics.
+	ctx, cancel := context.WithTimeout(t.Context(), time.Second)
+	defer cancel()
+
+	var triggerCount atomic.Uint32
+	var errCount atomic.Uint32
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func(cctx context.Context) {
+		defer wg.Done()
+		for {
+			select {
+			case <-cctx.Done():
+				return
+			case <-ro.BatchReady:
+				triggerCount.Add(1)
+				if err := ro.Write(); err != nil {
+					errCount.Add(1)
+				}
+			}
+		}
+	}(ctx)
+
+	// Wait for the trigger loop to exit. This should happen latest after the
+	// defined timeout.
+	wg.Wait()
+	cancel()
+
+	// Check for writing errors and make sure all metrics were written,
+	// including the ones added while writing took place
+	require.Equal(t, 1, int(triggerCount.Load()))
+}
+
 func TestRunningOutputInternalMetrics(t *testing.T) {
 	_ = NewRunningOutput(
 		&mockOutput{},