Repository: prometheus/prometheus (language: go)

Issue:
TestHandlerSendAllRemapPerAm is flaky
### What did you do?

TestHandlerSendAllRemapPerAm is flaky in CI

```
--- FAIL: TestHandlerSendAllRemapPerAm (0.00s)
    notifier_test.go:298: 
        	Error Trace:	/__w/prometheus/prometheus/notifier/notifier_test.go:292
        	            				/__w/prometheus/prometheus/notifier/notifier_test.go:298
        	Error:      	Received unexpected error:
        	            	length mismatch: [0[2bfa127][active] [11463f0][active] 1[a9e5532][active] [a65037f][active] 2[83e6977][active] [3bda65a][active] 3[c74be54][active] [61ab310][active] 4[1893cdb][active] [f5922dd][active] 5[bda1789][active] [3af7de5][active] 6[a717da7][active] [3c5ec44][active] 7[d139972][active] [5ce2f17][active] 8[fb73ff5][active] [ad4a6c7][active] 9[4fb397c][active] [1f4081a][active] 10[cc66b46][active] [ee7d0fd][active] 11[2d896fb][active] [38f6fd1][active] 12[ee7523e][active] [e3b8699][active] 13[4fa88f5][active] [30e2270][active] 14[aa20fca][active] [9103f8f][active] 15[627b672][active] [c630598][active] 16[cb30407][active] [54a21f8][active] 17[a4c9a12][active] [a3ed6ee][active] 18[0555f93][active] [6d3bac8][active] 19[0da8f6b][active] [0db5e6d][active] 20[de987e9][active] [6ba8dcc][active] 21[8ce5cd7][active] [6fd3743][active] 22[4bdd2f1][active] [a6ccabd][active] 23[8392973][active] [0baed4e][active] 24[300dbe6][active] [90a2250][active] 25[bc0428f][active] [1e84d22][active] 26[7ca27f3][active] [e36247f][active] 27[3175520][active] [1f49873][active] 28[e24699a][active] [8bad1d1][active] 29[ff7e16e][active] [1bbc4cc][active] 30[2d99dcc][active] [c58dd4d][active] 31[f6313ae][active] [d2e91db][active]] != [0[2bfa127][active] 1[a9e5532][active] 2[83e6977][active] 3[c74be54][active] 4[1893cdb][active] 5[bda1789][active] 6[a717da7][active] 7[d139972][active] 8[fb73ff5][active] 9[4fb397c][active] 10[cc66b46][active] 11[2d896fb][active] 12[ee7523e][active] 13[4fa88f5][active] 14[aa20fca][active] 15[627b672][active] 16[cb30407][active] 17[a4c9a12][active] 18[0555f93][active] 19[0da8f6b][active] 20[de987e9][active] 21[8ce5cd7][active] 22[4bdd2f1][active] 23[8392973][active] 24[300dbe6][active] 25[bc0428f][active] 26[7ca27f3][active] 27[3175520][active] 28[e24699a][active] 29[ff7e16e][active] 30[2d99dcc][active] 31[f6313ae][active]]
        	Test:       	TestHandlerSendAllRemapPerAm
FAIL
FAIL	github.com/prometheus/prometheus/notifier	2.241s
```

### What did you expect to see?

_No response_

### What did you see instead? Under which circumstances?

See https://github.com/prometheus/prometheus/actions/runs/8171014333/job/22338360519?pr=12532

### System information

_No response_

### Prometheus version

```text
6b490afed80ff98f2d471248eb288a80d971167e
```


### Prometheus configuration file

_No response_

### Alertmanager version

_No response_

### Alertmanager configuration file

_No response_

### Logs

_No response_


Discussion / hints:
Similar here: https://github.com/prometheus/prometheus/actions/runs/8186433482/job/22384813371
```
--- FAIL: TestHandlerSendAllRemapPerAm (0.01s)
    notifier_test.go:298: 
        	Error Trace:	/__w/prometheus/prometheus/notifier/notifier_test.go:292
        	            				/__w/prometheus/prometheus/notifier/notifier_test.go:298
        	Error:      	Received unexpected error:
        	            	length mismatch: [0[2bfa127][active] [11463f0][active] 1[a9e5532][active] [a65037f][active] 2[83e6977][active] [3bda65a][active] 3[c74be54][active] [61ab310][active] 4[1893cdb][active] [f5922dd][active] 5[bda1789][active] [3af7de5][active] 6[a717da7][active] [3c5ec44][active] 7[d139972][active] [5ce2f17][active] 8[fb73ff5][active] [ad4a6c7][active] 9[4fb397c][active] [1f4081a][active] 10[cc66b46][active] [ee7d0fd][active] 11[2d896fb][active] [38f6fd1][active] 12[ee7523e][active] [e3b8699][active] 13[4fa88f5][active] [30e2270][active] 14[aa20fca][active] [9103f8f][active] 15[627b672][active] [c630598][active] 16[cb30407][active] [54a21f8][active] 17[a4c9a12][active] [a3ed6ee][active] 18[0555f93][active] [6d3bac8][active] 19[0da8f6b][active] [0db5e6d][active] 20[de987e9][active] [6ba8dcc][active] 21[8ce5cd7][active] [6fd3743][active] 22[4bdd2f1][active] [a6ccabd][active] 23[8392973][active] [0baed4e][active] 24[300dbe6][active] [90a2250][active] 25[bc0428f][active] [1e84d22][active] 26[7ca27f3][active] [e36247f][active] 27[3175520][active] [1f49873][active] 28[e24699a][active] [8bad1d1][active] 29[ff7e16e][active] [1bbc4cc][active] 30[2d99dcc][active] [c58dd4d][active] 31[f6313ae][active] [d2e91db][active]] != [0[2bfa127][active] 1[a9e5532][active] 2[83e6977][active] 3[c74be54][active] 4[1893cdb][active] 5[bda1789][active] 6[a717da7][active] 7[d139972][active] 8[fb73ff5][active] 9[4fb397c][active] 10[cc66b46][active] 11[2d896fb][active] 12[ee7523e][active] 13[4fa88f5][active] 14[aa20fca][active] 15[627b672][active] 16[cb30407][active] 17[a4c9a12][active] 18[0555f93][active] 19[0da8f6b][active] 20[de987e9][active] 21[8ce5cd7][active] 22[4bdd2f1][active] 23[8392973][active] 24[300dbe6][active] 25[bc0428f][active] 26[7ca27f3][active] 27[3175520][active] 28[e24699a][active] 29[ff7e16e][active] 30[2d99dcc][active] 31[f6313ae][active]]
        	Test:       	TestHandlerSendAllRemapPerAm
FAIL
FAIL	github.com/prometheus/prometheus/notifier	2.427s
```
I think I found the bug https://github.com/prometheus/prometheus/pull/12551/files#r1516526822


Fix patch:
diff --git a/notifier/notifier.go b/notifier/notifier.go
index a6a99f41351..d1832402f78 100644
--- a/notifier/notifier.go
+++ b/notifier/notifier.go
@@ -481,14 +481,11 @@ func (n *Manager) sendAll(alerts ...*Alert) bool {
 
 		if len(ams.cfg.AlertRelabelConfigs) > 0 {
 			amAlerts = relabelAlerts(ams.cfg.AlertRelabelConfigs, labels.Labels{}, alerts)
-			// TODO(nabokihms): figure out the right way to cache marshalled alerts.
-			//   Now it works well only for happy cases.
-			v1Payload = nil
-			v2Payload = nil
-
 			if len(amAlerts) == 0 {
 				continue
 			}
+			// We can't use the cached values from previous iteration.
+			v1Payload, v2Payload = nil, nil
 		}
 
 		switch ams.cfg.APIVersion {
@@ -531,15 +528,20 @@ func (n *Manager) sendAll(alerts ...*Alert) bool {
 			}
 		}
 
+		if len(ams.cfg.AlertRelabelConfigs) > 0 {
+			// We can't use the cached values on the next iteration.
+			v1Payload, v2Payload = nil, nil
+		}
+
 		for _, am := range ams.ams {
 			wg.Add(1)
 
 			ctx, cancel := context.WithTimeout(n.ctx, time.Duration(ams.cfg.Timeout))
 			defer cancel()
 
-			go func(client *http.Client, url string) {
+			go func(ctx context.Context, client *http.Client, url string, payload []byte, count int) {
 				if err := n.sendOne(ctx, client, url, payload); err != nil {
-					level.Error(n.logger).Log("alertmanager", url, "count", len(amAlerts), "msg", "Error sending alert", "err", err)
+					level.Error(n.logger).Log("alertmanager", url, "count", count, "msg", "Error sending alert", "err", err)
 					n.metrics.errors.WithLabelValues(url).Inc()
 				} else {
 					numSuccess.Inc()
@@ -548,7 +550,7 @@ func (n *Manager) sendAll(alerts ...*Alert) bool {
 				n.metrics.sent.WithLabelValues(url).Add(float64(len(amAlerts)))
 
 				wg.Done()
-			}(ams.client, am.url().String())
+			}(ctx, ams.client, am.url().String(), payload, len(amAlerts))
 		}
 
 		ams.mtx.RUnlock()


Test patch: