Repository: prometheus/prometheus (language: go)

Issue:
RW - wrong status code for storage.ErrTooOldSample (out of order time window feature)
### What did you do?

The remote write [spec](https://prometheus.io/docs/concepts/remote_write_spec/#retries-backoff) specifies that a RW sender must perform retries when encountering a 5xx status code

When using the experimental out of order window feature, and attempting to ingest a sample older than the window,
I receive `500 Internal Server Error: too old sample`

Assuming there's no server mis-configuration or time sync issues, this error is truly non-recoverable and thus it makes no sense to retry it. Such scenario is likely when there can be significant lag between the RW sender and the receiver, e.g, when using a message queue.

The following lines (and perhaps others) seem like they should be adjusted to handle `storage.ErrTooOldSample` as well

https://github.com/prometheus/prometheus/blob/6b8e9453881bffe5fe3aec99a6f3462676a95489/storage/remote/write_handler.go#L71C1-L74C9


### What did you expect to see?

_No response_

### What did you see instead? Under which circumstances?

Sending RW requests (from a go service) with old samples, beyond the out of order window, yields a status code of 500 (should probably be 400) 
Prometheus 2.45.1

### System information

_No response_

### Prometheus version

_No response_

### Prometheus configuration file

_No response_

### Alertmanager version

_No response_

### Alertmanager configuration file

_No response_

### Logs

_No response_


Discussion / hints:

Fix patch:
diff --git a/storage/remote/write_handler.go b/storage/remote/write_handler.go
index 9891c6aae7c..971f7b457e8 100644
--- a/storage/remote/write_handler.go
+++ b/storage/remote/write_handler.go
@@ -68,7 +68,7 @@ func (h *writeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	err = h.write(r.Context(), req)
 	switch {
 	case err == nil:
-	case errors.Is(err, storage.ErrOutOfOrderSample), errors.Is(err, storage.ErrOutOfBounds), errors.Is(err, storage.ErrDuplicateSampleForTimestamp):
+	case errors.Is(err, storage.ErrOutOfOrderSample), errors.Is(err, storage.ErrOutOfBounds), errors.Is(err, storage.ErrDuplicateSampleForTimestamp), errors.Is(err, storage.ErrTooOldSample):
 		// Indicated an out of order sample is a bad request to prevent retries.
 		http.Error(w, err.Error(), http.StatusBadRequest)
 		return


Test patch: