Repository: prometheus/prometheus (language: go)

Issue:
Relabel rules: a label that is deleted then set will read back as empty
### What did you do?

A snippet from `relabel_configs:`
```
  - source_labels: [__meta_kubernetes_pod_annotation_XXX_metrics_port]
    action: replace
    regex: (.+)
    replacement: metrics
    target_label: __meta_kubernetes_pod_container_port_name
  - source_labels: [__meta_kubernetes_pod_container_port_name]
    regex: ^metrics$
    action: keep
```

### What did you expect to see?

All metrics from a pod with `XXX.metrics.port` annotation are scraped.
(As worked up to `2.42.0` version (inclusive).)

### What did you see instead? Under which circumstances?

No metrics from a pod.
After upgrade to `2.43.0`.

### System information

_No response_

### Prometheus version

_No response_

### Prometheus configuration file

_No response_

### Alertmanager version

_No response_

### Alertmanager configuration file

_No response_

### Logs

```text
Please, pay attention to:

https://github.com/prometheus/prometheus/blob/main/model/labels/labels.go#L492
https://github.com/prometheus/prometheus/blob/main/model/labels/labels.go#L519
https://github.com/prometheus/prometheus/blob/main/model/labels/labels.go#L535
```



Discussion / hints:
### Prometheus version
`2.43.0`
(I am sorry.)
What is the "error" from your title?

Also, I did not understand the "Please, pay attention to:" part.  Please explain again in a different way.
Can you provide the problematic set of labels that are dropped but should not, with the complete relabeling rules?
Hi, I'm sorry for being so monosyllabic.
Further info is as follows:

> What is the "error" from your title?

Well, I'd presumed when all metrics were gone after the upgrade and the `Target Labels` read `Dropped` within the `Service Discovery` view (Prometheus) there could be something wrong with relabeling.
 
> Also, I did not understand the "Please, pay attention to:" part. Please explain again in a different way.

There was a small misunderstanding with a colleague of mine over here, I C&P wrong links, sorry for that.

FMPOV I can see some inconsistency at
https://github.com/prometheus/prometheus/blame/release-2.43/model/labels/labels.go#L592
vs.
https://github.com/prometheus/prometheus/blame/release-2.43/model/labels/labels.go#L534
What if there's a same label within both `add` and `del` arrays?

TBH, the `Set()` function
https://github.com/prometheus/prometheus/blame/release-2.43/model/labels/labels.go#L517
seems suspicious to me as well - it ignores the `del` array and just sets the `add` array, but it might be just my ignorance.

Hopefully my additions will help.
Real use cases for the snippet
```
  - source_labels: [__meta_kubernetes_pod_annotation_XXX_metrics_port]
    action: replace
    regex: (.+)
    replacement: metrics
    target_label: __meta_kubernetes_pod_container_port_name
  - source_labels: [__meta_kubernetes_pod_container_port_name]
    regex: ^metrics$
    action: keep
```
as follows:

When there is the Discovered Label `__meta_kubernetes_pod_annotation_XXX_metrics_port="9091"` set
`2.42.0`: metrics are scraped
`2.43.0`: no metrics.

When there are both Discovered Labels `__meta_kubernetes_pod_annotation_XXX_metrics_port="9091"` and `__meta_kubernetes_pod_container_port_name="metrics"` set,
`2.42.0`: metrics are scraped
`2.43.0`: metrics are scraped.

When there is the Discovered Label `__meta_kubernetes_pod_container_port_name="metrics"` set,
`2.42.0`: metrics are scraped
`2.43.0`: metrics are scraped.
As for
> ... MPOV I can see some inconsistency at...

[metioned above](https://github.com/prometheus/prometheus/issues/12283#issuecomment-1519742169), we've tested steps:

1. `Del("LABEL")`
2. `Set("LABEL")`

from `Get("LABEL")` we've got nothing, from `Labels(...)` we've got the label `LABEL`.
OK, now I see, if you call `Del` followed by `Set` of the same label name, then Builder will sometimes not return the value.
That was unintended.


Fix patch:
diff --git a/model/labels/labels.go b/model/labels/labels.go
index 93524ddcfc9..9ac0e5b53fc 100644
--- a/model/labels/labels.go
+++ b/model/labels/labels.go
@@ -533,16 +533,15 @@ func (b *Builder) Set(n, v string) *Builder {
 }
 
 func (b *Builder) Get(n string) string {
-	for _, d := range b.del {
-		if d == n {
-			return ""
-		}
-	}
+	// Del() removes entries from .add but Set() does not remove from .del, so check .add first.
 	for _, a := range b.add {
 		if a.Name == n {
 			return a.Value
 		}
 	}
+	if slices.Contains(b.del, n) {
+		return ""
+	}
 	return b.base.Get(n)
 }
 
diff --git a/model/labels/labels_string.go b/model/labels/labels_string.go
index ff46103ebcd..6d54e98ab94 100644
--- a/model/labels/labels_string.go
+++ b/model/labels/labels_string.go
@@ -593,14 +593,15 @@ func (b *Builder) Set(n, v string) *Builder {
 }
 
 func (b *Builder) Get(n string) string {
-	if slices.Contains(b.del, n) {
-		return ""
-	}
+	// Del() removes entries from .add but Set() does not remove from .del, so check .add first.
 	for _, a := range b.add {
 		if a.Name == n {
 			return a.Value
 		}
 	}
+	if slices.Contains(b.del, n) {
+		return ""
+	}
 	return b.base.Get(n)
 }
 


Test patch:
diff --git a/model/labels/labels_test.go b/model/labels/labels_test.go
index 9e60c22516e..108d8b0de0b 100644
--- a/model/labels/labels_test.go
+++ b/model/labels/labels_test.go
@@ -607,6 +607,13 @@ func TestBuilder(t *testing.T) {
 			require.Equal(t, tcase.want.BytesWithoutLabels(nil, "aaa", "bbb"), b.Labels().Bytes(nil))
 		})
 	}
+	t.Run("set_after_del", func(t *testing.T) {
+		b := NewBuilder(FromStrings("aaa", "111"))
+		b.Del("bbb")
+		b.Set("bbb", "222")
+		require.Equal(t, FromStrings("aaa", "111", "bbb", "222"), b.Labels())
+		require.Equal(t, "222", b.Get("bbb"))
+	})
 }
 
 func TestScratchBuilder(t *testing.T) {
diff --git a/model/relabel/relabel_test.go b/model/relabel/relabel_test.go
index d277d778d10..b50ff4010a9 100644
--- a/model/relabel/relabel_test.go
+++ b/model/relabel/relabel_test.go
@@ -397,6 +397,34 @@ func TestRelabel(t *testing.T) {
 				"foo":              "bar",
 			}),
 		},
+		{ // From https://github.com/prometheus/prometheus/issues/12283
+			input: labels.FromMap(map[string]string{
+				"__meta_kubernetes_pod_container_port_name":         "foo",
+				"__meta_kubernetes_pod_annotation_XXX_metrics_port": "9091",
+			}),
+			relabel: []*Config{
+				{
+					Regex:  MustNewRegexp("^__meta_kubernetes_pod_container_port_name$"),
+					Action: LabelDrop,
+				},
+				{
+					SourceLabels: model.LabelNames{"__meta_kubernetes_pod_annotation_XXX_metrics_port"},
+					Regex:        MustNewRegexp("(.+)"),
+					Action:       Replace,
+					Replacement:  "metrics",
+					TargetLabel:  "__meta_kubernetes_pod_container_port_name",
+				},
+				{
+					SourceLabels: model.LabelNames{"__meta_kubernetes_pod_container_port_name"},
+					Regex:        MustNewRegexp("^metrics$"),
+					Action:       Keep,
+				},
+			},
+			output: labels.FromMap(map[string]string{
+				"__meta_kubernetes_pod_annotation_XXX_metrics_port": "9091",
+				"__meta_kubernetes_pod_container_port_name":         "metrics",
+			}),
+		},
 		{
 			input: labels.FromMap(map[string]string{
 				"a":  "foo",