Repository: kubernetes/kubernetes (language: go)

Issue:
Unable to reset metric's LabelValueAllowLists during test
### What happened?

While trying to add an integration test for `--allow-metric-labels` #128166 (more specifically, on this [commit](https://github.com/kubernetes/kubernetes/commit/36230b63ffd96b8fbe8acb50cf91bf8c9665793a)), I find that metrics' `LabelValueAllowLists` don't reset between tests though TestAPIServers are torn down and brought up in each tests. It caused the integration test failed because the `--allow-metric-labels` doesn't reflect on the targeted metric.

It should be due to that kube-apiserver metrics (and most system component metrics) are global(package level) variables and their [opts](https://github.com/kubernetes/kubernetes/blob/81ce66f059ec9c07cccf4069c8913e31959dea78/staging/src/k8s.io/component-base/metrics/opts.go#L57) are set once when they are imported and cannot be reset afterwards.

### What did you expect to happen?

There is a way to reset the `LabelValueAllowLists` [opts](https://github.com/kubernetes/kubernetes/blob/81ce66f059ec9c07cccf4069c8913e31959dea78/staging/src/k8s.io/component-base/metrics/opts.go#L57) in the test.

### How can we reproduce it (as minimally and precisely as possible)?

1. Add a test https://github.com/kubernetes/kubernetes/commit/e62ab1c2a172cbd04245f672e270169a0a8e041e
2. Run test suite on entire package, it would fail.
 ```
go test -v -timeout 30s k8s.io/kubernetes/test/integration/metrics
```
3. However, it would always success when only run the targeted test or comment out all the rest of the tests.
```
go test -v -timeout 30s k8s.io/kubernetes/test/integration/metrics -run ^testAPIServerMetricsLabelsWithAllowList$
```


### Anything else we need to know?

Solution options:

Option1: Add a `ResetLabelAllowList()` API to the Vec metrics and wrap it with `RestLabelAllowListForTest()` in the system component metrics packages.

Option2: Include the reset label allow list logic inside the current `Reset()` API, which can reduce the effort for extra adoption for the option1's new introduced API.

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.32.0-alpha.1.300+55b83c92b3b69c-dirty
```
</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>



Discussion / hints:
/sig instrumentation
/cc @richabanker @logicalhan 
/cc @mengjiao-liu 
/assign
Option 1, with its targeted reset of the LabelAllowList field for the metrics, is my preferred choice. We should also explicitly document this approach in the code as a test-only solution.

Separately, we should create a dedicated issue to address the broader problem of global metrics initialization in Kubernetes. This will be a significant undertaking that'll require a more comprehensive solution.

Fix patch:
diff --git a/staging/src/k8s.io/component-base/metrics/opts.go b/staging/src/k8s.io/component-base/metrics/opts.go
index 9f39c4443482c..43015169e7251 100644
--- a/staging/src/k8s.io/component-base/metrics/opts.go
+++ b/staging/src/k8s.io/component-base/metrics/opts.go
@@ -37,6 +37,14 @@ var (
 	allowListLock        sync.RWMutex
 )
 
+// ResetLabelValueAllowLists resets the allow lists for label values.
+// NOTE: This should only be used in test.
+func ResetLabelValueAllowLists() {
+	allowListLock.Lock()
+	defer allowListLock.Unlock()
+	labelValueAllowLists = map[string]*MetricLabelAllowList{}
+}
+
 // KubeOpts is superset struct for prometheus.Opts. The prometheus Opts structure
 // is purposefully not embedded here because that would change struct initialization
 // in the manner which people are currently accustomed.


Test patch:
diff --git a/staging/src/k8s.io/component-base/metrics/opts_test.go b/staging/src/k8s.io/component-base/metrics/opts_test.go
index b2e0fb0295cdb..7094f18761b98 100644
--- a/staging/src/k8s.io/component-base/metrics/opts_test.go
+++ b/staging/src/k8s.io/component-base/metrics/opts_test.go
@@ -208,3 +208,21 @@ metric2,label2: v3`,
 		})
 	}
 }
+
+func TestResetLabelValueAllowLists(t *testing.T) {
+	labelValueAllowLists = map[string]*MetricLabelAllowList{
+		"metric1": {
+			labelToAllowList: map[string]sets.Set[string]{
+				"label1": sets.New[string]("v1", "v2"),
+			},
+		},
+		"metric2": {
+			labelToAllowList: map[string]sets.Set[string]{
+				"label2": sets.New[string]("v3"),
+			},
+		},
+	}
+
+	ResetLabelValueAllowLists()
+	assert.Empty(t, labelValueAllowLists)
+}