Repository: open-telemetry/opentelemetry-operator

Issue:
Target allocation unstable in v0.121.0
### Component(s)

target allocator

### What happened?

## Description
Starting with the merge of https://github.com/open-telemetry/opentelemetry-operator/pull/3842 (determined with `git bisect`), target allocation became highly unstable in k8s clusters with churning pods. "Unstable" in this case means that scrape targets are frequently reallocated to different collector instances. If this happens frequently enough, it manifests as missed scrapes and gaps in time series.

## Steps to Reproduce
For a *relatively* simple reproduction, I took the following steps:
1. Spun up a k8s cluster in minikube with ~75 total pods. 
2. Ran target allocator and 5 collector replicas using least-weighted allocation strategy.
3. Ran a script to delete a pod from a deployment (it can be any deployment) at a regular interval.
4. Ran a script to watch the target allocator endpoints to see which collector is assigned to collect metrics for a given pod (it can once again be any pod - just not one that you're deleting regularly).

You can skip steps 1-3 if you have a k8s cluster where pods churn on a regular cadence (every few seconds).

## Expected Result
The collector pod that is assigned to scrape the pod being watched in step 4 should remain relatively consistent.

## Actual Result
The collector pod that is assigned to scrape the pod being watched in step 4 changes frequently (every few seconds).

### Kubernetes Version

v1.28.3

### Operator version

b990f36c5363a883ebcbcd94e57c7ecf2a79daa6

### Collector version

0.126.0

### Environment information

## Environment
OS: MacOS and Linux
Compiler(if manually compiled): go 1.24.2


### Log output

```shell

```

### Additional context

_No response_

Discussion:
@sfc-gh-rscott: I'm going to try to write a unit test to reproduce this more simply.@sfc-gh-rscott: I figured out the underlying issue. The old implementation of `processTargetGroups` that changed in [3842](https://github.com/open-telemetry/opentelemetry-operator/pull/3842) produced a slice of labels in a [deterministic order](https://github.com/prometheus/prometheus/blob/main/model/labels/labels.go#L422). The new implementation produces them in [non deterministic order](https://github.com/prometheus/prometheus/blob/main/model/labels/labels.go#L485-L486). They're fed to a hash function [here](https://github.com/open-telemetry/opentelemetry-operator/blob/main/cmd/otel-allocator/internal/target/target.go#L73-L105), which iterates over them in order to generate the input to the hasher. That hash is later used to make allocation decisions.

Calling `targetBuilder.Sort()` before `NewItem` in `processTargetGroups` fixes the issue.

I'm signing off for the day but I'll put together a proper unit test and a PR tomorrow unless someone beats me to it. I'm curious what @swiatekm thinks is the best strategy here to avoid regressions. We can easily assert that `processTargetGroups` produces sorted labels, but should we also check that they're sorted or just sort them there instead? Having them sorted seems to be an invariant requirement for `NewItem`.@swiatekm: Thank you for the detailed investigation @sfc-gh-rscott! A simple way to avoid regressions here would be to assert that `processTargetGroups` always gives the same output for a given input. That test currently fails due to nondeterministic map iteration order.

If you'd like to submit a fix, I'm happy to assign this issue to you. In terms of implementation, note that while you can sort `groupBuilder` without issue, since it's reset every loop revolution, sorting `targetBuilder` will cause invalid labels to be emitted. The code avoids allocations by using only a single slice to hold the labels at any given time and cutting it down back to only the group labels at each loop revolution. This only works if group labels never change, and sorting the whole slice will do that. Does that make sense?@sfc-gh-rscott: @swiatekm I'm not sure how to achieve stable ordering without doing the additional allocation. The simplest fix seems like reverting #3842. What do you think?@swiatekm: I'd accept a revert with a test as well.@sfc-gh-rscott: @swiatekm I opened https://github.com/open-telemetry/opentelemetry-operator/pull/4088 to resolve.

Solution patch:
diff --git a/.chloggen/fix-4082.yaml b/.chloggen/fix-4082.yaml
new file mode 100644
index 00000000..beb4a584
--- /dev/null
+++ b/.chloggen/fix-4082.yaml
@@ -0,0 +1,16 @@
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: bug_fix
+
+# The name of the component, or a single word describing the area of concern, (e.g. collector, target allocator, auto-instrumentation, opamp, github action)
+component: target allocator
+
+# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: ensure stable iteration order of target labels when generating hash
+
+# One or more tracking issues related to the change
+issues: [4082]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
diff --git a/.chloggen/fix-must-gather.yaml b/.chloggen/fix-must-gather.yaml
new file mode 100644
index 00000000..08e02a55
--- /dev/null
+++ b/.chloggen/fix-must-gather.yaml
@@ -0,0 +1,16 @@
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: bug_fix
+
+# The name of the component, or a single word describing the area of concern, (e.g. collector, target allocator, auto-instrumentation, opamp, github action)
+component: target allocator
+
+# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Fix OpenShift must-gather for Target Allocator
+
+# One or more tracking issues related to the change
+issues: [4084]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
diff --git a/cmd/gather/cluster/cluster.go b/cmd/gather/cluster/cluster.go
index b5567a6e..7ed04d65 100644
--- a/cmd/gather/cluster/cluster.go
+++ b/cmd/gather/cluster/cluster.go
@@ -248,6 +248,33 @@ func (c *Cluster) GetOpenTelemetryCollectors() error {
 	return nil
 }
 
+func (c *Cluster) GetTargetAllocators() error {
+	tas := otelv1alpha1.TargetAllocatorList{}
+
+	err := c.config.KubernetesClient.List(context.TODO(), &tas)
+	if err != nil {
+		return err
+	}
+
+	log.Println("TargetAllocators found:", len(tas.Items))
+
+	errorDetected := false
+
+	for _, ta := range tas.Items {
+		otelTA := ta
+		err := c.processOTELTargetAllocator(&otelTA)
+		if err != nil {
+			log.Fatalln(err)
+			errorDetected = true
+		}
+	}
+
+	if errorDetected {
+		return fmt.Errorf("something failed while getting the targetallocators")
+	}
+	return nil
+}
+
 func (c *Cluster) GetInstrumentations() error {
 	instrumentations := otelv1alpha1.InstrumentationList{}
 
@@ -281,7 +308,7 @@ func (c *Cluster) GetInstrumentations() error {
 
 func (c *Cluster) processOTELCollector(otelCol *otelv1beta1.OpenTelemetryCollector) error {
 	log.Printf("Processing OpenTelemetryCollector %s/%s", otelCol.Namespace, otelCol.Name)
-	folder, err := createOTELFolder(c.config.CollectionDir, otelCol)
+	folder, err := createOTELFolder(c.config.CollectionDir, otelCol.ObjectMeta)
 	if err != nil {
 		return err
 	}
@@ -295,11 +322,28 @@ func (c *Cluster) processOTELCollector(otelCol *otelv1beta1.OpenTelemetryCollect
 	return nil
 }
 
+func (c *Cluster) processOTELTargetAllocator(ta *otelv1alpha1.TargetAllocator) error {
+	log.Printf("Processing TargetAllocator %s/%s", ta.Namespace, ta.Name)
+	folder, err := createOTELFolder(c.config.CollectionDir, ta.ObjectMeta)
+	if err != nil {
+		return err
+	}
+	writeToFile(folder, ta)
+
+	err = c.processOwnedResources(ta, folder)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
 func (c *Cluster) processOwnedResources(owner interface{}, folder string) error {
 	resourceTypes := []struct {
 		list     client.ObjectList
 		apiCheck func() bool
 	}{
+		{&otelv1alpha1.TargetAllocatorList{}, func() bool { return true }},
 		{&appsv1.DaemonSetList{}, func() bool { return true }},
 		{&appsv1.DeploymentList{}, func() bool { return true }},
 		{&appsv1.StatefulSetList{}, func() bool { return true }},
@@ -401,6 +445,9 @@ func hasOwnerReference(obj client.Object, owner interface{}) bool {
 	case *otelv1beta1.OpenTelemetryCollector:
 		ownerKind = o.Kind
 		ownerUID = o.UID
+	case *otelv1alpha1.TargetAllocator:
+		ownerKind = o.Kind
+		ownerUID = o.UID
 	default:
 		return false
 	}
diff --git a/cmd/gather/cluster/write.go b/cmd/gather/cluster/write.go
index 76cbb726..bf345679 100644
--- a/cmd/gather/cluster/write.go
+++ b/cmd/gather/cluster/write.go
@@ -14,16 +14,15 @@ import (
 	"strings"
 
 	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/runtime/serializer/json"
 	cgocorev1 "k8s.io/client-go/kubernetes/typed/core/v1"
 	"sigs.k8s.io/controller-runtime/pkg/client"
-
-	"github.com/open-telemetry/opentelemetry-operator/apis/v1beta1"
 )
 
-func createOTELFolder(collectionDir string, otelCol *v1beta1.OpenTelemetryCollector) (string, error) {
+func createOTELFolder(collectionDir string, otelCol metav1.ObjectMeta) (string, error) {
 	outputDir := filepath.Join(collectionDir, "namespaces", otelCol.Namespace, otelCol.Name)
 	err := os.MkdirAll(outputDir, os.ModePerm)
 	if err != nil {
diff --git a/cmd/gather/main.go b/cmd/gather/main.go
index 8d0c4010..ab8d30af 100644
--- a/cmd/gather/main.go
+++ b/cmd/gather/main.go
@@ -68,6 +68,10 @@ func main() {
 	if err != nil {
 		log.Fatalln(err)
 	}
+	err = cluster.GetTargetAllocators()
+	if err != nil {
+		log.Fatalln(err)
+	}
 	err = cluster.GetInstrumentations()
 	if err != nil {
 		log.Fatalln(err)
diff --git a/cmd/otel-allocator/internal/target/discovery.go b/cmd/otel-allocator/internal/target/discovery.go
index 201e6bef..c72966d0 100644
--- a/cmd/otel-allocator/internal/target/discovery.go
+++ b/cmd/otel-allocator/internal/target/discovery.go
@@ -24,8 +24,6 @@ import (
 	allocatorWatcher "github.com/open-telemetry/opentelemetry-operator/cmd/otel-allocator/internal/watcher"
 )
 
-const labelBuilderPreallocSize = 100
-
 var (
 	targetsDiscovered = promauto.NewGaugeVec(prometheus.GaugeOpts{
 		Name: "opentelemetry_allocator_targets",
@@ -195,24 +193,25 @@ func (m *Discoverer) Reload() {
 
 // processTargetGroups processes the target groups and returns a map of targets.
 func (m *Discoverer) processTargetGroups(jobName string, groups []*targetgroup.Group, intoTargets []*Item) {
-	groupBuilder := labels.NewScratchBuilder(labelBuilderPreallocSize)
+	builder := labels.NewBuilder(labels.Labels{})
 	timer := prometheus.NewTimer(processTargetGroupsDuration.WithLabelValues(jobName))
 
 	defer timer.ObserveDuration()
 	var count float64 = 0
 	index := 0
 	for _, tg := range groups {
-		groupBuilder.Reset()
+		builder.Reset(labels.EmptyLabels())
 		for ln, lv := range tg.Labels {
-			groupBuilder.Add(string(ln), string(lv))
+			builder.Set(string(ln), string(lv))
 		}
+		groupLabels := builder.Labels()
 		for _, t := range tg.Targets {
 			count++
-			targetBuilder := groupBuilder
+			builder.Reset(groupLabels)
 			for ln, lv := range t {
-				targetBuilder.Add(string(ln), string(lv))
+				builder.Set(string(ln), string(lv))
 			}
-			item := NewItem(jobName, string(t[model.AddressLabel]), targetBuilder.Labels(), "")
+			item := NewItem(jobName, string(t[model.AddressLabel]), builder.Labels(), "")
 			intoTargets[index] = item
 			index++
 		}



Test patch:
diff --git a/cmd/gather/cluster/write_test.go b/cmd/gather/cluster/write_test.go
index 0d2e2669..494f5c6c 100644
--- a/cmd/gather/cluster/write_test.go
+++ b/cmd/gather/cluster/write_test.go
@@ -182,7 +182,7 @@ func TestCreateOTELFolder(t *testing.T) {
 		},
 	}
 
-	outputDir, err := createOTELFolder(collectionDir, otelCol)
+	outputDir, err := createOTELFolder(collectionDir, otelCol.ObjectMeta)
 
 	expectedDir := filepath.Join(collectionDir, "namespaces", otelCol.Namespace, otelCol.Name)
 	assert.NoError(t, err)
diff --git a/cmd/otel-allocator/internal/target/discovery_test.go b/cmd/otel-allocator/internal/target/discovery_test.go
index cf59a7a7..a3d907fa 100644
--- a/cmd/otel-allocator/internal/target/discovery_test.go
+++ b/cmd/otel-allocator/internal/target/discovery_test.go
@@ -16,6 +16,7 @@ import (
 	"github.com/prometheus/common/model"
 	promconfig "github.com/prometheus/prometheus/config"
 	"github.com/prometheus/prometheus/discovery"
+	"github.com/prometheus/prometheus/discovery/targetgroup"
 	"github.com/prometheus/prometheus/model/relabel"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -461,6 +462,41 @@ func TestDiscovery_NoConfig(t *testing.T) {
 	assert.Equal(t, expectedScrapeConfigs, scu.mockCfg)
 }
 
+func TestProcessTargetGroups_StableLabelIterationOrder(t *testing.T) {
+	// Labels are used to compute the hash of targets and hashing is
+	// reliant on consistent ordering of labels. Creating one label
+	// per letter of the english alphabet is enough to reliably
+	// reproduce inconsistent label ordering due to non-deterministic
+	// map iteration order + lack of sorting.
+	groupLabels := model.LabelSet{}
+	for i := 'a'; i <= 'm'; i++ {
+		groupLabels[model.LabelName(i)] = model.LabelValue(i)
+	}
+
+	targetLabels := model.LabelSet{}
+	for i := 'n'; i <= 'z'; i++ {
+		targetLabels[model.LabelName(i)] = model.LabelValue(i)
+	}
+
+	groups := []*targetgroup.Group{
+		{
+			Labels:  groupLabels,
+			Source:  "",
+			Targets: []model.LabelSet{targetLabels},
+		},
+	}
+
+	results := make([]*Item, 1)
+	var d *Discoverer
+	d.processTargetGroups("test", groups, results)
+
+	for i, l := range results[0].Labels {
+		expected := string(rune('a' + i))
+		assert.Equal(t, expected, l.Name, "unexpected label key at index %d", i)
+		assert.Equal(t, expected, l.Value, "unexpected label value at index %d", i)
+	}
+}
+
 func BenchmarkApplyScrapeConfig(b *testing.B) {
 	numConfigs := 1000
 	scrapeConfig := promconfig.ScrapeConfig{
diff --git a/tests/e2e-openshift/must-gather/check_must_gather.sh b/tests/e2e-openshift/must-gather/check_must_gather.sh
index 56221ebf..0d33f0b6 100755
--- a/tests/e2e-openshift/must-gather/check_must_gather.sh
+++ b/tests/e2e-openshift/must-gather/check_must_gather.sh
@@ -9,31 +9,32 @@ oc adm must-gather --dest-dir=$MUST_GATHER_DIR --image=ghcr.io/open-telemetry/op
 # Define required files and directories
 REQUIRED_ITEMS=(
   event-filter.html
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/olm/*opentelemetry-operato*.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/olm/clusterserviceversion-opentelemetry-operator-v*.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/olm/installplan-install-*.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/olm/subscription-opentelemetry-operator-v*-sub.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/service-stateful-collector-headless.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/service-stateful-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/deployment-stateful-targetallocator.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/service-stateful-collector-monitoring.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/poddisruptionbudget-stateful-targetallocator.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/poddisruptionbudget-stateful-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/service-stateful-targetallocator.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/configmap-stateful-collector-*.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/configmap-stateful-targetallocator.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/statefulset-stateful-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/opentelemetrycollector-stateful.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/serviceaccount-stateful-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/service-sidecar-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/opentelemetrycollector-sidecar.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/service-sidecar-collector-monitoring.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/configmap-sidecar-collector-*.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/serviceaccount-sidecar-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/service-sidecar-collector-headless.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/chainsaw-must-gather/instrumentation-nodejs.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/opentelemetry-operator-controller-manager-*
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/deployment-opentelemetry-operator-controller-manager.yaml
+  *-must-gather-sha256-*/olm/*opentelemetry-operato*.yaml
+  *-must-gather-sha256-*/olm/clusterserviceversion-opentelemetry-operator-v*.yaml
+  *-must-gather-sha256-*/olm/installplan-install-*.yaml
+  *-must-gather-sha256-*/olm/subscription-opentelemetry-operator-v*-sub.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/service-stateful-collector-headless.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/service-stateful-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/targetallocator-stateful.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/deployment-stateful-targetallocator.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/service-stateful-collector-monitoring.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/poddisruptionbudget-stateful-targetallocator.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/poddisruptionbudget-stateful-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/service-stateful-targetallocator.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/configmap-stateful-collector-*.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/configmap-stateful-targetallocator.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/statefulset-stateful-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/opentelemetrycollector-stateful.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/stateful/serviceaccount-stateful-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/service-sidecar-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/opentelemetrycollector-sidecar.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/service-sidecar-collector-monitoring.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/configmap-sidecar-collector-*.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/serviceaccount-sidecar-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-must-gather/sidecar/service-sidecar-collector-headless.yaml
+  *-must-gather-sha256-*/chainsaw-must-gather/instrumentation-nodejs.yaml
+  *-must-gather-sha256-*/opentelemetry-operator-controller-manager-*
+  *-must-gather-sha256-*/deployment-opentelemetry-operator-controller-manager.yaml
   timestamp
 )
 
diff --git a/tests/e2e-openshift/otlp-metrics-traces/check_must_gather.sh b/tests/e2e-openshift/otlp-metrics-traces/check_must_gather.sh
index 6c2d5b6e..b0923783 100755
--- a/tests/e2e-openshift/otlp-metrics-traces/check_must_gather.sh
+++ b/tests/e2e-openshift/otlp-metrics-traces/check_must_gather.sh
@@ -10,20 +10,20 @@ oc adm must-gather --dest-dir=$MUST_GATHER_DIR --image=ghcr.io/open-telemetry/op
 # Define required files and directories
 REQUIRED_ITEMS=(
   event-filter.html
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/olm/clusterserviceversion-opentelemetry-operator-*.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/olm/*opentelemetry-operator*.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/olm/installplan-install-*.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/olm/subscription-opentelemetry-operator-v*-sub.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/service-cluster-collector-collector-headless.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/deployment-cluster-collector-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/service-cluster-collector-collector-monitoring.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/opentelemetrycollector-cluster-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/configmap-cluster-collector-collector-*.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/serviceaccount-cluster-collector-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/service-cluster-collector-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/poddisruptionbudget-cluster-collector-collector.yaml
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/opentelemetry-operator-controller-manager-*
-  ghcr-io-open-telemetry-opentelemetry-operator-must-gather-sha256-*/deployment-opentelemetry-operator-controller-manager.yaml
+  *-must-gather-sha256-*/olm/clusterserviceversion-opentelemetry-operator-*.yaml
+  *-must-gather-sha256-*/olm/*opentelemetry-operator*.yaml
+  *-must-gather-sha256-*/olm/installplan-install-*.yaml
+  *-must-gather-sha256-*/olm/subscription-opentelemetry-operator-v*-sub.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/service-cluster-collector-collector-headless.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/deployment-cluster-collector-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/service-cluster-collector-collector-monitoring.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/opentelemetrycollector-cluster-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/configmap-cluster-collector-collector-*.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/serviceaccount-cluster-collector-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/service-cluster-collector-collector.yaml
+  *-must-gather-sha256-*/namespaces/chainsaw-otlp-metrics/cluster-collector/poddisruptionbudget-cluster-collector-collector.yaml
+  *-must-gather-sha256-*/opentelemetry-operator-controller-manager-*
+  *-must-gather-sha256-*/deployment-opentelemetry-operator-controller-manager.yaml
   timestamp
 )