Repository: kudobuilder/kuttl (go)

Issue:
Conditional execution of test steps
<!-- Please only use this template for submitting enhancement requests.
Implementing your enhancement will follow the KEP process: https://github.com/kudobuilder/kuttl/blob/main/keps/0001-kep-process.md
-->

**What would you like to be added**:

I would like to run a test step only if a condition is met. This would allow me to install and test cluster specific resources.

For example, Helm has a way to check if a certain API is available in the cluster by querying the `capabilities` object and install custom resources of the corresponding `kind`.

Here is an snippet  that creates a SCC resource only when the `security.openshift.io/v1` API is available:

```
{{ if .Capabilities.APIVersions.Has "security.openshift.io/v1" }}
---
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: airflow-scc
...
{{ end }}
```

As you'd guess, the custom resource above is only created if the target cluster of the Helm chart is an OpenShift cluster.

In a similar fashion, I would like to be able to run a `kuttl` test step that creates  the same SCC only if the tests are running against OpenShift or to be more general, only if the API `security.openshift.io/v1` is available.

**Why is this needed**:

We write many tests and would like them to be reusable without modification across kubernetes implementations.

Currently, to run the same tests on KinD and OpenShift for example, we'd have to maintain two largely redundant code bases for each target cluster. This duplicates the effort to create and maintain these tests.


Pull request:
Support for test run labels/selectors
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://github.com/kudobuilder/kuttl/blob/main/CONTRIBUTING.md
2. Make sure you have added and ran the tests before submitting your PR
3. If the PR is unfinished, start it as a Draft PR: https://github.blog/2019-02-14-introducing-draft-pull-requests/
-->

# **What this PR does / why we need it**:

Make it possible to associate label sets with test runs and put label selectors for individual files [as discussed on Slack](https://kubernetes.slack.com/archives/CG3HTFCMV/p1687170514079399). The purpose is conditional ignoring of selected files with the goal of having a single test directory but with some variation to allow for testing on different platforms.

# Why a new resource type?

I was not able to find any appropriate place.
* A `TestStep` object applies to all files in a given step, and only single such object [is permitted](https://github.com/kudobuilder/kuttl/blob/f6d64c915c8dd9e2da354562c3d5c6fcf88aec2b/pkg/test/step.go#L527) per step. This might be too coarse granularity, since for example in a given step we might only want to vary some assertions and keep the apply files constant.
* Similarly, only a single `TestAssert` is used in a given step (although unlike for `TestStep`, additional `TestAssert` objects seem to be silently ignored rather than complained about). Also, `TestAssert` objects are only active if they appear in an NN-assert*.yaml file - they seem to be ignored in all other files, [including errors](https://github.com/kudobuilder/kuttl/issues/456) files and varying the apply files is a goal.

# Simple usage example

## Setup

```
[mowsiany@mowsiany x]$ head -n 99 tt/*
==> tt/01-assert-cat.yaml <==
apiVersion: kuttl.dev/v1beta1
kind: TestFile
testRunSelector:
  matchLabels:
    flavor: cat
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test
data:
  key: fluffy

==> tt/01-assert-dog.yaml <==
apiVersion: kuttl.dev/v1beta1
kind: TestFile
testRunSelector:
  matchLabels:
    flavor: dog
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test
data:
  key: rex

==> tt/01-create.yaml <==
apiVersion: v1
kind: ConfigMap
metadata:
  name: test
data:
  key: fluffy
[mowsiany@mowsiany x]$ make -C .. cli
make: Entering directory '/home/mowsiany/go/src/github.com/kudobuilder/kuttl'
go build -ldflags "-X github.com/kudobuilder/kuttl/pkg/version.gitVersion=0.15.0 -X github.com/kudobuilder/kuttl/pkg/version.gitCommit=e7d8400e -X github.com/kudobuilder/kuttl/pkg/version.buildDate=2023-08-22T08:49:58Z" -o bin/kubectl-kuttl ./cmd/kubectl-kuttl
make: Leaving directory '/home/mowsiany/go/src/github.com/kudobuilder/kuttl'
```

## A `flavor=cat` test run

```
[mowsiany@mowsiany x]$ ../bin/kubectl-kuttl test . --start-kind --timeout 15 --test-run-labels flavor=cat
2023/08/22 10:39:41 running without a 'kuttl-test.yaml' configuration
2023/08/22 10:39:41 kutt-test config testdirs is overridden with args: [ . ]
=== RUN   kuttl
    harness.go:465: starting setup
    harness.go:252: running tests with KIND.
    harness.go:176: temp folder created /tmp/nix-shell.mdjyec/nix-shell.UvjEuW/kuttl3988091570
    harness.go:158: Starting KIND cluster
    kind.go:66: Adding Containers to KIND...
    harness.go:278: Successful connection to cluster at: https://127.0.0.1:40829
    harness.go:363: running tests
    harness.go:75: going to run test suite with timeout of 15 seconds for each step
    harness.go:375: testsuite: . has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/tt
=== PAUSE kuttl/harness/tt
=== CONT  kuttl/harness/tt
Skipping file "/home/mowsiany/go/src/github.com/kudobuilder/kuttl/x/tt/01-assert-dog.yaml", label selector does not match test run labels.
    logger.go:42: 10:40:10 | tt | Creating namespace: kuttl-test-shining-cat
    logger.go:42: 10:40:10 | tt/1-create | starting test step 1-create
    logger.go:42: 10:40:10 | tt/1-create | ConfigMap:kuttl-test-shining-cat/test created
    logger.go:42: 10:40:10 | tt/1-create | test step completed 1-create
    logger.go:42: 10:40:10 | tt | tt events from ns kuttl-test-shining-cat:
    logger.go:42: 10:40:10 | tt | Deleting namespace: kuttl-test-shining-cat
=== NAME  kuttl
    harness.go:408: run tests finished
    harness.go:516: cleaning up
    harness.go:525: collecting cluster logs to kind-logs-1692693615
    harness.go:573: removing temp folder: "/tmp/nix-shell.mdjyec/nix-shell.UvjEuW/kuttl3988091570"
    harness.go:579: tearing down kind cluster
--- PASS: kuttl (36.29s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/tt (5.20s)
PASS
```

## A `flavor=dog` run


```
[mowsiany@mowsiany x]$ ../bin/kubectl-kuttl test . --start-kind --timeout 15 --test-run-labels flavor=dog
2023/08/22 10:40:33 running without a 'kuttl-test.yaml' configuration
2023/08/22 10:40:33 kutt-test config testdirs is overridden with args: [ . ]
=== RUN   kuttl
    harness.go:465: starting setup
    harness.go:252: running tests with KIND.
    harness.go:176: temp folder created /tmp/nix-shell.mdjyec/nix-shell.UvjEuW/kuttl3909311779
    harness.go:158: Starting KIND cluster
    kind.go:66: Adding Containers to KIND...
    harness.go:278: Successful connection to cluster at: https://127.0.0.1:40901
    harness.go:363: running tests
    harness.go:75: going to run test suite with timeout of 15 seconds for each step
    harness.go:375: testsuite: . has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/tt
=== PAUSE kuttl/harness/tt
=== CONT  kuttl/harness/tt
Skipping file "/home/mowsiany/go/src/github.com/kudobuilder/kuttl/x/tt/01-assert-cat.yaml", label selector does not match test run labels.
    logger.go:42: 10:41:01 | tt | Creating namespace: kuttl-test-integral-porpoise
    logger.go:42: 10:41:01 | tt/1-create | starting test step 1-create
    logger.go:42: 10:41:01 | tt/1-create | ConfigMap:kuttl-test-integral-porpoise/test created
    logger.go:42: 10:41:17 | tt/1-create | test step failed 1-create
    case.go:366: failed in step 1-create
    case.go:368: --- ConfigMap:kuttl-test-integral-porpoise/test
        +++ ConfigMap:kuttl-test-integral-porpoise/test
        @@ -1,8 +1,18 @@
         apiVersion: v1
         data:
        -  key: rex
        +  key: fluffy
         kind: ConfigMap
         metadata:
        +  managedFields:
        +  - apiVersion: v1
        +    fieldsType: FieldsV1
        +    fieldsV1:
        +      f:data:
        +        .: {}
        +        f:key: {}
        +    manager: kubectl-kuttl
        +    operation: Update
        +    time: "2023-08-22T08:41:01Z"
           name: test
           namespace: kuttl-test-integral-porpoise
         
        
    case.go:368: resource ConfigMap:kuttl-test-integral-porpoise/test: .data.key: value mismatch, expected: rex != actual: fluffy
    logger.go:42: 10:41:17 | tt | tt events from ns kuttl-test-integral-porpoise:
    logger.go:42: 10:41:17 | tt | Deleting namespace: kuttl-test-integral-porpoise
=== NAME  kuttl
    harness.go:408: run tests finished
    harness.go:516: cleaning up
    harness.go:525: collecting cluster logs to kind-logs-1692693682
    harness.go:573: removing temp folder: "/tmp/nix-shell.mdjyec/nix-shell.UvjEuW/kuttl3909311779"
    harness.go:579: tearing down kind cluster
--- FAIL: kuttl (49.76s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/kind-logs-1692693615 (5.20s)
        --- FAIL: kuttl/harness/tt (20.57s)
FAIL
[mowsiany@mowsiany x]$ 
```

# Real-world usage example

See [this test directory](https://github.com/stackrox/stackrox/tree/porridge/ROX-16724/operator/tests/central/basic-central) (see the files ending in `(non-)openshift.yaml`).

<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
-->
Fixes #379


Patch:
diff --git a/pkg/kuttlctl/cmd/values.go b/pkg/kuttlctl/cmd/values.go
new file mode 100644
index 0000000..244891d
--- /dev/null
+++ b/pkg/kuttlctl/cmd/values.go
@@ -0,0 +1,30 @@
+package cmd
+
+import (
+	"fmt"
+
+	"k8s.io/apimachinery/pkg/labels"
+)
+
+type labelSetValue labels.Set
+
+func (v *labelSetValue) String() string {
+	return labels.Set(*v).String()
+}
+
+func (v *labelSetValue) Set(s string) error {
+	l, err := labels.ConvertSelectorToLabelsMap(s)
+	if err != nil {
+		return fmt.Errorf("cannot parse label set: %w", err)
+	}
+	*v = labelSetValue(l)
+	return nil
+}
+
+func (v *labelSetValue) Type() string {
+	return "labelSet"
+}
+
+func (v labelSetValue) AsLabelSet() labels.Set {
+	return labels.Set(v)
+}


Test patch:
diff --git a/pkg/apis/testharness/v1beta1/test_types.go b/pkg/apis/testharness/v1beta1/test_types.go
index 9298142..675fd29 100644
--- a/pkg/apis/testharness/v1beta1/test_types.go
+++ b/pkg/apis/testharness/v1beta1/test_types.go
@@ -13,6 +13,19 @@ type RestConfig struct {
 
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
 
+// TestFile contains attributes of a single test file.
+type TestFile struct {
+	// The type meta object, should always be a GVK of kuttl.dev/v1beta1/TestFile.
+	metav1.TypeMeta `json:",inline"`
+	// Set labels or the test suite name.
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	// Which test runs should this file be used in. Empty selector matches all test runs.
+	TestRunSelector *metav1.LabelSelector `json:"testRunSelector,omitempty"`
+}
+
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+
 // TestSuite configures which tests should be loaded.
 type TestSuite struct {
 	// The type meta object, should always be a GVK of kuttl.dev/v1beta1/TestSuite or kuttl.dev/v1beta1/TestSuite.
diff --git a/pkg/apis/testharness/v1beta1/zz_generated.deepcopy.go b/pkg/apis/testharness/v1beta1/zz_generated.deepcopy.go
index 68f8d6c..4698a0c 100644
--- a/pkg/apis/testharness/v1beta1/zz_generated.deepcopy.go
+++ b/pkg/apis/testharness/v1beta1/zz_generated.deepcopy.go
@@ -20,6 +20,7 @@ limitations under the License.
 package v1beta1
 
 import (
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	runtime "k8s.io/apimachinery/pkg/runtime"
 )
 
@@ -147,6 +148,37 @@ func (in *TestCollector) DeepCopy() *TestCollector {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *TestFile) DeepCopyInto(out *TestFile) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	if in.TestRunSelector != nil {
+		in, out := &in.TestRunSelector, &out.TestRunSelector
+		*out = new(v1.LabelSelector)
+		(*in).DeepCopyInto(*out)
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new TestFile.
+func (in *TestFile) DeepCopy() *TestFile {
+	if in == nil {
+		return nil
+	}
+	out := new(TestFile)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *TestFile) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *TestStep) DeepCopyInto(out *TestStep) {
 	*out = *in
diff --git a/pkg/kuttlctl/cmd/test.go b/pkg/kuttlctl/cmd/test.go
index a54698d..9b80db6 100644
--- a/pkg/kuttlctl/cmd/test.go
+++ b/pkg/kuttlctl/cmd/test.go
@@ -61,6 +61,7 @@ func newTestCmd() *cobra.Command { //nolint:gocyclo
 	reportName := "kuttl-report"
 	namespace := ""
 	suppress := []string{}
+	var runLabels labelSetValue
 
 	options := harness.TestSuite{}
 
@@ -229,6 +230,7 @@ For more detailed documentation, visit: https://kuttl.dev`,
 				harness := test.Harness{
 					TestSuite: options,
 					T:         t,
+					RunLabels: runLabels.AsLabelSet(),
 				}
 
 				harness.Run()
@@ -257,6 +259,7 @@ For more detailed documentation, visit: https://kuttl.dev`,
 	testCmd.Flags().StringVar(&reportName, "report-name", "kuttl-report", "Name for the report.  Report location determined by --artifacts-dir and report file type determined by --report.")
 	testCmd.Flags().StringVarP(&namespace, "namespace", "n", "", "Namespace to use for tests. Provided namespaces must exist prior to running tests.")
 	testCmd.Flags().StringSliceVar(&suppress, "suppress-log", []string{}, "Suppress logging for these kinds of logs (events).")
+	testCmd.Flags().Var(&runLabels, "test-run-labels", "Labels to use for this test run.")
 	// This cannot be a global flag because pkg/test/utils.RunTests calls flag.Parse which barfs on unknown top-level flags.
 	// Putting it here at least does not advertise it on a level where using it is impossible.
 	test.SetFlags(testCmd.Flags())
diff --git a/pkg/test/case.go b/pkg/test/case.go
index 97814b0..b709325 100644
--- a/pkg/test/case.go
+++ b/pkg/test/case.go
@@ -18,6 +18,7 @@ import (
 	eventsbeta1 "k8s.io/api/events/v1beta1"
 	k8serrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/util/wait"
 	"k8s.io/client-go/discovery"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -40,6 +41,7 @@ type Case struct {
 	SkipDelete         bool
 	Timeout            int
 	PreferredNamespace string
+	RunLabels          labels.Set
 
 	Client          func(forceNew bool) (client.Client, error)
 	DiscoveryClient func() (discovery.DiscoveryInterface, error)
@@ -458,13 +460,14 @@ func (t *Case) LoadTestSteps() error {
 
 	for index, files := range testStepFiles {
 		testStep := &Step{
-			Timeout:    t.Timeout,
-			Index:      int(index),
-			SkipDelete: t.SkipDelete,
-			Dir:        t.Dir,
-			Asserts:    []client.Object{},
-			Apply:      []client.Object{},
-			Errors:     []client.Object{},
+			Timeout:       t.Timeout,
+			Index:         int(index),
+			SkipDelete:    t.SkipDelete,
+			Dir:           t.Dir,
+			TestRunLabels: t.RunLabels,
+			Asserts:       []client.Object{},
+			Apply:         []client.Object{},
+			Errors:        []client.Object{},
 		}
 
 		for _, file := range files {
diff --git a/pkg/test/case_test.go b/pkg/test/case_test.go
index 0f47945..21a9a06 100644
--- a/pkg/test/case_test.go
+++ b/pkg/test/case_test.go
@@ -1,12 +1,14 @@
 package test
 
 import (
+	"fmt"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/labels"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
 	harness "github.com/kudobuilder/kuttl/pkg/apis/testharness/v1beta1"
@@ -18,10 +20,12 @@ import (
 func TestLoadTestSteps(t *testing.T) {
 	for _, tt := range []struct {
 		path      string
+		runLabels labels.Set
 		testSteps []Step
 	}{
 		{
-			"test_data/with-overrides/",
+			"test_data/with-overrides",
+			labels.Set{},
 			[]Step{
 				{
 					Name:  "with-test-step-name-override",
@@ -52,7 +56,8 @@ func TestLoadTestSteps(t *testing.T) {
 							"qosClass": "BestEffort",
 						}),
 					},
-					Errors: []client.Object{},
+					Errors:        []client.Object{},
+					TestRunLabels: labels.Set{},
 				},
 				{
 					Name:  "test-assert",
@@ -96,7 +101,8 @@ func TestLoadTestSteps(t *testing.T) {
 							"qosClass": "BestEffort",
 						}),
 					},
-					Errors: []client.Object{},
+					Errors:        []client.Object{},
+					TestRunLabels: labels.Set{},
 				},
 				{
 					Name:  "pod",
@@ -124,7 +130,8 @@ func TestLoadTestSteps(t *testing.T) {
 							"qosClass": "BestEffort",
 						}),
 					},
-					Errors: []client.Object{},
+					Errors:        []client.Object{},
+					TestRunLabels: labels.Set{},
 				},
 				{
 					Name:  "name-overridden",
@@ -164,12 +171,14 @@ func TestLoadTestSteps(t *testing.T) {
 							"restartPolicy": "Never",
 						}),
 					},
-					Errors: []client.Object{},
+					Errors:        []client.Object{},
+					TestRunLabels: labels.Set{},
 				},
 			},
 		},
 		{
 			"test_data/list-pods",
+			labels.Set{},
 			[]Step{
 				{
 					Name:  "pod",
@@ -217,6 +226,101 @@ func TestLoadTestSteps(t *testing.T) {
 							},
 						},
 					},
+					Errors:        []client.Object{},
+					TestRunLabels: labels.Set{},
+				},
+			},
+		},
+		{
+			"test_data/test-run-labels",
+			labels.Set{},
+			[]Step{
+				{
+					Name:          "",
+					Index:         1,
+					TestRunLabels: labels.Set{},
+					Apply:         []client.Object{},
+					Asserts:       []client.Object{},
+					Errors:        []client.Object{},
+				},
+			},
+		},
+		{
+			"test_data/test-run-labels",
+			labels.Set{"flavor": "a"},
+			[]Step{
+				{
+					Name:          "create-a",
+					Index:         1,
+					TestRunLabels: labels.Set{"flavor": "a"},
+					Apply: []client.Object{
+						&unstructured.Unstructured{
+							Object: map[string]interface{}{
+								"apiVersion": "v1",
+								"kind":       "ConfigMap",
+								"metadata": map[string]interface{}{
+									"name": "test",
+								},
+								"data": map[string]interface{}{
+									"flavor": "a",
+								},
+							},
+						},
+					},
+					Asserts: []client.Object{
+						&unstructured.Unstructured{
+							Object: map[string]interface{}{
+								"apiVersion": "v1",
+								"kind":       "ConfigMap",
+								"metadata": map[string]interface{}{
+									"name": "test",
+								},
+								"data": map[string]interface{}{
+									"flavor": "a",
+								},
+							},
+						},
+					},
+					Errors: []client.Object{},
+				},
+			},
+		},
+		{
+			"test_data/test-run-labels",
+			labels.Set{"flavor": "b"},
+			[]Step{
+				{
+					Name:          "create-b",
+					Index:         1,
+					TestRunLabels: labels.Set{"flavor": "b"},
+					Apply: []client.Object{
+						&unstructured.Unstructured{
+							Object: map[string]interface{}{
+								"apiVersion": "v1",
+								"kind":       "ConfigMap",
+								"metadata": map[string]interface{}{
+									"name": "test",
+								},
+								"data": map[string]interface{}{
+									"flavor": "b",
+								},
+							},
+						},
+					},
+					Asserts: []client.Object{
+						&unstructured.Unstructured{
+							Object: map[string]interface{}{
+								"apiVersion": "v1",
+								"kind":       "ConfigMap",
+								"metadata": map[string]interface{}{
+									"name": "test",
+								},
+								"data": map[string]interface{}{
+									"flavor": "b",
+								},
+							},
+						},
+					},
 					Errors: []client.Object{},
 				},
 			},
@@ -224,8 +328,8 @@ func TestLoadTestSteps(t *testing.T) {
 	} {
 		tt := tt
 
-		t.Run(tt.path, func(t *testing.T) {
-			test := &Case{Dir: tt.path, Logger: testutils.NewTestLogger(t, tt.path)}
+		t.Run(fmt.Sprintf("%s/%s", tt.path, tt.runLabels), func(t *testing.T) {
+			test := &Case{Dir: tt.path, Logger: testutils.NewTestLogger(t, tt.path), RunLabels: tt.runLabels}
 
 			err := test.LoadTestSteps()
 			assert.Nil(t, err)
@@ -238,11 +342,11 @@ func TestLoadTestSteps(t *testing.T) {
 			assert.Equal(t, len(tt.testSteps), len(testStepsVal))
 			for index := range tt.testSteps {
 				tt.testSteps[index].Dir = tt.path
-				assert.Equal(t, tt.testSteps[index].Apply, testStepsVal[index].Apply)
-				assert.Equal(t, tt.testSteps[index].Asserts, testStepsVal[index].Asserts)
-				assert.Equal(t, tt.testSteps[index].Errors, testStepsVal[index].Errors)
-				assert.Equal(t, tt.testSteps[index].Step, testStepsVal[index].Step)
-				assert.Equal(t, tt.testSteps[index].Dir, testStepsVal[index].Dir)
+				assert.Equal(t, tt.testSteps[index].Apply, testStepsVal[index].Apply, "apply objects need to match")
+				assert.Equal(t, tt.testSteps[index].Asserts, testStepsVal[index].Asserts, "assert objects need to match")
+				assert.Equal(t, tt.testSteps[index].Errors, testStepsVal[index].Errors, "error objects need to match")
+				assert.Equal(t, tt.testSteps[index].Step, testStepsVal[index].Step, "step object needs to match")
+				assert.Equal(t, tt.testSteps[index].Dir, testStepsVal[index].Dir, "dir needs to match")
 				assert.Equal(t, tt.testSteps[index], testStepsVal[index])
 			}
 		})
diff --git a/pkg/test/harness.go b/pkg/test/harness.go
index b221a5b..379ae6a 100644
--- a/pkg/test/harness.go
+++ b/pkg/test/harness.go
@@ -18,6 +18,7 @@ import (
 	volumetypes "github.com/docker/docker/api/types/volume"
 	docker "github.com/docker/docker/client"
 	"gopkg.in/yaml.v2"
+	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/client-go/discovery"
 	"k8s.io/client-go/rest"
@@ -53,6 +54,7 @@ type Harness struct {
 	stopping      bool
 	bgProcesses   []*exec.Cmd
 	report        *report.Testsuites
+	RunLabels     labels.Set
 }
 
 // LoadTests loads all of the tests in a given directory.
@@ -85,6 +87,7 @@ func (h *Harness) LoadTests(dir string) ([]*Case, error) {
 			Dir:                filepath.Join(dir, file.Name()),
 			SkipDelete:         h.TestSuite.SkipDelete,
 			Suppress:           h.TestSuite.Suppress,
+			RunLabels:          h.RunLabels,
 		})
 	}
 
diff --git a/pkg/test/step.go b/pkg/test/step.go
index fa3fb55..ba73c9b 100644
--- a/pkg/test/step.go
+++ b/pkg/test/step.go
@@ -11,7 +11,9 @@ import (
 	"time"
 
 	k8serrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/apimachinery/pkg/util/wait"
@@ -36,7 +38,8 @@ type Step struct {
 	Index      int
 	SkipDelete bool
 
-	Dir string
+	Dir           string
+	TestRunLabels labels.Set
 
 	Step   *harness.TestStep
 	Assert *harness.TestAssert
@@ -495,9 +498,9 @@ func (s *Step) String() string {
 //     if seen, mark a test immediately failed.
 //   - All other YAML files are considered resources to create.
 func (s *Step) LoadYAML(file string) error {
-	objects, err := testutils.LoadYAMLFromFile(file)
-	if err != nil {
-		return fmt.Errorf("loading %s: %s", file, err)
+	skipFile, objects, err := s.loadOrSkipFile(file)
+	if skipFile || err != nil {
+		return err
 	}
 
 	if err = s.populateObjectsByFileName(filepath.Base(file), objects); err != nil {
@@ -579,6 +582,38 @@ func (s *Step) LoadYAML(file string) error {
 	return nil
 }
 
+func (s *Step) loadOrSkipFile(file string) (bool, []client.Object, error) {
+	loadedObjects, err := testutils.LoadYAMLFromFile(file)
+	if err != nil {
+		return false, nil, fmt.Errorf("loading %s: %s", file, err)
+	}
+
+	var objects []client.Object
+	shouldSkip := false
+	testFileObjEncountered := false
+
+	for i, object := range loadedObjects {
+		if testFileObject, ok := object.(*harness.TestFile); ok {
+			if testFileObjEncountered {
+				return false, nil, fmt.Errorf("more than one TestFile object encountered in file %q", file)
+			}
+			testFileObjEncountered = true
+			selector, err := metav1.LabelSelectorAsSelector(testFileObject.TestRunSelector)
+			if err != nil {
+				return false, nil, fmt.Errorf("unrecognized test run selector in object %d of %q: %w", i, file, err)
+			}
+			if selector.Empty() || selector.Matches(s.TestRunLabels) {
+				continue
+			}
+			fmt.Printf("Skipping file %q, label selector does not match test run labels.\n", file)
+			shouldSkip = true
+		} else {
+			objects = append(objects, object)
+		}
+	}
+	return shouldSkip, objects, nil
+}
+
 // populateObjectsByFileName populates s.Asserts, s.Errors, and/or s.Apply for files containing
 // "assert", "errors", or no special string, respectively.
 func (s *Step) populateObjectsByFileName(fileName string, objects []client.Object) error {
diff --git a/pkg/test/test_data/test-run-labels/01-assert-a.yaml b/pkg/test/test_data/test-run-labels/01-assert-a.yaml
new file mode 100644
index 0000000..46faf4f
--- /dev/null
+++ b/pkg/test/test_data/test-run-labels/01-assert-a.yaml
@@ -0,0 +1,12 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestFile
+testRunSelector:
+  matchLabels:
+    flavor: a
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: test
+data:
+  flavor: a
diff --git a/pkg/test/test_data/test-run-labels/01-assert-b.yaml b/pkg/test/test_data/test-run-labels/01-assert-b.yaml
new file mode 100644
index 0000000..5d8e7be
--- /dev/null
+++ b/pkg/test/test_data/test-run-labels/01-assert-b.yaml
@@ -0,0 +1,12 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestFile
+testRunSelector:
+  matchLabels:
+    flavor: b
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: test
+data:
+  flavor: b
diff --git a/pkg/test/test_data/test-run-labels/01-create-a.yaml b/pkg/test/test_data/test-run-labels/01-create-a.yaml
new file mode 100644
index 0000000..559453e
--- /dev/null
+++ b/pkg/test/test_data/test-run-labels/01-create-a.yaml
@@ -0,0 +1,12 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestFile
+testRunSelector:
+  matchLabels:
+    flavor: a
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: test
+data:
+  flavor: a
\ No newline at end of file
diff --git a/pkg/test/test_data/test-run-labels/01-create-b.yaml b/pkg/test/test_data/test-run-labels/01-create-b.yaml
new file mode 100644
index 0000000..ece5b5f
--- /dev/null
+++ b/pkg/test/test_data/test-run-labels/01-create-b.yaml
@@ -0,0 +1,12 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestFile
+testRunSelector:
+  matchLabels:
+    flavor: b
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: test
+data:
+  flavor: b
\ No newline at end of file
diff --git a/pkg/test/utils/kubernetes.go b/pkg/test/utils/kubernetes.go
index ebcdcce..b7706de 100644
--- a/pkg/test/utils/kubernetes.go
+++ b/pkg/test/utils/kubernetes.go
@@ -400,6 +400,8 @@ func ConvertUnstructured(in client.Object) (client.Object, error) {
 		return in, nil
 	}
 	switch {
+	case kind == "TestFile":
+		converted = &harness.TestFile{}
 	case kind == "TestStep":
 		converted = &harness.TestStep{}
 	case kind == "TestAssert":