Repository: kyverno/chainsaw
Pull request #401 (2023-11-09 06:09:13)

feat: Custom Test File name
Fix : https://github.com/kyverno/chainsaw/issues/397

shubham-cmyk: @eddycharly PTAL
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/kyverno/chainsaw/pull/401?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kyverno) Report
> Merging [#401](https://app.codecov.io/gh/kyverno/chainsaw/pull/401?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kyverno) (63ffde6) into [main](https://app.codecov.io/gh/kyverno/chainsaw/commit/657c4eb72976bfdcc2a5241791b7edc86705e9ef?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kyverno) (657c4eb) will **increase** coverage by `0.40%`.
> Report is 3 commits behind head on main.
> The diff coverage is `100.00%`.

```diff
@@            Coverage Diff             @@
##             main     #401      +/-   ##
==========================================
+ Coverage   86.65%   87.05%   +0.40%     
==========================================
  Files          49       48       -1     
  Lines        1656     1607      -49     
==========================================
- Hits         1435     1399      -36     
+ Misses        173      160      -13     
  Partials       48       48              
```


| [Files](https://app.codecov.io/gh/kyverno/chainsaw/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kyverno) | Coverage Δ | |
|---|---|---|
| [pkg/commands/test/command.go](https://app.codecov.io/gh/kyverno/chainsaw/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kyverno#diff-cGtnL2NvbW1hbmRzL3Rlc3QvY29tbWFuZC5nbw==) | `83.06% <100.00%> (+0.47%)` | :arrow_up: |

... and [4 files with indirect coverage changes](https://app.codecov.io/gh/kyverno/chainsaw/pull/401/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kyverno)

:mega: Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or [Firefox](https://addons.mozilla.org/en-US/firefox/addon/codecov/) today!


Patch:
diff --git a/.crds/chainsaw.kyverno.io_configurations.yaml b/.crds/chainsaw.kyverno.io_configurations.yaml
index 6ab06c92..6a28b276 100644
--- a/.crds/chainsaw.kyverno.io_configurations.yaml
+++ b/.crds/chainsaw.kyverno.io_configurations.yaml
@@ -89,6 +89,11 @@ spec:
                 items:
                   type: string
                 type: array
+              testFile:
+                default: chainsaw-test.yaml
+                description: TestFile is the name of the file containing the test
+                  to run.
+                type: string
               timeouts:
                 description: Global timeouts configuration. Applies to all tests/test
                   steps if not overridden.
diff --git a/pkg/apis/v1alpha1/configuration_spec.go b/pkg/apis/v1alpha1/configuration_spec.go
index f80bcb49..c83ee6c6 100644
--- a/pkg/apis/v1alpha1/configuration_spec.go
+++ b/pkg/apis/v1alpha1/configuration_spec.go
@@ -66,4 +66,9 @@ type ConfigurationSpec struct {
 	// +kubebuilder:validation:Minimum:=1
 	// +optional
 	RepeatCount *int `json:"repeatCount,omitempty"`
+
+	// TestFile is the name of the file containing the test to run.
+	// +kubebuilder:default:="chainsaw-test.yaml"
+	// +optional
+	TestFile string `json:"testFile,omitempty"`
 }
diff --git a/pkg/data/crds/chainsaw.kyverno.io_configurations.yaml b/pkg/data/crds/chainsaw.kyverno.io_configurations.yaml
index 6ab06c92..6a28b276 100644
--- a/pkg/data/crds/chainsaw.kyverno.io_configurations.yaml
+++ b/pkg/data/crds/chainsaw.kyverno.io_configurations.yaml
@@ -89,6 +89,11 @@ spec:
                 items:
                   type: string
                 type: array
+              testFile:
+                default: chainsaw-test.yaml
+                description: TestFile is the name of the file containing the test
+                  to run.
+                type: string
               timeouts:
                 description: Global timeouts configuration. Applies to all tests/test
                   steps if not overridden.
diff --git a/website/docs/apis/chainsaw.v1alpha1.md b/website/docs/apis/chainsaw.v1alpha1.md
index 77aa2e62..8abfe366 100644
--- a/website/docs/apis/chainsaw.v1alpha1.md
+++ b/website/docs/apis/chainsaw.v1alpha1.md
@@ -135,6 +135,7 @@ during the testing process.</p>
 | `excludeTestRegex` | `string` |  |  | <p>ExcludeTestRegex is used to exclude tests based on a regular expression.</p> |
 | `includeTestRegex` | `string` |  |  | <p>IncludeTestRegex is used to include tests based on a regular expression.</p> |
 | `repeatCount` | `int` |  |  | <p>RepeatCount indicates how many times the tests should be executed.</p> |
+| `testFile` | `string` |  |  | <p>TestFile is the name of the file containing the test to run.</p> |
 
 ## `Create`     {#chainsaw-kyverno-io-v1alpha1-Create}
 


Test patch:
diff --git a/pkg/commands/test/command.go b/pkg/commands/test/command.go
index 304a4e67..8b8bae1f 100644
--- a/pkg/commands/test/command.go
+++ b/pkg/commands/test/command.go
@@ -24,6 +24,7 @@ import (
 
 type options struct {
 	config              string
+	testFile            string
 	applyTimeout        metav1.Duration
 	assertTimeout       metav1.Duration
 	errorTimeout        metav1.Duration
@@ -86,6 +87,9 @@ func Command() *cobra.Command {
 			}
 			// flags take precedence over configuration file
 			flags := cmd.Flags()
+			if flagutils.IsSet(flags, "test-file") {
+				configuration.Spec.TestFile = options.testFile
+			}
 			if flagutils.IsSet(flags, "apply-timeout") {
 				configuration.Spec.Timeouts.Apply = &options.applyTimeout
 			}
@@ -137,6 +141,7 @@ func Command() *cobra.Command {
 			if flagutils.IsSet(flags, "exclude-test-regex") {
 				configuration.Spec.ExcludeTestRegex = options.excludeTestRegex
 			}
+			fmt.Fprintf(out, "- Using test file: %s\n", configuration.Spec.TestFile)
 			fmt.Fprintf(out, "- TestDirs %v\n", configuration.Spec.TestDirs)
 			fmt.Fprintf(out, "- SkipDelete %v\n", configuration.Spec.SkipDelete)
 			fmt.Fprintf(out, "- FailFast %v\n", configuration.Spec.FailFast)
@@ -172,7 +177,7 @@ func Command() *cobra.Command {
 			}
 			// loading tests
 			fmt.Fprintln(out, "Loading tests...")
-			tests, err := discovery.DiscoverTests("chainsaw-test.yaml", configuration.Spec.TestDirs...)
+			tests, err := discovery.DiscoverTests(configuration.Spec.TestFile, configuration.Spec.TestDirs...)
 			if err != nil {
 				return err
 			}
@@ -209,6 +214,7 @@ func Command() *cobra.Command {
 			return err
 		},
 	}
+	cmd.Flags().StringVar(&options.testFile, "test-file", "chainsaw-test.yaml", "Name of the test file.")
 	cmd.Flags().DurationVar(&options.applyTimeout.Duration, "apply-timeout", timeout.DefaultApplyTimeout, "The apply timeout to use as default for configuration.")
 	cmd.Flags().DurationVar(&options.assertTimeout.Duration, "assert-timeout", timeout.DefaultAssertTimeout, "The assert timeout to use as default for configuration.")
 	cmd.Flags().DurationVar(&options.errorTimeout.Duration, "error-timeout", timeout.DefaultErrorTimeout, "The error timeout to use as default for configuration.")
diff --git a/pkg/commands/test/command_test.go b/pkg/commands/test/command_test.go
index 7b2438e6..00c69e08 100644
--- a/pkg/commands/test/command_test.go
+++ b/pkg/commands/test/command_test.go
@@ -113,6 +113,7 @@ func TestChainsawCommand(t *testing.T) {
 	}, {
 		name: "all flags",
 		args: []string{
+			"--test-file=custom-test.yaml",
 			"--apply-timeout=100s",
 			"--assert-timeout=100s",
 			"--error-timeout=100s",
diff --git a/pkg/config/load_test.go b/pkg/config/load_test.go
index cd2089c1..6e16418c 100644
--- a/pkg/config/load_test.go
+++ b/pkg/config/load_test.go
@@ -48,6 +48,7 @@ func TestLoad(t *testing.T) {
 				Name: "default",
 			},
 			Spec: v1alpha1.ConfigurationSpec{
+				TestFile:         "chainsaw-test.yaml",
 				SkipDelete:       false,
 				FailFast:         false,
 				ReportFormat:     "",
@@ -77,6 +78,7 @@ func TestLoad(t *testing.T) {
 					Cleanup: &metav1.Duration{Duration: 5 * time.Second},
 					Exec:    &metav1.Duration{Duration: 10 * time.Second},
 				},
+				TestFile:         "custom-test.yaml",
 				SkipDelete:       true,
 				FailFast:         true,
 				Parallel:         ptr.To(4),
diff --git a/testdata/commands/test/all_flags.txt b/testdata/commands/test/all_flags.txt
index bce2868c..fae6000a 100644
--- a/testdata/commands/test/all_flags.txt
+++ b/testdata/commands/test/all_flags.txt
@@ -1,4 +1,5 @@
 Loading default configuration...
+- Using test file: custom-test.yaml
 - TestDirs [.]
 - SkipDelete false
 - FailFast false
diff --git a/testdata/commands/test/config_all_fields.txt b/testdata/commands/test/config_all_fields.txt
index 5ded8bd9..e3734baa 100644
--- a/testdata/commands/test/config_all_fields.txt
+++ b/testdata/commands/test/config_all_fields.txt
@@ -1,4 +1,5 @@
 Loading config (../../../testdata/commands/test/config/config_all_fields.yaml)...
+- Using test file: chainsaw-test.yaml
 - TestDirs [/path/to/testdir1 /path/to/testdir2]
 - SkipDelete true
 - FailFast true
diff --git a/testdata/commands/test/default.txt b/testdata/commands/test/default.txt
index 30b833ba..1bea6f2a 100644
--- a/testdata/commands/test/default.txt
+++ b/testdata/commands/test/default.txt
@@ -1,4 +1,5 @@
 Loading default configuration...
+- Using test file: chainsaw-test.yaml
 - TestDirs []
 - SkipDelete false
 - FailFast false
diff --git a/testdata/commands/test/with_regex.txt b/testdata/commands/test/with_regex.txt
index c1e0bbaa..bf294f60 100644
--- a/testdata/commands/test/with_regex.txt
+++ b/testdata/commands/test/with_regex.txt
@@ -1,4 +1,5 @@
 Loading default configuration...
+- Using test file: chainsaw-test.yaml
 - TestDirs []
 - SkipDelete false
 - FailFast false
diff --git a/testdata/commands/test/with_repeat_count.txt b/testdata/commands/test/with_repeat_count.txt
index 2d083149..ee21269b 100644
--- a/testdata/commands/test/with_repeat_count.txt
+++ b/testdata/commands/test/with_repeat_count.txt
@@ -1,4 +1,5 @@
 Loading default configuration...
+- Using test file: chainsaw-test.yaml
 - TestDirs []
 - SkipDelete false
 - FailFast false
diff --git a/testdata/commands/test/with_suppress.txt b/testdata/commands/test/with_suppress.txt
index b4d40a02..4ace357b 100644
--- a/testdata/commands/test/with_suppress.txt
+++ b/testdata/commands/test/with_suppress.txt
@@ -1,4 +1,5 @@
 Loading default configuration...
+- Using test file: chainsaw-test.yaml
 - Timeout 30s
 - TestDirs []
 - SkipDelete false
diff --git a/testdata/commands/test/with_test_dirs.txt b/testdata/commands/test/with_test_dirs.txt
index 05412f6f..74540994 100644
--- a/testdata/commands/test/with_test_dirs.txt
+++ b/testdata/commands/test/with_test_dirs.txt
@@ -1,4 +1,5 @@
 Loading default configuration...
+- Using test file: chainsaw-test.yaml
 - TestDirs [dir1,dir2,dir3]
 - SkipDelete false
 - FailFast false
diff --git a/testdata/commands/test/with_timeout.txt b/testdata/commands/test/with_timeout.txt
index e424b0eb..878de64b 100644
--- a/testdata/commands/test/with_timeout.txt
+++ b/testdata/commands/test/with_timeout.txt
@@ -1,4 +1,5 @@
 Loading default configuration...
+- Using test file: chainsaw-test.yaml
 - TestDirs []
 - SkipDelete false
 - FailFast false
diff --git a/testdata/config/custom-config.yaml b/testdata/config/custom-config.yaml
index 20f74093..d345afab 100644
--- a/testdata/config/custom-config.yaml
+++ b/testdata/config/custom-config.yaml
@@ -11,6 +11,7 @@ spec:
     cleanup: 5s
     exec: 10s
   skipDelete: true
+  testFile: custom-test.yaml
   failFast: true
   parallel: 4
   reportFormat: "JSON"
diff --git a/website/docs/commands/chainsaw_test.md b/website/docs/commands/chainsaw_test.md
index fea0664a..23e0d9e0 100644
--- a/website/docs/commands/chainsaw_test.md
+++ b/website/docs/commands/chainsaw_test.md
@@ -48,6 +48,7 @@ chainsaw test [flags]... [test directories]...
       --report-name string                  The name of the report to create. (default "chainsaw-report")
       --skip-delete                         If set, do not delete the resources after running the tests.
       --test-dir stringArray                Directories containing test cases to run.
+      --test-file string                    Name of the test file. (default "chainsaw-test.yaml")
 ```
 
 ### SEE ALSO