Project: Operator Framework (App Definition and Development / Application Definition & Image Build)
File: custom-tests.md

title: Writing Custom Scorecard Tests weight: 50 This guide outlines the steps which can be followed to extend the existing scorecard tests and implement operator specific custom tests. The following steps explain creating of a custom test image which can be used with Scorecard to run operator specific tests. As an example, let us start by creating a sample go repository containing the test bundle data, custom scorecard tests and a Makefile to help us build a test image. The sample test image repository present has the following project structure: ``` $ tree . . ... config | ... scorecard bases config.yaml kustomization.yaml patches basic.config.yaml olm.config.yaml Makefile bundle manifests cache.example.commemcachedcrd.yaml memcached-operator.clusterserviceversion.yaml metadata annotations.yaml tests scorecard config.yaml go.mod go.sum images custom-scorecard-tests Dockerfile bin entrypoint user_setup cmd test main.go custom-scorecard-tests internal tests tests.go ``` `config/scorecard` - Contains a kustomization for generating a config from a base and set of overlays. `bundle/` - Contains bundle manifests and metadata under test. `bundle/tests/scorecard/config.yaml` - Configuration file generated by `make bundle` from the `config/scorecard` kustomization. `images/custom-scorecard-tests/main.go` - Scorecard test binary. `internal/tests/tests.go` - Contains the implementation of custom tests specific to the operator. Scorecard currently implements a few and tests for the image bundle, custom resources and custom resource definitions. Additional tests specific to the operator can also be included in the test suite of scorecard. The `tests.go` file is where the custom tests are implemented in the sample test image project. These tests use `scapiv1alpha3.TestResult` struct to populate the result, which is then converted to json format for the output. For example, the format of a simple custom sample test can be as follows: ```Go package tests import ( apimanifests "github.com/operator-framework/api/pkg/manifests" scapiv1alpha3 "github.com/operator-framework/api/pkg/apis/scorecard/v1alpha3" ) const ( CustomTest1Name = "customtest1" ) // CustomTest1 func CustomTest1(bundle *apimanifests.Bundle) scapiv1alpha3.TestStatus { r := scapiv1alpha3.TestResult{} r.Name = CustomTest1Name r.State = scapiv1alpha3.PassState r.Errors = make([]string, 0) r.Suggestions = make([]string, 0) // Implement relevant custom test logic here return wrapResult(r) } ``` The includes test definitions and metadata to run the test. This file is constructed using a kustomization under `config/scorecard`, with overlays for test sets. For the example `CustomTest1` function, add the following to `config/scorecard/patches/customtest1.config.yaml`. ```yaml op: add path: /stages/0/tests/- value: image: quay.io/<username>/custom-scorecard-tests:latest entrypoint: custom-scorecard-tests customtest1 labels: suite: custom test: customtest1 ``` The important fields to note here are: `image` - name and tag of the test image which was specified in the Makefile. `labels` - the name of the `test` and `suite` the test function belongs to. This can be specified in the `operator-sdk scorecard` command to run the desired test. Next, add a to your `config/scorecard/kustomization.yaml`: ```yaml patchesJson6902: ... path: patches/customtest1.config.yaml target: group: scorecard.operatorframework.io version: v1alpha3 kind: Configuration name: config ``` Once you run `make bundle`, the `bundle/tests/scorecard/config.yaml` will be (re)generated with your custom

If generating a config file outside of the on-disk bundle, you can run: ```console $ kustomize build config/scorecard > path/to/config.yaml ``` Note: The default location of `config.yaml` inside the bundle is `<bundle directory>/tests/scorecard/config.yaml`. It can be overridden using the `--config` flag. For more details regarding the configuration file refer to . The scorecard test image implementation requires the bundle under test to be present in the test image. The `apimanifests.GetBundleFromDir()` function reads the pod's bundle to fetch the manifests and scorecard configuration from desired path. ```Go cfg, err := apimanifests.GetBundleFromDir(scorecard.PodBundleRoot) if err != nil { log.Fatal(err.Error()) } ``` The scorecard binary uses `config.yaml` file to locate tests and execute the them as Pods which scorecard creates. Custom test images are included into Pods that scorecard creates, passing in the bundle contents on a shared mount point to the test image container. The specific custom test that is executed is driven by the config.yaml's entry-point command and arguments. An example custom scorecard test implementation is present . The names with which the tests are identified in `config.yaml` and would be passed in the `scorecard` command, are to be specified here. ```Go ... switch entrypoint[0] { case tests.CustomTest1Name: result = tests.CustomTest1(cfg) ... } ... ``` The result of the custom tests which is in `scapiv1alpha3.TestResult` format, is converted to json for output. ```Go prettyJSON, err := json.MarshalIndent(result, "", " ") if err != nil { log.Fatal("Failed to generate json", err) } fmt.Printf("%s\n", string(prettyJSON)) ``` The names of the custom tests are also included in `printValidTests()` function: ```Go func printValidTests() (result scapiv1alpha3.TestStatus) { ... str := fmt.Sprintf("Valid tests for this image include: %s", tests.CustomTest1Name) result.Errors = append(result.Errors, str) ... } ``` The SDK project makefile contains targets to build the sample custom test image. The current makefile is found . You can use this makefile as a reference for your own custom test image makefile. To build the sample custom test image, run: ``` make image/custom-scorecard-tests ``` The `operator-sdk scorecard` command is used to execute the scorecard tests by specifying the location of test bundle in the command. The name or suite of the tests which are to be executed can be specified with the `--selector` flag. The command will create scorecard pods with the image specified in `config.yaml` for the respective test. For example, the `CustomTest1Name` test provides the following json output. ```console $ operator-sdk scorecard <bundlediror_image> --selector=suite=custom -o json --wait-time=32s --skip-cleanup=false { "kind": "TestList", "apiVersion": "scorecard.operatorframework.io/v1alpha3", "items": [ { "kind": "Test", "apiVersion": "scorecard.operatorframework.io/v1alpha3", "spec": { "image":

"entrypoint": [ "custom-scorecard-tests", "customtest1" ], "labels": { "suite": "custom", "test": "customtest1" } }, "status": { "results": [ { "name": "customtest1", "log": "an ISV custom test", "state": "pass" } ] } } ] } ``` Note: More details on the usage of `operator-sdk scorecard` command and its flags can be found in the The `--skip-cleanup` flag can be used when executing the `operator-sdk scorecard` command to cause the scorecard created test pods to be unremoved. This is useful when debugging or writing new tests so that you can view the test logs or the pod manifests. The `--test-output` flag can be used when executing the `operator-sdk scorecard` command with a config specifying to store the output of the scorecard tests in a specific directory. Any persistent volume data will be stored in the specified local directory upon completion of the scorecard tests. ```console $ operator-sdk scorecard ./bundle --test-output=/mytestoutput ``` Note: By default, the gathered test output will be stored in `$(pwd)/test-output`. The following options are useful to prevent downloading the images from external registries during scorecard job execution. That could be a case of disconnected environments or to prevent an impact of the external registry's pull limits. The `--storage-image` flag can be used when executing the `operator-sdk scorecard` command to overwrite the default `busybox` image used by the Scorecard pod. The `--untar-image` flag can be used when executing the `operator-sdk scorecard` command to overwrite the default untar image used by the Scorecard pod. The scorecard inserts an `initContainer` into the test pods it creates. The `initContainer` serves the purpose of uncompressing the operator bundle contents, mounting them into a shared mount point accessible by test images. The operator bundle contents are stored within a ConfigMap, uniquely built for each scorecard test execution. Upon scorecard completion, the ConfigMap is removed as part of normal cleanup, along with the test pods created by scorecard. Scorecard does not deploy service accounts, RBAC resources, or namespaces for your test but instead considers these resources to be outside its scope. You can however specify whichever service account your tests require, like `config/rbac/service_account.yaml` in Go operator projects, and then specify that service account from the command line: ```console $ operator-sdk scorecard <bundlediror_image> --service-account=my-project-controller-manager ``` Also, you can specify a non-default namespace that scorecard will run in: ```console $ operator-sdk scorecard <bundlediror_image> --namespace=my-project-system ``` If you do not specify either of these flags, the default namespace and service account will be used by the scorecard to run test pods. Some custom tests might require or be better implemented to return more than a single test result. For this case, scorecard's output API allows to be defined for a single test. Within your custom tests you might require connecting to the Kube API. In golang, you could use the API for example to check Kube resources within your tests, or even create custom resources. Your custom test image is being executed within a Pod, so you can use an in-cluster connection to invoke the Kube API. <!-- TODO: this file shouldn't refer to the top-level operator-sdk repo as a reference, but a sample (in testdata?) -->