Project: Operator Framework (App Definition and Development / Application Definition & Image Build)
File: scorecard-custom-tests-2.md

title: Scorecard Custom Test Framework authors: "@jemccorm" reviewers: "@joelandford" "@zeus" approvers: "@shurley" "@dmesser" creation-date: 2020-03-19 last-updated: 2020-03-19 status: implementable \[ \] Enhancement is `implementable` \[ \] Design details are appropriately documented from clear requirements \[ \] Test plan is defined \[ \] Graduation criteria for dev preview, tech preview, GA \[ \] User-facing documentation is created This enhancement proposal outlines the SDK scorecards design for running of custom tests, specifically tests that are outside of what Red Hat includes by default within the scorecards tests. The proposed design would provide both a means to write custom tests and also convert the existing scorecard tests (basic and olm) into the custom test format. The scorecard needs a robust means of enabling end-users of the SDK to write and execute their own operator tests. Ideally, the custom test framework would decouple test logic from within the SDK itself, making the SDK and scorecard logic in particular more flexible and maintainable in the long run. Document a design for scorecard to execute end-user or custom tests. Document stories/steps required to deliver scorecard custom test functionality. This is a design proposal and does not include a proof-of-concept. This design proposal is to establish a pattern for implementing custom scorecard tests and also support the goal of migrating existing scorecard tests into the same format as the custom tests. With this proposal, the scorecard will change as follows: scorecard will not start the operator being tested scorecard will not create Custom Resources on behalf of test scorecard will execute tests as Pods based on scorecard configuration scorecard will require a valid bundle image or on-disk bundle to determine what information will be passed to scorecard tests scorecard will determine test configurations by a YAML/DSL file that is either passed on the command line or included in the bundle contents scorecard will only run OLM tests if OLM manifests are found in the bundle scorecard command syntax will look as follows: operator-sdk scorecard --dsl <point-to-yaml> --bundle <bundle-path/image> -l <selector> I am as an Operator developer, I'd like to be able to construct custom tests and execute them via the scorecard. Scorecard will include an example custom test that end users could use as a reference when writing their custom tests. Scorecard internal tests will be converted to the custom test format and therefore be externalized from the scorecard binary itself. Scorecard would process tests based on an on-disk directory or bundle image that contains the scorecard configuration. I am as an Operator developer, would like to be able to create tests using the same Assert syntax adopted by and check its results aggregated within scorecard results. Implementation Notes The Scorecard would support the creation of a Kuttl test image and then execute tests via Kuttl, however, it would be transparent for users since its results will be integrated into scorecard

test image - this is a container image built by end-users to contain a custom test implementation test configuration (DSL) - this is a file that contains the test definitions and labels which determine what tests are run and how they are configured Custom Test Definition - A custom test is any sort of test logic required, written in any programming language that satisfies the scorecard test requirements for output format and execution. A custom test is based on a container image, that image is created by the end-user using whatever tools they want to use. The test image must produce container log output that conforms to the scorecard Test Result output. The output is a single Test Result, so a single run of a test image should execute and output the result for exactly one test. Here is a sample of test output in the required format: ``` { "name": "Spec fields with descriptors", "description": "All spec fields have matching descriptors in the CSV", "state": "fail", "suggestions": [ "Add a spec descriptor for size" ], "crname": "example-memcached" } ``` The test image will be executed as a Pod by the scorecard with a restart policy of `never`. Also, for each test execution, scorecard would also make the bundle image contents available to test images by mounting the bundle image contents into the following location inside the test pod: ``` /scorecard/bundle/ ``` This allows tests to have access to all the bundle contents if they are required for the test logic. The scorecard does not make assumptions about what the test does with this proposed design. The test could for example: not use the CR's at all (e.g. doing a static bundle validation) use the CRs in the CSVs alm-examples annotation (e.g. to verify they validate the associated CRD schema) use some other CRs built (or maybe mounted) into the test image For the case of executing a test, the bundle image will be passed as a scorecard command flag: ``` operator-sdk scorecard --bundle='quay.io/myorg/mytest' --selector=suite=basic ``` Custom tests are expected to produce log output that conforms to the scorecard v1alpha2 Test Result output. This allows scorecard to parse the custom test log output and aggregate the results like any other scorecard test. The scorecard expected output would be in JSON, and would match the following definition: https://github.com/operator-framework/operator-sdk/blob/master/pkg/apis/scorecard/v1alpha2/types.go#L36 The scorecard v1alpha2 Test Result output contains information such as pass or fail status and suggestions/errors produced by the test that the end user would want to see in the aggregated scorecard

The developer of a custom test would have a workflow similar to: write a custom test binary in the language of their choice, sample custom test within the SDK repository would be available as a starting point test the custom test binary locally (out-of-cluster) to make sure the output is produced that matches v1alpha2 ScorecardTestResult format scorecard creates a scorecard DSL config file for their test image when the test binary works as expected, the developer would build and push the test image to a container registry accessible to the scorecard test environment scorecard could build a bundle image that contains the scorecard DSL config NOTE: the Basic and OLM test manifests would be added automatically to the bundle image as part of the bundle image build process. This allows users to run the Basic and OLM tests along with their custom tests. the developer would run the scorecard passing in their operator bundle image they just built along with any other scorecard flags (e.g. selector). The case of a developer implementing a kuttl test would be as follows: developer places kuttl test assets into a kuttl test directory known to scorecard scorecard creates the scorecard DSL config file scorecard runs the tests which are based on a scorecard kuttl test image if the test results meet their goals, then scorecard could build a bundle image that contains the kuttl tests in a distributable format (e.g. bundle image) In this design proposal, custom tests are container images, described in the scorecard DSL yaml file. End users would add their scorecard DSL yaml file into their operator bundle image or pass it into scorecard using a command line flag. Custom test images are stored in a container registry outside the control of scorecard. The scorecard DSL command flag would take precedence over the scorecard DSL config file found in the bundle image if specified. The scorecard DSL yaml file would be accessible from a local disk location to scorecard or from within a bundle image. Scorecard would assume that the scorecard DSL yaml file would be located in the bundle image at the following location: ``` /scorecard/config.yaml ``` The scorecard DSL format is yet to be finalized but it would need to include configuration settings for each test including labels. Here is a sample of what the scorecard DSL configuration would look like: ``` tests: image: quay.io/someuser/customtest1:v0.0.1 description: some ISV test labels: suite: custom test: customtest1 image: quay.io/someuser/customtest2:v0.0.1 description: some ISV test labels: test: customtest2 image: quay.io/operator-framework/scorecard-basictests:v0.0.1 description: redhat - check spec entrypoint: checkspec labels: suite: basic test: basic-check-spec-test image: quay.io/operator-framework/scorecard-basictests:v0.0.1 description: redhat - check status entrypoint: checkstatus labels: suite: basic test: basic-check-status-test image: quay.io/operator-framework/scorecard-olmtests:v0.0.1 description: redhat - CRDs have validation entrypoint: crdvalidation labels: suite: olm test: olm-crds-have-validation-test image: quay.io/operator-framework/scorecard-olmtests:v0.0.1 description: redhat - CRDs have resources entrypoint: crdhasresources labels: suite: olm test: olm-crds-have-resources-test image: quay.io/operator-framework/scorecard-olmtests:v0.0.1 description: redhat - spec descriptors entrypoint: crdspecdescriptors labels: suite: olm test: olm-spec-descriptors-test image: quay.io/operator-framework/scorecard-olmtests:v0.0.1 description: redhat - status descriptors entrypoint: crdstatusdescriptors labels: suite: olm test: olm-status-descriptors-test image: quay.io/operator-framework/scorecard-olmtests:v0.0.1 description: redhat - bundle validation entrypoint: bundlevalidation labels: suite: olm test: olm-bundle-validation-test image: quay.io/operator-framework/scorecard-kuttltests:v0.0.1 description: redhat - kuttl tests labels: suite: kuttl ``` Users could override the default test images shipped with scorecard by adding scorecard command flags as follows: ``` operator-sdk scorecard --olm-image=quay.io/operator-framework/scorecard-olmtests:dev operator-sdk scorecard --basic-image=quay.io/operator-framework/scorecard-basictests:dev operator-sdk scorecard --kuttl-image=quay.io/operator-framework/scorecard-kuttltests:dev ``` Scorecard tests would be executed from the SDK scorecard CLI as is the case

Scorecard would execute the tests, capture the results, and provide feedback to the end-user by means of the current scorecard API (e.g. ScorecardTestResult). Command line flags would allow the end-user a means to specify which tests are executed and upon which test environment to run the tests. For example: ``` operator-sdk scorecard -o text --selector=suite=isvcustom --bundle='some/bundle/image:v0.0.1' ``` Running the above command would have scorecard perform the following: fetch the scorecard DSL yaml configuration from the bundle image select the tests to run based on the scorecard DSL configuration and the selector iterate through each test, construct a test Pod create the test Pod when the test pod completes, fetch the test image Pod log output for aggregation of test results display the test results Scorecard would allow you to run tests on a local Kube cluster as is the case today. Provisioning the test cluster is external to the scope of the scorecard itself. Note, that with this proposal, scorecard would not be responsible for installing OLM or depend on OLM being installed in order to run. With the above custom test capability, there is benefit to Red Hat in migrating the existing scorecard internal tests to be based on the custom test format. Today there are 2 internal scorecard tests that make use of the scorecard proxy, with these tests migrating to the custom test format, the scorecard proxy is likely not necessary which would also reduce the overall scorecard implementation complexity. The changes to scorecard are large and would require a migration off of the current scorecard. For custom tests, it might be possible to write the custom test image using ansible's callback plugin framework to produce container image log output in the scorecard required format (e.g. v1alpha2). The complexity of the test might dictate whether this is viable or not, the fallback would be for the custom test to be developed in python, Java, or golang as discussed in this proposal. Writing a custom test image in a Helm chart was not evaluated or pursued in this proposal. What is likely for a Helm operator is that the operator owner would need to write custom test images using either Python, Java, golang, or some other programming language. The alternatives to this design proposal might include: Other existing/open-source test frameworks could be evaluated for use within the SDK for implementing the desired custom test functionality. Other test frameworks within Redhat being planned or within use could be evaluated. The proposed changes to the scorecard solve the immediate need to support custom or user-provided tests, both simple and complex tests. However, the longer term implication for scorecard is that all of its tests would evolve to use a single testing format that is far more flexible that what exists today in the current implementation. The proposed design focuses heavily on separation of concerns, turning scorecard into a test runner essentially, moving test implementations into their own concern (eg. container images). () ()