Repository: redhat-certification/chart-verifier
Pull request #106 (2021-05-14 12:42:14)

chart-verifier output contains more than the report
The chart-verifier output since the introduction of the chart-testing
check includes `helm test` and other command output that aren't
expected.

The output should contain only the report's YAML (or JSON)
representation.


Patch:
diff --git a/docs/helm-chart-checks.md b/docs/helm-chart-checks.md
index ca7aeb5..0ebfc2c 100644
--- a/docs/helm-chart-checks.md
+++ b/docs/helm-chart-checks.md
@@ -3,10 +3,11 @@
 Helm chart checks are a set of checks against which the Red Hat Helm chart-verifier tool verifies and validates whether a Helm chart is qualified for a certification from Red Hat. These checks contain metadata that have certain parameters and values with which a Helm chart must comply to be certified by Red Hat. A Red Hat-certified Helm chart qualifies in terms of readiness for distribution in the [Red Hat OpenShift Helm Chart Repository](https://github.com/openshift-helm-charts).
 
 ## Key features
-- You can execute the checks individually. Individual checks can be included or excluded during the verification process using the command line options.
+- You can execute all of the mandatory checks.
+  - Include or exclude individual checks during the verification process by using the command line options.
 - Each check is independent, and there is no specific execution order.
-- Executing the checks includes running the `helm lint` and `helm template` commands against the chart for Red Hat image certification.
-- The checks are configurable. For example, if a chart requires additional values to be compliant with the checks, configure the values using the options available. The options are similar to those used by the `helm lint` and `helm template` commands.
+- Executing the checks includes running the `helm lint`, `helm template`, `helm install`, and `helm test` commands against the chart for Red Hat image certification.
+- The checks are configurable. For example, if a chart requires additional values to be compliant with the checks, configure the values using the available options. The options are similar to those used by the `helm lint` and `helm template` commands.
 - When there are no error messages, the `helm-lint` check passes the verification and is successful. Messages such as `Warning` and `info` do not cause the check to fail.
 
 ## Types of Helm chart checks
@@ -17,13 +18,13 @@ Helm chart checks are categorized into the following types:
 | Check type | Description
 |---|---
 | Mandatory | Checks are required to pass and be successful for certification.
-| Recommended | Checks are about to become mandatory; we recommend fixing the check failures, if any.
+| Recommended | Checks are about to become mandatory; we recommend fixing any check failures.
 | Optional | Checks are ready for customer testing. Checks can fail and still pass the verification for certification.
 | Experimental | New checks introduced for testing purposes or beta versions.
 > **_NOTE:_**  The current release of Helm charts verify only the mandatory type of checks.
 
 ## Default set of checks for a Helm chart
-The following table lists the default set of checks with details including the names of the checks, their type, and description.These checks must be successful for a Helm chart to be certified by Red Hat.
+The following table lists the default set of checks with details including the names of the checks, their type, and description. These checks must be successful for a Helm chart to be certified by Red Hat.
 
 #### Table 2: Helm chart default checks
 
@@ -38,77 +39,77 @@ The following table lists the default set of checks with details including the n
 | `not-contain-csi-objects` | Mandatory | Checks that the Helm chart does not include Container Storage Interface (CSI)[¹](https://gist.github.com/Srivaralakshmi/eb3c9bf1d65ec297035f4a8d26057620#-for-more-information-on-csi-see-container-storage-interface) objects.
 | `images-are-certified` | Mandatory | Checks that the images referenced by the Helm chart are Red Hat-certified.
 | `helm-lint` | Mandatory | Checks that the chart is well formed by running the `helm lint` command.
+| `chart-testing` | Mandatory | Installs the chart and verifies it on a Red Hat OpenShift Container Platform cluster.
 | `contains-values` | Mandatory | Checks that the Helm chart contains the `values`[²](https://gist.github.com/Srivaralakshmi/eb3c9bf1d65ec297035f4a8d26057620#-for-more-information-on-the-values-file-see-values-and-best-practices-for-using-values) file.
 
 ###### ¹ For more information on CSI, see [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md).
 
 ###### ² For more information on the `values` file, see [`values`](https://helm.sh/docs/chart_template_guide/values_files/) and [Best Practices for using values](https://helm.sh/docs/chart_best_practices/values/).
 
-## Using the docker command for Helm chart checks
-This section provides help on the basic usage of Helm chart checks with the docker command.
+## Using the podman or docker command for Helm chart checks
+This section provides help on the basic usage of Helm chart checks with the podman or docker command.
 
 ### Prerequisites
-- A container engine and the `docker` command installed.
+- A container engine and the Podman or Docker CLI installed.
 - Internet connection to check that the images are Red Hat certified.
-- itHub profile to submit the chart to the [Red Hat OpenShift Helm Charts Repository](https://github.com/openshift-helm-charts).
+- GitHub profile to submit the chart to the [Red Hat OpenShift Helm Charts Repository](https://github.com/openshift-helm-charts).
+- Red Hat OpenShift Container Platform cluster.
 
 ### Procedure
-To perform the tasks related to Helm chart checks, run the following commands:
-
-- Runs all the available checks for the chart using a `uri`.
-
- ```ruby
- $ docker run -it --rm quay.io/redhat-certification/chart-verifier verify <chart-uri>
- ```
-- Runs all the checks available locally on your system for the chart, from the same directory as the chart.
-
- ```ruby
- $ docker run -v $(pwd):/charts --rm quay.io/redhat-certification/chart-verifier verify /charts/<chart>
- ```
-- Gets the list of options for the `verify` command.
-
- ```ruby
- $ docker run -it --rm quay.io/redhat-certification/chart-verifier verify help
- ```
- The output is similar to the following example:
- ```ruby
- $ docker run -it --rm quay.io/redhat-certification/chart-verifier verify help`
-
- Verifies a Helm chart by checking some of its characteristics
-
- Usage:
- chart-verifier verify <chart-uri> [flags]
-
- Flags:
-   -S, --chart-set strings          set values for the chart (can specify multiple or separate values with commas:     key1=val1,key2=val2)
-   -F, --chart-set-file strings     set values from respective files specified via the command line (can specify multiple or  separate values with commas: key1=path1,key2=path2)
-   -X, --chart-set-string strings   set STRING values for the chart (can specify multiple or separate values with commas: key1=val1,key2=val2)
-   -f, --chart-values strings       specify values in a YAML file or a URL (can specify multiple)
-   -x, --disable strings            all checks will be enabled except the informed ones
-   -e, --enable strings             only the informed checks will be enabled
-   -h, --help                       help for verify
-   -o, --output string              the output format: default, json or yaml
-   -s, --set strings                overrides a configuration, e.g: dummy.ok=false
-
- Global Flags:
-       --config string   config file (default is $HOME/.chart-verifier.yaml)
-- Runs a subset of the checks.
-
- ```ruby
- $ docker run -it --rm quay.io/redhat-certification/chart-verifier verify -e images-are-certified,helm-lint
- ```
-- Runs all the checks except a subset.
-
- ```ruby
- $ docker run -it --rm quay.io/redhat-certification/chart-verifier verify -x images-are-certified,helm-lint
- ```
-- Provides chart-override values.
-
- ```ruby
- $ docker run -it --rm quay.io/redhat-certification/chart-verifier verify -S default.port=8080 images-are-certified,helm-lint
- ```
-- Provides chart-override values in a file.
-
- ```ruby
- $ docker run -it --rm quay.io/redhat-certification/chart-verifier verify -F overrides.yaml images-are-certified,helm-lint
- ```
+
+- Run all the available checks for the chart using a `uri`:
+
+  ```
+  $ podman run -it --rm quay.io/redhat-certification/chart-verifier verify <chart-uri>
+  ```
+- Run all the checks available locally on your system for the chart, from the same directory as the chart:
+
+  ```
+  $ podman run -v $(pwd):/charts --rm quay.io/redhat-certification/chart-verifier verify /charts/<chart>
+  ```
+- Get the list of options for the `verify` command:
+
+  ```
+  $ podman run -it --rm quay.io/redhat-certification/chart-verifier verify help
+  ```
+  The output is similar to the following example:
+  ```
+  Verifies a Helm chart by checking some of its characteristics
+
+  Usage:
+    chart-verifier verify <chart-uri> [flags]
+
+  Flags:
+     -S, --chart-set strings          set values for the chart (can specify multiple or separate values with commas: key1=val1,key2=val2)
+     -F, --chart-set-file strings     set values from respective files specified via the command line (can specify multiple or separate values with commas: key1=path1,key2=path2)
+     -X, --chart-set-string strings   set STRING values for the chart (can specify multiple or separate values with commas: key1=val1,key2=val2)
+     -f, --chart-values strings       specify values in a YAML file or a URL (can specify multiple)
+     -x, --disable strings            all checks will be enabled except the informed ones
+     -e, --enable strings             only the informed checks will be enabled
+     -h, --help                       help for verify
+     -o, --output string              the output format: default, json or yaml
+     -s, --set strings                overrides a configuration, e.g: dummy.ok=false
+
+   Global Flags:
+         --config string   config file (default is $HOME/.chart-verifier.yaml)
+  ```
+- Run a subset of the checks:
+
+  ```
+  $ podman run -it --rm quay.io/redhat-certification/chart-verifier verify -e images-are-certified,helm-lint
+  ```
+- Run all the checks except a subset:
+
+  ```
+  $ podman run -it --rm quay.io/redhat-certification/chart-verifier verify -x images-are-certified,helm-lint
+  ```
+- Provide chart-override values:
+
+  ```
+  $ podman run -it --rm quay.io/redhat-certification/chart-verifier verify -S default.port=8080 images-are-certified,helm-lint
+  ```
+- Provide chart-override values in a file:
+
+  ```
+  $ podman run -it --rm quay.io/redhat-certification/chart-verifier verify -F overrides.yaml images-are-certified,helm-lint
+  ```
diff --git a/pkg/tool/helm.go b/pkg/tool/helm.go
new file mode 100644
index 0000000..bf0c3a7
--- /dev/null
+++ b/pkg/tool/helm.go
@@ -0,0 +1,46 @@
+package tool
+
+import (
+	"github.com/helm/chart-testing/v3/pkg/exec"
+	"github.com/helm/chart-testing/v3/pkg/tool"
+)
+
+// Helm is an interface to the helm binary; it is a thin layer on top of the Helm abstraction offered by chart-testing
+// to silence output being streamed to Stdout.
+type Helm struct {
+	tool.Helm
+	exec.ProcessExecutor
+	extraArgs []string
+}
+
+func NewHelm(exec exec.ProcessExecutor, extraArgs []string) Helm {
+	return Helm{
+		tool.NewHelm(exec, extraArgs),
+		exec,
+		extraArgs,
+	}
+}
+
+func (h Helm) InstallWithValues(chart string, valuesFile string, namespace string, release string) error {
+	var values []string
+	if valuesFile != "" {
+		values = []string{"--values", valuesFile}
+	}
+
+	if _, err := h.RunProcessAndCaptureOutput("helm", "install", release, chart, "--namespace", namespace,
+		"--wait", values, h.extraArgs); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (h Helm) Test(namespace string, release string) error {
+	_, err := h.RunProcessAndCaptureOutput("helm", "test", release, "--namespace", namespace, h.extraArgs)
+	return err
+}
+
+func (h Helm) DeleteRelease(namespace string, release string) {
+	_, _ = h.RunProcessAndCaptureOutput("helm", "uninstall", release, "--namespace", namespace, h.extraArgs)
+
+}
diff --git a/pkg/tool/kubectl.go b/pkg/tool/kubectl.go
new file mode 100644
index 0000000..841c7ea
--- /dev/null
+++ b/pkg/tool/kubectl.go
@@ -0,0 +1,56 @@
+package tool
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/helm/chart-testing/v3/pkg/exec"
+	"github.com/helm/chart-testing/v3/pkg/tool"
+)
+
+// Kubectl is an interface to the helm binary; it is a thin layer on top of the Kubectl abstraction offered by
+// chart-testing to silence output being streamed to Stdout.
+type Kubectl struct {
+	tool.Kubectl
+	exec.ProcessExecutor
+}
+
+func NewKubectl(exec exec.ProcessExecutor) Kubectl {
+	return Kubectl{
+		tool.NewKubectl(exec),
+		exec,
+	}
+}
+
+func (k Kubectl) WaitForDeployments(namespace string, selector string) error {
+	output, err := k.RunProcessAndCaptureOutput(
+		"kubectl", "get", "deployments", "--namespace", namespace, "--selector", selector, "--output", "jsonpath={.items[*].metadata.name}")
+	if err != nil {
+		return err
+	}
+
+	deployments := strings.Fields(output)
+	for _, deployment := range deployments {
+		deployment = strings.Trim(deployment, "'")
+		_, err := k.RunProcessAndCaptureOutput("kubectl", "rollout", "status", "deployment", deployment, "--namespace", namespace)
+		if err != nil {
+			return err
+		}
+
+		// 'kubectl rollout status' does not return a non-zero exit code when rollouts fail.
+		// We, thus, need to double-check here.
+		//
+		// Just after rollout, pods from the previous deployment revision may still be in a
+		// terminating state.
+		unavailable, err := k.RunProcessAndCaptureOutput("kubectl", "get", "deployment", deployment, "--namespace", namespace, "--output",
+			`jsonpath={.status.unavailableReplicas}`)
+		if err != nil {
+			return err
+		}
+		if unavailable != "" && unavailable != "0" {
+			return fmt.Errorf("%s replicas unavailable", unavailable)
+		}
+	}
+
+	return nil
+}


Test patch:
diff --git a/pkg/chartverifier/checks/charttesting.go b/pkg/chartverifier/checks/charttesting.go
index 839099e..9277685 100644
--- a/pkg/chartverifier/checks/charttesting.go
+++ b/pkg/chartverifier/checks/charttesting.go
@@ -7,8 +7,8 @@ import (
 	"github.com/helm/chart-testing/v3/pkg/chart"
 	"github.com/helm/chart-testing/v3/pkg/config"
 	"github.com/helm/chart-testing/v3/pkg/exec"
-	"github.com/helm/chart-testing/v3/pkg/tool"
 	"github.com/helm/chart-testing/v3/pkg/util"
+	"github.com/redhat-certification/chart-verifier/pkg/tool"
 )
 
 // buildChartTestingConfiguration computes the chart testing related