Repository: code-ready/crc
Pull request #1363 (2020-07-07 09:22:53)

Prepare to update ~/.kube/config with crc contexts
This is preliminary work for updating kubeconfig


openshift-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/code-ready/crc/pull/1363#" title="Author self-approved">guillaumerose</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=code-ready%2Fcrc).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/code-ready/crc/blob/master/OWNERS)~~ [guillaumerose]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
praveenkumar: @guillaumerose The biggest challenge for this to work every single time is how our auth operator able to serve request. Since we just wait for a 3 mins and assume everything is up and running for the cluster but in real that is not true for some of the scenario (like proxy case) or even a bit slow system, that's the reason we have `crc status` command to check the operator status and the login is taken care by auth so if that is not available in 3 mins then this is going to error out and our end user will think something broken with CRC, which might not be the case. 

We thought of https://github.com/code-ready/crc/pull/242 but some part of it still not implemented. 

openshift-ci-robot: @guillaumerose: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/e2e-crc | 157c5e46ffb9ca7d4d97b5de7293f73b004f7078 | [link](https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/code-ready_crc/1363/pull-ci-code-ready-crc-master-e2e-crc/1280487301318709248) | `/test e2e-crc`

[Full PR test history](https://prow.svc.ci.openshift.org/pr-history?org=code-ready&repo=crc&pr=1363). [Your PR dashboard](https://prow.svc.ci.openshift.org/pr?query=is:pr+state:open+author:guillaumerose). Please help us cut down on flakes by [linking to](https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/code-ready/crc/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
praveenkumar: /hold
openshift-ci-robot: @guillaumerose: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/e2e-crc | 8cc26511a9ab17eb8526656758e0f55bc2d3f18f | [link](https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/code-ready_crc/1363/pull-ci-code-ready-crc-master-e2e-crc/1282608643732148224) | `/test e2e-crc`

[Full PR test history](https://prow.svc.ci.openshift.org/pr-history?org=code-ready&repo=crc&pr=1363). [Your PR dashboard](https://prow.svc.ci.openshift.org/pr?query=is:pr+state:open+author:guillaumerose). Please help us cut down on flakes by [linking to](https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/code-ready/crc/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
openshift-ci-robot: @guillaumerose: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/e2e-crc | 62c8727d26454bcc42164bd1d5b6626015276ef4 | [link](https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/code-ready_crc/1363/pull-ci-code-ready-crc-master-e2e-crc/1282635553136709632) | `/test e2e-crc`

[Full PR test history](https://prow.svc.ci.openshift.org/pr-history?org=code-ready&repo=crc&pr=1363). [Your PR dashboard](https://prow.svc.ci.openshift.org/pr?query=is:pr+state:open+author:guillaumerose). Please help us cut down on flakes by [linking to](https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/code-ready/crc/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
openshift-ci-robot: @guillaumerose: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/e2e-crc | 5c2e25ac560bdbdb73e3bb44ebac93c3461a253e | [link](https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/code-ready_crc/1363/pull-ci-code-ready-crc-master-e2e-crc/1282686487652995072) | `/test e2e-crc`

[Full PR test history](https://prow.svc.ci.openshift.org/pr-history?org=code-ready&repo=crc&pr=1363). [Your PR dashboard](https://prow.svc.ci.openshift.org/pr?query=is:pr+state:open+author:guillaumerose). Please help us cut down on flakes by [linking to](https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/code-ready/crc/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
guillaumerose: I added code to wait the authentication operator. Ready for a new round of review!

/retest
/unhold

openshift-ci-robot: @guillaumerose: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/e2e-crc | 5c2e25ac560bdbdb73e3bb44ebac93c3461a253e | [link](https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/code-ready_crc/1363/pull-ci-code-ready-crc-master-e2e-crc/1283300370205904896) | `/test e2e-crc`

[Full PR test history](https://prow.svc.ci.openshift.org/pr-history?org=code-ready&repo=crc&pr=1363). [Your PR dashboard](https://prow.svc.ci.openshift.org/pr?query=is:pr+state:open+author:guillaumerose). Please help us cut down on flakes by [linking to](https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/code-ready/crc/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
openshift-ci-robot: @praveenkumar: The `/retest` command does not accept any targets.
The following commands are available to trigger jobs:
* `/test e2e-crc`
* `/test images`

Use `/test all` to run all jobs.


<details>

In response to [this](https://github.com/code-ready/crc/pull/1363#issuecomment-658879611):

>/retest this please


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
praveenkumar: /test e2e-crc
praveenkumar: /retest this please
openshift-ci-robot: @praveenkumar: The `/retest` command does not accept any targets.
The following commands are available to trigger jobs:
* `/test e2e-crc`
* `/test images`

Use `/test all` to run all jobs.


<details>

In response to [this](https://github.com/code-ready/crc/pull/1363#issuecomment-659197351):

>/retest this please


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
praveenkumar: Overall this PR looks good but we still have issue with merging the kubeconfig file looks like the token are not updated in case of merge, this is limitation of kubeconfig merge side. :( 

```yaml
$ oc config view --kubeconfig /tmp/crc588552271/crc-admin 
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://api.crc.testing:6443
  name: api-crc-testing:6443
contexts:
- context:
    cluster: api-crc-testing:6443
    namespace: default
    user: kube:admin
  name: crc-admin
current-context: crc-admin
kind: Config
preferences: {}
users:
- name: kube:admin
  user:
    token: LhtL4sIvSID3oJuimLzWIXuedx0DOtbRS1-B2sM0_II
$ oc config view --kubeconfig /tmp/crc588552271/crc-dev 
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://api.crc.testing:6443
  name: api-crc-testing:6443
contexts:
- context:
    cluster: api-crc-testing:6443
    user: developer
  name: crc-dev
current-context: crc-dev
kind: Config
preferences: {}
users:
- name: developer
  user:
    token: b_2RzZUeiQ1yFKsBSziC6K6UT9Hz_tl9oyS-L3jPb3I
$ export KUBECONFIG=/tmp/crc588552271/crc-dev:/tmp/crc588552271/crc-admin:~/.kube/config
$ oc config view
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://api.crc.testing:6443
  name: api-crc-testing:6443
contexts:
- context:
    cluster: api-crc-testing:6443
    namespace: default
    user: kube:admin
  name: crc-admin
- context:
    cluster: api-crc-testing:6443
    user: developer
  name: crc-dev
current-context: crc-dev
kind: Config
preferences: {}
users:
- name: developer
  user:
    token: nd8VTMdYe60t5vrB_tVClg8xEBsIWfhT2OCWsF3hAWQ
- name: kube:admin
  user:
    token: VfIpu5_TZTvCkVTEOYG4TwtQNGB0pFQiC3WmnE2_SGs
```

The another problem is if someone set the `KUBECONFIG` env with merge type `foo:bar:~/.kube/config`  then we are honoring this variable but it will fails with no file found when writing back to kubeconfig file.

```
ERRO open foo:bar:~/.kube/config: no such file or directory 
```

I am happy to take other commits (mostly refactor) as it is and the last one can be a separate PR.
openshift-ci-robot: @guillaumerose: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/e2e-crc | f899772245869a69003aafd164cbcaae84720946 | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/code-ready_crc/1363/pull-ci-code-ready-crc-master-e2e-crc/1284026953816346624) | `/test e2e-crc`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=code-ready&repo=crc&pr=1363). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:guillaumerose).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/crc/cluster/clusteroperator.go b/pkg/crc/cluster/clusteroperator.go
index e6ddbd456..517abdd4a 100644
--- a/pkg/crc/cluster/clusteroperator.go
+++ b/pkg/crc/cluster/clusteroperator.go
@@ -2,6 +2,7 @@ package cluster
 
 import (
 	"encoding/json"
+	"errors"
 	"fmt"
 	"time"
 
@@ -35,24 +36,38 @@ type ClusterStatus struct {
 	Disabled    bool
 }
 
-func GetClusterOperatorStatus(ocConfig oc.OcConfig) (*ClusterStatus, error) {
-	cs := &ClusterStatus{}
+func GetClusterOperatorStatus(ocConfig oc.OcConfig, operator string) (*ClusterStatus, error) {
+	return getStatus(ocConfig, []string{operator})
+}
+
+func GetClusterOperatorsStatus(ocConfig oc.OcConfig) (*ClusterStatus, error) {
+	return getStatus(ocConfig, []string{})
+}
+
+func getStatus(ocConfig oc.OcConfig, selector []string) (*ClusterStatus, error) {
+	cs := &ClusterStatus{
+		Available: true,
+	}
+
 	data, stderr, err := ocConfig.RunOcCommand("get", "co", "-ojson")
 	if err != nil {
 		return cs, fmt.Errorf("%s", stderr)
 	}
 
 	var co K8sResource
-
-	err = json.Unmarshal([]byte(data), &co)
-	if err != nil {
+	if err := json.Unmarshal([]byte(data), &co); err != nil {
 		return cs, err
 	}
-	cs.Available = true
+
+	found := false
 	for _, c := range co.Items {
 		if contains(c.Metadata.Name, ignoreClusterOperators) {
 			continue
 		}
+		if len(selector) > 0 && !contains(c.Metadata.Name, selector) {
+			continue
+		}
+		found = true
 		for _, con := range c.Status.Conditions {
 			switch con.Type {
 			case "Available":
@@ -82,6 +97,9 @@ func GetClusterOperatorStatus(ocConfig oc.OcConfig) (*ClusterStatus, error) {
 			}
 		}
 	}
+	if !found {
+		return nil, errors.New("no cluster operator found")
+	}
 	return cs, nil
 }
 
diff --git a/pkg/crc/machine/machine.go b/pkg/crc/machine/machine.go
index 69708b3c6..35c3b8f87 100644
--- a/pkg/crc/machine/machine.go
+++ b/pkg/crc/machine/machine.go
@@ -602,7 +602,7 @@ func Status(statusConfig ClusterStatusConfig) (ClusterStatusResult, error) {
 
 		// check if all the clusteroperators are running
 		ocConfig := oc.UseOCWithConfig(statusConfig.Name)
-		operatorsStatus, err := cluster.GetClusterOperatorStatus(ocConfig)
+		operatorsStatus, err := cluster.GetClusterOperatorsStatus(ocConfig)
 		if err != nil {
 			openshiftStatus = "Not Reachable"
 			logging.Debug(err.Error())
diff --git a/pkg/crc/oc/oc.go b/pkg/crc/oc/oc.go
index 4b624ac89..b1fa70ce5 100644
--- a/pkg/crc/oc/oc.go
+++ b/pkg/crc/oc/oc.go
@@ -10,7 +10,6 @@ import (
 type OcRunner interface {
 	Run(args ...string) (string, string, error)
 	RunPrivate(args ...string) (string, string, error)
-	GetKubeconfigPath() string
 }
 
 type OcConfig struct {
@@ -25,15 +24,30 @@ type OcLocalRunner struct {
 }
 
 func (oc OcLocalRunner) Run(args ...string) (string, string, error) {
+	args = append(args, "--kubeconfig", oc.KubeconfigPath)
 	return crcos.RunWithDefaultLocale(oc.OcBinaryPath, args...)
 }
 
 func (oc OcLocalRunner) RunPrivate(args ...string) (string, string, error) {
+	args = append(args, "--kubeconfig", oc.KubeconfigPath)
 	return crcos.RunWithDefaultLocalePrivate(oc.OcBinaryPath, args...)
 }
 
-func (oc OcLocalRunner) GetKubeconfigPath() string {
-	return oc.KubeconfigPath
+type OcEnvRunner struct {
+	OcBinaryPath  string
+	KubeconfigEnv string
+}
+
+func (oc OcEnvRunner) Run(args ...string) (string, string, error) {
+	return crcos.RunWithDefaultLocaleAndEnv(oc.OcBinaryPath, args, map[string]string{
+		"KUBECONFIG": oc.KubeconfigEnv,
+	})
+}
+
+func (oc OcEnvRunner) RunPrivate(args ...string) (string, string, error) {
+	return crcos.RunWithDefaultLocalePrivateAndEnv(oc.OcBinaryPath, args, map[string]string{
+		"KUBECONFIG": oc.KubeconfigEnv,
+	})
 }
 
 // UseOcWithConfig return the oc binary along with valid kubeconfig
@@ -42,23 +56,29 @@ func UseOCWithConfig(machineName string) OcConfig {
 		OcBinaryPath:   filepath.Join(constants.CrcOcBinDir, constants.OcBinaryName),
 		KubeconfigPath: filepath.Join(constants.MachineInstanceDir, machineName, "kubeconfig"),
 	}
-	return NewOcConfig(localRunner, constants.DefaultContext, constants.DefaultName)
+	return OcConfig{
+		Runner:  localRunner,
+		Context: constants.DefaultContext,
+		Cluster: constants.DefaultName,
+	}
 }
 
 func (oc OcConfig) RunOcCommand(args ...string) (string, string, error) {
-	args = append(args, "--kubeconfig", oc.Runner.GetKubeconfigPath(), "--context", oc.Context, "--cluster", oc.Cluster)
+	if oc.Context != "" {
+		args = append(args, "--context", oc.Context)
+	}
+	if oc.Cluster != "" {
+		args = append(args, "--cluster", oc.Cluster)
+	}
 	return oc.Runner.Run(args...)
 }
 
 func (oc OcConfig) RunOcCommandPrivate(args ...string) (string, string, error) {
-	args = append(args, "--kubeconfig", oc.Runner.GetKubeconfigPath(), "--context", oc.Context, "--cluster", oc.Cluster)
-	return oc.Runner.RunPrivate(args...)
-}
-
-func NewOcConfig(runner OcRunner, context string, clusterName string) OcConfig {
-	return OcConfig{
-		Runner:  runner,
-		Context: context,
-		Cluster: clusterName,
+	if oc.Context != "" {
+		args = append(args, "--context", oc.Context)
+	}
+	if oc.Cluster != "" {
+		args = append(args, "--cluster", oc.Cluster)
 	}
+	return oc.Runner.RunPrivate(args...)
 }
diff --git a/pkg/os/exec.go b/pkg/os/exec.go
index f5f93f55b..dd59e645c 100644
--- a/pkg/os/exec.go
+++ b/pkg/os/exec.go
@@ -14,7 +14,7 @@ func runCmd(command string, args []string, env map[string]string) (string, strin
 	if len(env) != 0 {
 		cmd.Env = os.Environ()
 		for key, value := range env {
-			cmd.Env = ReplaceEnv(cmd.Env, key, value)
+			cmd.Env = ReplaceOrAddEnv(cmd.Env, key, value)
 		}
 	}
 	stdOut := new(bytes.Buffer)
@@ -52,9 +52,25 @@ func RunWithPrivilege(reason string, cmdAndArgs ...string) (string, string, erro
 }
 
 func RunWithDefaultLocale(command string, args ...string) (string, string, error) {
-	return run(command, args, map[string]string{"LC_ALL": "C", "LANG": "C"})
+	return RunWithDefaultLocaleAndEnv(command, args, map[string]string{})
 }
 
 func RunWithDefaultLocalePrivate(command string, args ...string) (string, string, error) {
-	return runPrivate(command, args, map[string]string{"LC_ALL": "C", "LANG": "C"})
+	return RunWithDefaultLocalePrivateAndEnv(command, args, map[string]string{})
+}
+
+func RunWithDefaultLocaleAndEnv(command string, args []string, extraEnv map[string]string) (string, string, error) {
+	env := map[string]string{"LC_ALL": "C", "LANG": "C"}
+	for k, v := range extraEnv {
+		env[k] = v
+	}
+	return run(command, args, env)
+}
+
+func RunWithDefaultLocalePrivateAndEnv(command string, args []string, extraEnv map[string]string) (string, string, error) {
+	env := map[string]string{"LC_ALL": "C", "LANG": "C"}
+	for k, v := range extraEnv {
+		env[k] = v
+	}
+	return runPrivate(command, args, env)
 }
diff --git a/pkg/os/util.go b/pkg/os/util.go
index 39053153d..38fbfebeb 100644
--- a/pkg/os/util.go
+++ b/pkg/os/util.go
@@ -37,19 +37,25 @@ func CurrentOS() OS {
 	panic("Unexpected OS type")
 }
 
-// ReplaceEnv changes the value of an environment variable
+// ReplaceOrAddEnv changes the value of an environment variable if it exists otherwise add the new variable
 // It drops the existing value and appends the new value in-place
-func ReplaceEnv(variables []string, varName string, value string) []string {
+func ReplaceOrAddEnv(variables []string, varName string, value string) []string {
 	var result []string
+
+	found := false
 	for _, e := range variables {
 		pair := strings.Split(e, "=")
 		if pair[0] != varName {
 			result = append(result, e)
 		} else {
+			found = true
 			result = append(result, fmt.Sprintf("%s=%s", varName, value))
 		}
 	}
 
+	if !found {
+		result = append(result, fmt.Sprintf("%s=%s", varName, value))
+	}
 	return result
 }
 


Test patch:
diff --git a/pkg/crc/cluster/clusteroperator_test.go b/pkg/crc/cluster/clusteroperator_test.go
index e33009451..9f959021e 100644
--- a/pkg/crc/cluster/clusteroperator_test.go
+++ b/pkg/crc/cluster/clusteroperator_test.go
@@ -5,30 +5,57 @@ import (
 	"path/filepath"
 	"testing"
 
-	"github.com/code-ready/crc/pkg/crc/errors"
 	"github.com/code-ready/crc/pkg/crc/oc"
+
+	"github.com/code-ready/crc/pkg/crc/errors"
 	"github.com/stretchr/testify/assert"
 )
 
-func TestGetClusterOperatorStatus(t *testing.T) {
-	status, err := GetClusterOperatorStatus(oc.OcConfig{
-		Runner: &mockRunner{file: filepath.Join("testdata", "co.json")},
-	})
-	assert.NoError(t, err)
-	assert.Equal(t, &ClusterStatus{
+var (
+	available = &ClusterStatus{
 		Available: true,
-	}, status)
+	}
+	progressing = &ClusterStatus{
+		Available:   true,
+		Progressing: true,
+	}
+)
+
+func TestGetClusterOperatorsStatus(t *testing.T) {
+	status, err := GetClusterOperatorsStatus(ocConfig("co.json"))
+	assert.NoError(t, err)
+	assert.Equal(t, available, status)
 }
 
-func TestGetClusterOperatorStatusProgressing(t *testing.T) {
-	status, err := GetClusterOperatorStatus(oc.OcConfig{
-		Runner: &mockRunner{file: filepath.Join("testdata", "co-progressing.json")},
-	})
+func TestGetClusterOperatorsStatusProgressing(t *testing.T) {
+	status, err := GetClusterOperatorsStatus(ocConfig("co-progressing.json"))
 	assert.NoError(t, err)
-	assert.Equal(t, &ClusterStatus{
-		Available:   true,
-		Progressing: true,
-	}, status)
+	assert.Equal(t, progressing, status)
+}
+
+func TestGetClusterOperatorStatus(t *testing.T) {
+	status, err := GetClusterOperatorStatus(ocConfig("co.json"), "authentication")
+	assert.NoError(t, err)
+	assert.Equal(t, available, status)
+
+	status, err = GetClusterOperatorStatus(ocConfig("co-progressing.json"), "authentication")
+	assert.NoError(t, err)
+	assert.Equal(t, progressing, status)
+
+	status, err = GetClusterOperatorStatus(ocConfig("co-progressing.json"), "cloud-credential")
+	assert.NoError(t, err)
+	assert.Equal(t, available, status)
+}
+
+func TestGetClusterOperatorStatusNotFound(t *testing.T) {
+	_, err := GetClusterOperatorStatus(ocConfig("co-progressing.json"), "not-found")
+	assert.EqualError(t, err, "no cluster operator found")
+}
+
+func ocConfig(s string) oc.OcConfig {
+	return oc.OcConfig{
+		Runner: &mockRunner{file: filepath.Join("testdata", s)},
+	}
 }
 
 type mockRunner struct {
diff --git a/pkg/crc/errors/multierror_test.go b/pkg/crc/errors/multierror_test.go
new file mode 100644
index 000000000..70395ac07
--- /dev/null
+++ b/pkg/crc/errors/multierror_test.go
@@ -0,0 +1,51 @@
+package errors
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestRetryAfter(t *testing.T) {
+	calls := 0
+	ret := RetryAfter(10, func() error {
+		calls++
+		return nil
+	}, 0)
+	assert.NoError(t, ret)
+	assert.Equal(t, 1, calls)
+}
+
+func TestRetryAfterFailure(t *testing.T) {
+	calls := 0
+	ret := RetryAfter(10, func() error {
+		calls++
+		return errors.New("failed")
+	}, 0)
+	assert.EqualError(t, ret, "failed")
+	assert.Equal(t, 1, calls)
+}
+
+func TestRetryAfterMaxAttempts(t *testing.T) {
+	calls := 0
+	ret := RetryAfter(3, func() error {
+		calls++
+		return &RetriableError{Err: errors.New("failed")}
+	}, 0)
+	assert.EqualError(t, ret, "Temporary Error: failed\nTemporary Error: failed\nTemporary Error: failed")
+	assert.Equal(t, 3, calls)
+}
+
+func TestRetryAfterSuccessAfterFailures(t *testing.T) {
+	calls := 0
+	ret := RetryAfter(5, func() error {
+		calls++
+		if calls < 3 {
+			return &RetriableError{Err: errors.New("failed")}
+		}
+		return nil
+	}, 0)
+	assert.NoError(t, ret)
+	assert.Equal(t, 3, calls)
+}
diff --git a/pkg/crc/oc/oc_linux_test.go b/pkg/crc/oc/oc_linux_test.go
new file mode 100644
index 000000000..8d233a0ef
--- /dev/null
+++ b/pkg/crc/oc/oc_linux_test.go
@@ -0,0 +1,43 @@
+package oc
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestRunCommand(t *testing.T) {
+	ocConfig := OcConfig{
+		Runner: OcLocalRunner{
+			OcBinaryPath:   "/bin/echo",
+			KubeconfigPath: "kubeconfig-file",
+		},
+		Context: "a-context",
+		Cluster: "a-cluster",
+	}
+	stdout, _, err := ocConfig.RunOcCommand("a-command")
+	assert.NoError(t, err)
+	assert.Equal(t, "a-command --context a-context --cluster a-cluster --kubeconfig kubeconfig-file\n", stdout)
+}
+
+func TestEnvRunner(t *testing.T) {
+	runner := OcEnvRunner{
+		OcBinaryPath:  "printenv",
+		KubeconfigEnv: "some-env",
+	}
+	stdout, _, err := runner.Run()
+	assert.NoError(t, err)
+	assert.Contains(t, stdout, "KUBECONFIG=some-env\n")
+}
+
+func TestRunCommandWithoutContextAndCluster(t *testing.T) {
+	ocConfig := OcConfig{
+		Runner: OcLocalRunner{
+			OcBinaryPath:   "/bin/echo",
+			KubeconfigPath: "kubeconfig-file",
+		},
+	}
+	stdout, _, err := ocConfig.RunOcCommand("a-command")
+	assert.NoError(t, err)
+	assert.Equal(t, "a-command --kubeconfig kubeconfig-file\n", stdout)
+}
diff --git a/pkg/os/util_test.go b/pkg/os/util_test.go
new file mode 100644
index 000000000..cb6bc4ecf
--- /dev/null
+++ b/pkg/os/util_test.go
@@ -0,0 +1,24 @@
+package os
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestReplaceEnv(t *testing.T) {
+	env := []string{"HOME=/home/user/", "PATH=/bin:/sbin:/usr/bin", "LC_ALL=de_DE.UTF8"}
+	replaced := ReplaceOrAddEnv(env, "LC_ALL", "C")
+
+	assert.Len(t, replaced, 3)
+	assert.Equal(t, env[2], "LC_ALL=de_DE.UTF8")
+	assert.Equal(t, replaced[2], "LC_ALL=C")
+}
+
+func TestAddEnv(t *testing.T) {
+	env := []string{"HOME=/home/user/", "PATH=/bin:/sbin:/usr/bin", "LC_ALL=de_DE.UTF8"}
+	replaced := ReplaceOrAddEnv(env, "KUBECONFIG", "some-data")
+
+	assert.Len(t, replaced, 4)
+	assert.Equal(t, replaced[3], "KUBECONFIG=some-data")
+}
diff --git a/test/integration/crcsuite/crcsuite.go b/test/integration/crcsuite/crcsuite.go
index 543dc867c..fc5ba72f5 100644
--- a/test/integration/crcsuite/crcsuite.go
+++ b/test/integration/crcsuite/crcsuite.go
@@ -184,7 +184,7 @@ func CheckClusterOperatorsWithRetry(retryCount int, retryWait string) error {
 
 	ocConfig := oc.UseOCWithConfig("crc")
 	for i := 0; i < retryCount; i++ {
-		s, err := cluster.GetClusterOperatorStatus(ocConfig)
+		s, err := cluster.GetClusterOperatorsStatus(ocConfig)
 		if err != nil {
 			return err
 		}