Repository: redhat-developer/service-binding-operator
Pull request #634 (2020-08-24 08:39:04)

[acceptance-tests] Increase various timeouts to make acceptance tests less flaky
### Motivation

Currently, the time to reach the required condition in cases below is exceeded sometimes. In that case the test fails even if the condition is met eventually.  Those relatively small timeouts make the acceptnace tests flaky.

### Changes

This PR increases the timeout for:
* "Get SBR info by jsonpath" to 10 minutes (was 25s)
* "Get SBR info by jq" to 10 minutes (was 25s)
* "Start local SBO instance" to 120s (was 60s)
* "Wait for the Node.js app to re-deploy" to 2 minutes (was 60s)

### Testing

`make test-acceptnace`


openshift-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **pmacik**
You can assign the PR to them by writing `/assign @pmacik` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=redhat-developer%2Fservice-binding-operator).

<details open>
Needs approval from an approver in each of these files:

- **[test/OWNERS](https://github.com/redhat-developer/service-binding-operator/blob/master/test/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["pmacik"]} -->
openshift-ci-robot: @pmacik: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/unit | 9793a9a8f7eafabd58e66ab94e90c512d0078ca8 | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-unit/1297815686508187648) | `/test unit`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=redhat-developer&repo=service-binding-operator&pr=634). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:pmacik).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
openshift-ci-robot: @pmacik: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/unit | 9793a9a8f7eafabd58e66ab94e90c512d0078ca8 | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-unit/1297815686508187648) | `/test unit`
ci/prow/e2e | 9793a9a8f7eafabd58e66ab94e90c512d0078ca8 | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-e2e/1297815686449467392) | `/test e2e`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=redhat-developer&repo=service-binding-operator&pr=634). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:pmacik).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
pmacik: /test 4.5-acceptance 
pmacik: /retest

openshift-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **avni-sharma**
You can assign the PR to them by writing `/assign @avni-sharma` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=redhat-developer%2Fservice-binding-operator).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/redhat-developer/service-binding-operator/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["avni-sharma"]} -->
openshift-ci-robot: @pmacik: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/e2e | 9793a9a8f7eafabd58e66ab94e90c512d0078ca8 | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-e2e/1297815686449467392) | `/test e2e`
ci/prow/4.5-e2e | 83d423e1ddc0bdf8027cf95222b356a1f10cff97 | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-4.5-e2e/1297831633906307072) | `/test 4.5-e2e`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=redhat-developer&repo=service-binding-operator&pr=634). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:pmacik).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
pmacik: /retest
DhritiShikhar: Still fails for me acceptance tests
DhritiShikhar: shared the failure log on channel
openshift-ci-robot: @pmacik: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/4.5-e2e | 1dd1972d2962c18559393eaf869d681cb15613cb | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-4.5-e2e/1297849912876077056) | `/test 4.5-e2e`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=redhat-developer&repo=service-binding-operator&pr=634). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:pmacik).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
openshift-ci-robot: @pmacik: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/4.5-e2e | 1dd1972d2962c18559393eaf869d681cb15613cb | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-4.5-e2e/1297849912876077056) | `/test 4.5-e2e`
ci/prow/e2e | 1dd1972d2962c18559393eaf869d681cb15613cb | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-e2e/1297849912783802368) | `/test e2e`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=redhat-developer&repo=service-binding-operator&pr=634). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:pmacik).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
openshift-ci-robot: @pmacik: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ci/prow/4.5-e2e | 1dd1972d2962c18559393eaf869d681cb15613cb | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-4.5-e2e/1297849912876077056) | `/test 4.5-e2e`
ci/prow/e2e | 1dd1972d2962c18559393eaf869d681cb15613cb | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-e2e/1297849912783802368) | `/test e2e`
ci/prow/4.5-unit | ca8a79502fbb9122380a95cf15f91111e96bfc53 | [link](https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-developer_service-binding-operator/634/pull-ci-redhat-developer-service-binding-operator-master-4.5-unit/1297875578669502464) | `/test 4.5-unit`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=redhat-developer&repo=service-binding-operator&pr=634). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:pmacik).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
DhritiShikhar: Checked the latest commit.
Fails -> https://paste.centos.org/view/dfb9d2ea
pedjak: @pmacik out of the scope of this PR, but could introduce a global configurable timeout that could be configured from the command line?
pedjak: /lgtm
/approve
openshift-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/redhat-developer/service-binding-operator/pull/634#issuecomment-679221637" title="Approved">pedjak</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=redhat-developer%2Fservice-binding-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/redhat-developer/service-binding-operator/blob/master/OWNERS)~~ [pedjak]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Patch:
diff --git a/hack/deploy-sbo-local.sh b/hack/deploy-sbo-local.sh
index a4fa54bc..3550d045 100755
--- a/hack/deploy-sbo-local.sh
+++ b/hack/deploy-sbo-local.sh
@@ -24,7 +24,7 @@ operator-sdk --verbose run --local --namespace="$OPERATOR_NAMESPACE" --operator-
 
 SBO_PID=$!
 
-attempts=12
+attempts=24
 while [ -z "$(grep 'Starting workers' $SBO_LOCAL_LOG)" ]; do
     if [[ $attempts -ge 0 ]]; then
         sleep 5


Test patch:
diff --git a/test/acceptance/features/steps/openshift.py b/test/acceptance/features/steps/openshift.py
index 76b0da3e..b28a3542 100644
--- a/test/acceptance/features/steps/openshift.py
+++ b/test/acceptance/features/steps/openshift.py
@@ -170,26 +170,26 @@ spec:
         exit_code | should.be_equal_to(0)
         return env_from
 
-    def get_resource_info_by_jsonpath(self, resource_type, name, namespace, json_path, wait=False):
+    def get_resource_info_by_jsonpath(self, resource_type, name, namespace, json_path, wait=False, interval=5, timeout=120):
         output, exit_code = self.cmd.run(f'oc get {resource_type} {name} -n {namespace} -o "jsonpath={json_path}"')
         if exit_code != 0:
             if wait:
-                attempts = 5
+                attempts = timeout/interval
                 while exit_code != 0 and attempts > 0:
                     output, exit_code = self.cmd.run(f'oc get {resource_type} {name} -n {namespace} -o "jsonpath={json_path}"')
                     attempts -= 1
-                    time.sleep(5)
+                    time.sleep(interval)
         exit_code | should.be_equal_to(0).desc(f'Exit code should be 0:\n OUTPUT:\n{output}')
         return output
 
-    def get_resource_info_by_jq(self, resource_type, name, namespace, jq_expression, wait=False):
+    def get_resource_info_by_jq(self, resource_type, name, namespace, jq_expression, wait=False, interval=5, timeout=120):
         output, exit_code = self.cmd.run(f'oc get {resource_type} {name} -n {namespace} -o json | jq -rc \'{jq_expression}\'')
         if exit_code != 0:
             if wait:
-                attempts = 5
+                attempts = timeout/interval
                 while exit_code != 0 and attempts > 0:
                     output, exit_code = self.cmd.run(f'oc get {resource_type} {name} -n {namespace} -o json | jq -rc \'{jq_expression}\'')
                     attempts -= 1
-                    time.sleep(5)
+                    time.sleep(interval)
         exit_code | should.be_equal_to(0).desc(f'Exit code should be 0:\n OUTPUT:\n{output}')
         return output.rstrip("\n")
diff --git a/test/acceptance/features/steps/steps.py b/test/acceptance/features/steps/steps.py
index d7adfaa9..31cca29a 100644
--- a/test/acceptance/features/steps/steps.py
+++ b/test/acceptance/features/steps/steps.py
@@ -157,7 +157,7 @@ def sbr_is_applied(context):
 @then(u'application should be re-deployed')
 def then_application_redeployed(context):
     application = context.nodejs_app
-    application.get_redeployed_pod_name(context.nodejs_app_original_pod_name) | should_not.be_none.desc(
+    application.get_redeployed_pod_name(context.nodejs_app_original_pod_name, timeout=120) | should_not.be_none.desc(
         "There is a running pod of the application different from the original one before redeployment.")
 
 
@@ -174,7 +174,7 @@ def then_app_is_connected_to_db(context, db_name):
 def then_sbo_jsonpath_is(context, json_path, sbr_name, json_value_regex):
     openshift = Openshift()
     openshift.search_resource_in_namespace("servicebindingrequests", sbr_name, context.namespace.name) | should_not.be_none.desc("SBR {sbr_name} exists")
-    result = openshift.get_resource_info_by_jsonpath("sbr", sbr_name, context.namespace.name, json_path, wait=True)
+    result = openshift.get_resource_info_by_jsonpath("sbr", sbr_name, context.namespace.name, json_path, wait=True, timeout=600)
     result | should_not.be_none.desc("jsonpath result")
     re.fullmatch(json_value_regex, result) | should_not.be_none.desc("SBO jsonpath result \"{result}\" should match \"{json_value_regex}\"")
 
@@ -184,7 +184,7 @@ def then_sbo_jsonpath_is(context, json_path, sbr_name, json_value_regex):
 def then_sbo_jq_is(context, jq_expression, sbr_name, json_value_regex):
     openshift = Openshift()
     openshift.search_resource_in_namespace("servicebindingrequests", sbr_name, context.namespace.name) | should_not.be_none.desc("SBR {sbr_name} exists")
-    result = openshift.get_resource_info_by_jq("sbr", sbr_name, context.namespace.name, jq_expression, wait=True)
+    result = openshift.get_resource_info_by_jq("sbr", sbr_name, context.namespace.name, jq_expression, wait=True, timeout=600)
     result | should_not.be_none.desc("jq result")
     re.fullmatch(json_value_regex, result) | should_not.be_none.desc("SBO jq result \"{result}\" should match \"{json_value_regex}\"")