Repository: awslabs/aws-orbit-workbench (language: python)

Issue:
[BUG] - CodeBuild for lake-admin regression stuck with k8s dynamic client unauthorized error.
**Describe the bug**
PyTest based lake-admin regression code build stage hangs for EKS cluster connection via DynamicClient. 

Based on the availability of DynamicClient, OrbitJob status is checked for finalizing the PyTest status. 

Same code from developer local machine runs as expected to PASS or FAIL the PyTest. 

**To Reproduce**
Steps to reproduce the behavior:
Run lake-admin regression state in dev account.

**Expected behavior**
The PyTest running inside the codebuild has to check the OrbitJob status periodically and finalize the status of the PyTest.

**Screenshots**


WARNING  kubetest:utils.py:130 got api exception while waiting: 401
--
953 | Reason: Unauthorized
954 | HTTP response headers: HTTPHeaderDict({'Audit-Id': '74627bd8-d6c3-42d1-8fb1-19b9d7bc0550', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 06 Oct 2021 03:47:37 GMT', 'Content-Length': '129'})
955 | HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\n'
956 | Original traceback:
957 | File "/codebuild/output/src916/src/s3/00/.venv/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 55, in inner
958 | resp = func(self, *args, **kwargs)
959 |  
960 | File "/codebuild/output/src916/src/s3/00/.venv/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 237, in request
961 | return self.client.call_api(
962 |  
963 | File "/codebuild/output/src916/src/s3/00/.venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
964 | return self.__call_api(resource_path, method,
965 |  
966 | File "/codebuild/output/src916/src/s3/00/.venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
967 | response_data = self.request(
968 |  
969 | File "/codebuild/output/src916/src/s3/00/.venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
970 | return self.rest_client.GET(url,
971 |  
972 | File "/codebuild/output/src916/src/s3/00/.venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 239, in GET
973 | return self.request("GET", url,
974 |  
975 | File "/codebuild/output/src916/src/s3/00/.venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 233, in request
976 | raise ApiException(http_resp=r)
977 | - generated xml file: /codebuild/output/src916/src/s3/01/test/regressions/regression_report.xml -
978 | =========================== short test summary info ============================
979 | FAILED test_lake_admin.py::test_lakeadmin_2_image_with_apps - TimeoutError: t...
980 | !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
981 | !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 1 failures !!!!!!!!!!!!!
982 | =================== 1 failed, 2 passed in 7204.35s (2:00:04) ===================



**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.



Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 218f06974..a6b59542d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -20,6 +20,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Updated SDK to use OrbitJobs instead of Jobs and CronJobs
 - Changed Lake Creator notebook
 - Added sleep to Orbit SDK based OrbitJob CR job status check
+- Fixed the expiring kubeconfig issue for long living k8s jobs
 
 ### **Removed**
 
diff --git a/samples/notebooks/M-Admin/2-Image_with_apps.ipynb b/samples/notebooks/M-Admin/2-Image_with_apps.ipynb
index 835115505..f799b82e7 100644
--- a/samples/notebooks/M-Admin/2-Image_with_apps.ipynb
+++ b/samples/notebooks/M-Admin/2-Image_with_apps.ipynb
@@ -2,8 +2,6 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "49c3e7ec",
-   "metadata": {},
    "source": [
     "<!--\n",
     "#  Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.\n",
@@ -26,158 +24,150 @@
     "## Content\n",
     "1. [Configuration](#Configuration)\n",
     "2. [Build Image](#Build-Image)\n"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "ed879d3e",
-   "metadata": {},
    "source": [
     "### Configuration"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "7fc0d6b1",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "image_name = 'jupyter-with-apps'\n",
     "folder_name = '/home/jovyan/shared/samples/notebooks/M-Admin/extra_user_apps/'"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b228e90f",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "%cd $folder_name"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "2805031c",
-   "metadata": {},
    "source": [
     "### Build Image"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "a6a3ca5e",
-   "metadata": {},
    "source": [
     "Lets see how  orbit build image works..."
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "15558c62",
-   "metadata": {},
    "source": [
     "get our orbit env and team names"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "46816057",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "env_name = %env AWS_ORBIT_ENV\n",
     "team_name = %env AWS_ORBIT_TEAM_SPACE\n",
     "user_name = %env USERNAME\n",
     "namespace = %env AWS_ORBIT_USER_SPACE\n",
     "(env_name,team_name, user_name, namespace)"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "5194cce1",
-   "metadata": {},
    "source": [
     "Repository name will be created from the image name prefixed by the env context.  Users are only able to manipulate ECR repos that start with 'orbit-{env_name}/users/'"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "ce37daa7",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "repository_name = (f\"orbit-{env_name}/users/{image_name}\")\n",
     "repository_name"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "145f7e48",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "!aws ecr delete-repository --repository-name $repository_name --force"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "9c9b42c8",
-   "metadata": {},
    "source": [
     "adding a file to our docker as an example"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "26868a1c",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "pwd = %pwd\n",
     "pwd"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "source": [
+    "image_build_time_out = 60"
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "e60b33ab",
-   "metadata": {},
    "source": [
     "Now lets run the command"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "d57d0b13",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "%%time\n",
     "\n",
-    "output = !orbit build image -e $env_name -d $pwd -n $image_name \n",
+    "output = !orbit build image -e $env_name -d $pwd -n $image_name --timeout $image_build_time_out\n",
     "output"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "51e20708",
-   "metadata": {},
    "source": [
     "Lets get the image address from the output of the previous command"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "a9b2b9c7",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "look_for = 'ECR Image Address='\n",
     "image = None\n",
@@ -188,14 +178,13 @@
     "\n",
     "assert(image != None)       \n",
     "    "
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "50ade237",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "# check that the image was built\n",
     "import json\n",
@@ -205,30 +194,27 @@
     "im = json.loads(images)\n",
     "print(im['imageIds'])\n",
     "assert(len(im['imageIds']) > 0)"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "b1e3e7f0",
-   "metadata": {},
    "source": [
     "### Building the configuration for the image"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "861ba405",
-   "metadata": {},
    "source": [
     "## Users can now use the new profile to use new apps"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "70b4efcf",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "import json\n",
     "\n",
@@ -258,50 +244,46 @@
     "\n",
     "with open(\"podsetting_data.json\", 'w') as f:\n",
     "    json.dump(podsetting, f)"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "a18bcbe4",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "!orbit build podsetting --help "
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b3aaa27b",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "!orbit build podsetting --debug -e $env_name -t $team_name  podsetting_data.json"
-   ]
+   ],
+   "outputs": [],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "c6319649",
-   "metadata": {},
    "source": [
     "***"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "f1c9e2c9",
-   "metadata": {},
    "source": [
     "## OPTIONAL: You can use kubectl to load the podsettings as indicated below \n",
     "\n",
     "### (convert the cells to code from markdown)"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "c0f69308",
-   "metadata": {},
    "source": [
     "import json\n",
     "\n",
@@ -346,23 +328,22 @@
     "        runAsUser: 1000\n",
     "    \"\"\".format(team_name=team_name,env_name=env_name,customname=customname,image=image)\n",
     ")\n"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "id": "fe1c51e6",
-   "metadata": {},
    "source": [
     "!kubectl apply -f podsetting.yaml -n {team_name}"
-   ]
+   ],
+   "metadata": {}
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "43b8ba37",
-   "metadata": {},
+   "source": [],
    "outputs": [],
-   "source": []
+   "metadata": {}
   }
  ],
  "metadata": {
@@ -386,4 +367,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}
\ No newline at end of file


Test patch:
diff --git a/test/regressions/common_utils.py b/test/regressions/common_utils.py
index 4459d13c8..7c18ef566 100644
--- a/test/regressions/common_utils.py
+++ b/test/regressions/common_utils.py
@@ -12,88 +12,6 @@
 #    See the License for the specific language governing permissions and
 #    limitations under the License.
 
-import time
-import os
-import logging
-from typing import Union
-from kubernetes.client.rest import ApiException
-from kubernetes import config as k8_config
-from kubetest import condition
-
 JOB_COMPLETION_STATUS = "Complete"
 JOB_FAILED_STATUS = "Failed"
 
-log = logging.getLogger("kubetest")
-
-def load_kube_config():
-    if "AWS_WEB_IDENTITY_TOKEN_FILE" in os.environ and "eks.amazonaws.com" in os.environ["AWS_WEB_IDENTITY_TOKEN_FILE"]:
-        k8_config.load_incluster_config()
-    else:
-        k8_config.load_kube_config()
-
-def wait_for_custom_condition(
-    condition: condition.Condition,
-    timeout: int = None,
-    interval: Union[int, float] = 1,
-    fail_on_api_error: bool = True,
-) -> None:
-    """Wait for a condition to be met.
-
-    Args:
-        condition: The Condition to wait for.
-        timeout: The maximum time to wait, in seconds, for the condition to be met.
-            If unspecified, this function will wait indefinitely. If specified and
-            the timeout is met or exceeded, a TimeoutError will be raised.
-        interval: The time, in seconds, to wait before re-checking the condition.
-        fail_on_api_error: Fail the condition checks if a Kubernetes API error is
-            incurred. An API error can be raised for a number of reasons, including
-            a Pod being restarted and temporarily unavailable. Disabling this will
-            cause those errors to be ignored, allowing the check to continue until
-            timeout or resolution. (default: True).
-
-    Raises:
-        TimeoutError: The specified timeout was exceeded.
-    """
-    log.info(f"waiting for condition: {condition}")
-
-    # define the maximum time to wait. once this is met, we should
-    # stop waiting.
-    max_time = None
-    if timeout is not None:
-        max_time = time.time() + timeout
-
-    # start the wait block
-    start = time.time()
-    count = 0
-    retry_count = 3
-    while True:
-        if max_time and time.time() >= max_time:
-            raise TimeoutError(
-                f"timed out ({timeout}s) while waiting for condition {condition}"
-            )
-
-        # check if the condition is met and break out if it is
-        try:
-            if condition.check():
-                break
-        except ApiException as e:
-            log.info(type(e))
-            log.warning(f"got api exception while waiting: {e}")
-            if count < retry_count:
-                if e.reason == 'Unauthorized':
-                    load_kube_config()
-                    log.info("loading the kubeconfig during Unauthorized exception and retrying")
-                    count+=1
-                    continue
-                else:
-                    raise
-            else:
-                log.info(f"retried loading kubeconfig {retry_count} times")
-                raise
-
-        # if the condition is not met, sleep for the interval
-        # to re-check later
-        time.sleep(interval)
-
-    end = time.time()
-    log.info(f"wait completed (total={end-start}s) {condition}")
diff --git a/test/regressions/custom_resources.py b/test/regressions/custom_resources.py
index e39a151e1..aa744100c 100644
--- a/test/regressions/custom_resources.py
+++ b/test/regressions/custom_resources.py
@@ -21,8 +21,9 @@
 from kubetest.manifest import render
 from kubetest.objects import ApiObject
 from kubetest.client import TestClient
-from kubetest import condition
-from common_utils import JOB_COMPLETION_STATUS, JOB_FAILED_STATUS, wait_for_custom_condition
+from kubetest import condition, utils
+from common_utils import JOB_COMPLETION_STATUS, JOB_FAILED_STATUS
+from kubernetes import config as k8_config
 
 # This elminates some collection warnings for all tests using the TestClient
 TestClient.__test__ = False
@@ -72,23 +73,25 @@ def namespace(self, name: str) -> None:
         else:
             raise AttributeError("Cannot set namespace - object already has a namespace")
 
+    def refresh_api_client(self) -> None:
+        c = self.api_clients.get(self.version)
+        # If we didn't find the client in the api_clients dict, use the
+        # preferred version.
+        if c is None:
+            c = self.api_clients.get("preferred")
+            if c is None:
+                raise ValueError(
+                    "unknown version specified and no preferred version " f"defined for resource ({self.version})"
+                )
+        # If we did find it, initialize that client version.
+        self._api_client = c(client=api_client.ApiClient()).resources.get(
+            group=self.group, api_version=self.api_version, kind=self.kind
+        )
+
     @property
     def api_client(self) -> dynamic.DynamicClient:
-        if self._api_client is None:
-            c = self.api_clients.get(self.version)
-            # If we didn't find the client in the api_clients dict, use the
-            # preferred version.
-            if c is None:
-                log.warning(f"unknown version ({self.version}), falling back to preferred version")
-                c = self.api_clients.get("preferred")
-                if c is None:
-                    raise ValueError(
-                        "unknown version specified and no preferred version " f"defined for resource ({self.version})"
-                    )
-            # If we did find it, initialize that client version.
-            self._api_client = c(client=api_client.ApiClient()).resources.get(
-                group=self.group, api_version=self.api_version, kind=self.kind
-            )
+        self.refresh_api_client()
+        k8_config.load_kube_config()
         return self._api_client
 
     @classmethod
@@ -245,7 +248,7 @@ def wait_until_job_completes(
             self.is_complete,
         )
         # Wait until Orbit job completes
-        wait_for_custom_condition(
+        utils.wait_for_condition(
             condition=job_complete_condition,
             timeout=timeout,
             interval=interval,
diff --git a/test/regressions/pytest.ini b/test/regressions/pytest.ini
index 16d5a62cd..149482bea 100644
--- a/test/regressions/pytest.ini
+++ b/test/regressions/pytest.ini
@@ -13,5 +13,9 @@ markers =
 log_cli = 1
 log_cli_level = INFO
 log_file = ./test_results/test_log.log
+log_format = %(asctime)s %(levelname)s %(message)s
+log_date_format = %Y-%m-%d %H:%M:%S
+log_cli_format = %(asctime)s %(levelname)s %(message)s
+log_cli_date_format = %Y-%m-%d %H:%M:%S
 log_file_level = INFO
 addopts = --strict-markers --rootdir . -c ./pytest.ini --kube-config ~/.kube/config -v -s --maxfail=1 --lf
\ No newline at end of file
diff --git a/test/regressions/test_lake_admin.py b/test/regressions/test_lake_admin.py
index 9cf78dba3..3ff0b754d 100644
--- a/test/regressions/test_lake_admin.py
+++ b/test/regressions/test_lake_admin.py
@@ -91,7 +91,7 @@ def test_lakeadmin_2_image_with_apps(kube: TestClient) -> None:
     # Logic to wait till OrbitJob creates
     lakeadmin.wait_until_ready(timeout=60)
     # Logic to pass or fail the pytest
-    lakeadmin.wait_until_job_completes(timeout=7200)
+    lakeadmin.wait_until_job_completes(timeout=7200, interval=30)
     current_status = lakeadmin.get_status().get("orbitJobOperator").get("jobStatus")
     logger.info(f"current_status={current_status}")
     #Cleanup