Repository: linkerd/linkerd2
Pull request #4039 (2020-02-11 05:38:29)

Separate single Actions workflow into multiple workflows
Depends on #4033

## Motivation

If any job fails in the current GH Actions workflow, a re-run on the same
commit SHA requires re-running *all* jobs--regardless if the job already
passed in the previous run.

This can be problematic when dealing with flakiness in the integration tests.

If a test fails due to flakiness in `cloud_integration_tests`, all the unit
tests, static checks, and `kind_integration_tests` are re-run which leads to
lots of waiting and dealing with the possibility of flakiness in earlier jobs.

With this change, individual workflows can now be re-run without triggering
all other jobs to complete again first.

## Solution

`workflow.yml` is now split into:
- `static_checks.yml`
- `unit_tests.yml`
- `kind_integration.yml`
- `cloud_integration.yml`

### Workflows

`statc_checks.yml` performs checks related to dependencies, linting, and
formatting.

`unit_tests.yml` performs the Go and JS unit tests.

`kind_integration.yml` builds the images (on Packet or the GH Action VM) and
runs the integration tests on a KinD cluster. This workflow continues to run
for **all** PRs and pushes to `master` and tags.

`cloud_integration.yml` builds the images only on Packet. This is because
forked repositories do not need to trigger this workflow. It then creates a
unique GKE cluster and runs the integration tests on the cluster.

### The actual flow of work..

A forked repository or non-forked repository opening a PR triggers:
- `static_checks`
- `unit_tests`
- `kind_integration_tests`

These workflows all run in parallel and are invidivually re-runnable.

A push to `master` or tags triggers:
- `static_checks`
- `unit_tests`
- `kind_integration_tests`
- `cloud_integration_tests`

These workflows also all run in parallel, including the `docker_build` step of
both integration test workflows. This has not conflicted in testing as it
takes place on the same Packet host and just utilizes docker layer caching
well.

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>


kleimkuhler: In #4033 `Go Format` was renamed to `Go format` and `Validate go deps` was renamed to `Go dependencies`, so these required checks will not actually run. Those checks will be updated once it merges.
alpeb: Thinking about the upcoming automated release work, what do you think about moving `chart_deploy` into a separate workflow that would be triggered by tags with the shape `(edge-.*|stable-*)`? We'd remove the `tags: *` trigger from all the other workflows, and so when we tag the release (after master would have gone green in the other workflows) only this new workflow would get triggered, unlike the current situation where we have to wait for everything to run again.
This new workflow would also contain the upcoming automated release jobs.
alpeb: ... to be more precise about the current situation, after the changelog changes have merged, we have to  manually cancel the build, and then tag, and it's upon this last event that the final tests are run and the release (currently just the Helm artifact) is pushed. If that last run fails, that might require pushing a fix, and some of us are familiar with the untagging/tagging business that requires :unamused: 
What I'm thinking is that we leave that first merge CI run to continue, and if it's green, tag the release to trigger the release steps.
I'm getting a little ahead into the release automation work here, but I think laying out the workflows like I described could be a good starting point.
kleimkuhler: > what do you think about moving chart_deploy into a separate workflow ..

So this would relax the requirement that the integration tests pass for the tag (whether it's edge or stable) since the workflow would run right away instead of waiting for `cloud_integration_tests` to complete like it does currently.

However, as you also mention the the first merge would just continue, and once it's green we *know* that tagging that commit as a release is okay.

I would be okay with that change and think it makes sense! A push to master in order to release an edge or stable shouldn't require cancelling one CI run in order to let the other (tagged) workflow run. For automating releases, it would be a lot easier if the workflow was what you are suggesting and `chart_deploy` was specific to those `(edge-.*|stable-*)` conditions.
kleimkuhler: > iiuc, this could push docker images with edge/stable release tags, even if unit or integration tests are failing. i don't think this push should happen until all tests have passed.

@siggy and I talked about this directly and decided this okay to keep around for this change. This behavior exists in the current workflow file, and this change just persists that behavior.

The possibility of tags being pushed and integration later failing should be addressed however as part of the future release automation work.

As @alpeb mentioned in this [comment](https://github.com/linkerd/linkerd2/pull/4039#issuecomment-584791082), `chart_deploy` and other "release only" jobs should be moved to a release workflow that is responsible for pushing tagged images, deploying helm charts, etc.
siggy: To close the loop, I'm onboard moving forward with this change, provided we ensure all tests are passing before any docker images tagged as `edge` or `stable` are pushed to a public repository. This requirement could be enforced initially via careful documentation and deploy instructions, but in the long run this should be enforced programmatically.

Also agree it would be nice if `chart_deploy` and all other release-specific steps were separated out into their own workflow. This is a secondary concern compared to ensuring a release workflow only runs after all tests have passed.

Patch:
diff --git a/.github/workflows/cloud_integration.yml b/.github/workflows/cloud_integration.yml
new file mode 100644
index 000000000..6e3756b01
--- /dev/null
+++ b/.github/workflows/cloud_integration.yml
@@ -0,0 +1,134 @@
+name: Cloud integration
+on:
+  push:
+    branches:
+    - master
+    tags:
+    - "*"
+jobs:
+  docker_build:
+    name: Docker build
+    runs-on: ubuntu-18.04
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Set environment variables from scripts
+      run: |
+        . bin/_tag.sh
+        echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag)
+    - name: Setup SSH config for Packet
+      run: |
+        mkdir -p ~/.ssh/
+        touch ~/.ssh/id && chmod 600 ~/.ssh/id
+        echo "${{ secrets.DOCKER_SSH_CONFIG }}"  > ~/.ssh/config
+        echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id
+        echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
+        ssh linkerd-docker docker version
+    - name: Build docker images
+      env:
+        DOCKER_HOST: ssh://linkerd-docker
+        DOCKER_TRACE: 1
+      run: |
+        export PATH="`pwd`/bin:$PATH"
+        bin/docker-build
+  docker_push:
+    name: Docker push
+    runs-on: ubuntu-18.04
+    needs: [docker_build]
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Set environment variables from scripts
+      run: |
+        . bin/_tag.sh
+        echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag)
+    - name: Configure gcloud
+      uses: linkerd/linkerd2-action-gcloud@v1.0.0
+      with:
+        cloud_sdk_service_account_key: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }}
+        gcp_project: ${{ secrets.GCP_PROJECT }}
+        gcp_zone: ${{ secrets.GCP_ZONE }}
+    - name: Docker SSH setup
+      run: |
+        mkdir -p ~/.ssh/
+        touch ~/.ssh/id && chmod 600 ~/.ssh/id
+        echo "${{ secrets.DOCKER_SSH_CONFIG }}"  > ~/.ssh/config
+        echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id
+        echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
+        ssh linkerd-docker docker version
+    - name: Push docker images to registry
+      env:
+        DOCKER_HOST: ssh://linkerd-docker
+      run: |
+        export PATH="`pwd`/bin:$PATH"
+        bin/docker-push-deps
+        bin/docker-push $TAG
+        bin/docker-retag-all $TAG master
+        bin/docker-push master
+  cloud_integration_tests:
+    name: Cloud integration tests
+    runs-on: ubuntu-18.04
+    needs: [docker_push]
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Install linkerd CLI
+      id: install_cli
+      run: |
+        TAG="$(CI_FORCE_CLEAN=1 bin/root-tag)"
+        image="gcr.io/linkerd-io/cli-bin:$TAG"
+        id=$(bin/docker create $image)
+        bin/docker cp "$id:/out/linkerd-linux" "$HOME/.linkerd"
+        $HOME/.linkerd version --client
+        # validate CLI version matches the repo
+        [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]]
+        echo "Installed Linkerd CLI version: $TAG"
+        echo "::set-output name=tag::$TAG"
+    - name: Create GKE cluster
+      uses: linkerd/linkerd2-action-gcloud@v1.0.0
+      with:
+        cloud_sdk_service_account_key: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }}
+        gcp_project: ${{ secrets.GCP_PROJECT }}
+        gcp_zone: ${{ secrets.GCP_ZONE }}
+        create: true
+        name: testing-${{ steps.install_cli.outputs.tag }}-${{ github.run_id }}
+    - name: Run integration tests
+      env:
+        GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
+      run: |
+        export PATH="`pwd`/bin:$PATH"
+        echo "$GITCOOKIE_SH" | bash
+        version="$($HOME/.linkerd version --client --short | tr -cd '[:alnum:]-')"
+        bin/test-run $HOME/.linkerd linkerd-$version
+    - name: CNI tests
+      run: |
+        export TAG="$($HOME/.linkerd version --client --short)"
+        go test -cover -race -v -mod=readonly ./cni-plugin/test -integration-tests
+  chart_deploy:
+    name: Helm chart deploy
+    if: startsWith(github.ref, 'refs/tags')
+    runs-on: ubuntu-18.04
+    needs: [cloud_integration_tests]
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Configure gsutils
+      uses: linkerd/linkerd2-action-gcloud@v1.0.0
+      with:
+        cloud_sdk_service_account_key: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }}
+        gcp_project: ${{ secrets.GCP_PROJECT }}
+        gcp_zone: ${{ secrets.GCP_ZONE }}
+    - name: Edge Helm chart creation and upload
+      if: startsWith(github.ref, 'refs/tags/edge')
+      run: |
+        mkdir -p target/helm
+        gsutil cp gs://helm.linkerd.io/edge/index.yaml target/helm/index-pre.yaml
+        bin/helm-build package
+        gsutil rsync target/helm gs://helm.linkerd.io/edge
+    - name: Stable Helm chart creation and upload
+      if: startsWith(github.ref, 'refs/tags/stable')
+      run: |
+        mkdir -p target/helm
+        gsutil cp gs://helm.linkerd.io/stable/index.yaml target/helm/index-pre.yaml
+        bin/helm-build package
+        gsutil rsync target/helm gs://helm.linkerd.io/stable
diff --git a/.github/workflows/kind_integration.yml b/.github/workflows/kind_integration.yml
new file mode 100644
index 000000000..dfb07ce1a
--- /dev/null
+++ b/.github/workflows/kind_integration.yml
@@ -0,0 +1,176 @@
+name: KinD integration
+on:
+  pull_request: {}
+  push:
+    branches:
+    - master
+    tags:
+    - "*"
+jobs:
+  docker_build:
+    name: Docker build
+    runs-on: ubuntu-18.04
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Set environment variables from scripts
+      run: |
+        . bin/_tag.sh
+        echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag)
+
+        . bin/_docker.sh
+        echo ::set-env name=DOCKER_REGISTRY::$DOCKER_REGISTRY
+    - name: Setup SSH config for Packet
+      if: github.event_name == 'push' || !github.event.pull_request.head.repo.fork
+      run: |
+        mkdir -p ~/.ssh/
+        touch ~/.ssh/id && chmod 600 ~/.ssh/id
+        echo "${{ secrets.DOCKER_SSH_CONFIG }}"  > ~/.ssh/config
+        echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id
+        echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
+        ssh linkerd-docker docker version
+        echo ::set-env name=DOCKER_HOST::ssh://linkerd-docker
+    - name: Build docker images
+      env:
+        DOCKER_TRACE: 1
+      run: |
+        export PATH="`pwd`/bin:$PATH"
+        bin/docker-build
+    - name: Create artifact with CLI and image archives (Forked repositories)
+      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
+      env:
+        ARCHIVES: /home/runner/archives
+      run: |
+        mkdir -p $ARCHIVES
+
+        for image in proxy controller web cni-plugin debug cli-bin grafana; do
+          docker save "$DOCKER_REGISTRY/$image:$TAG" > $ARCHIVES/$image.tar || tee save_fail &
+        done
+
+        # Wait for `docker save` background processes to complete. Exit early
+        # if any job failed.
+        wait < <(jobs -p)
+        test -f save_fail && exit 1 || true
+    # `with.path` values do not support environment variables yet, so an
+    # absolute path is used here.
+    #
+    # https://github.com/actions/upload-artifact/issues/8
+    - name: Upload artifact (Forked repositories)
+      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
+      uses: actions/upload-artifact@v1
+      with:
+        name: image-archives
+        path: /home/runner/archives
+  kind_integration_tests:
+    strategy:
+      matrix:
+        integration_test: [deep, upgrade, helm, helm_upgrade, custom_domain, external_issuer]
+    needs: [docker_build]
+    name: Integration tests (${{ matrix.integration_test }})
+    runs-on: ubuntu-18.04
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Try to load cached Go modules
+      uses: actions/cache@v1
+      with:
+        path: ~/go/pkg/mod
+        key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-go-
+    - name: Set environment variables from scripts
+      run: |
+        . bin/_tag.sh
+        echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag)
+
+        . bin/_docker.sh
+        echo ::set-env name=DOCKER_REGISTRY::$DOCKER_REGISTRY
+    - name: Setup SSH config for Packet
+      if: github.event_name == 'push' || !github.event.pull_request.head.repo.fork
+      run: |
+        mkdir -p ~/.ssh/
+        touch ~/.ssh/id && chmod 600 ~/.ssh/id
+        echo "${{ secrets.DOCKER_SSH_CONFIG }}"  > ~/.ssh/config
+        echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id
+        echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
+    - name: Download image archives (Forked repositories)
+      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
+      uses: actions/download-artifact@v1
+      with:
+        name: image-archives
+    - name: Load cli-bin image into local docker images
+      if: github.event_name == 'push' || !github.event.pull_request.head.repo.fork
+      run: |
+        # `docker load` only accepts input from STDIN, so pipe the image
+        # archive into the command.
+        #
+        # In order to pipe the image archive, set `DOCKER_HOST` for a single
+        # command and `docker save` the CLI image from the Packet host.
+        DOCKER_HOST=ssh://linkerd-docker docker save "$DOCKER_REGISTRY/cli-bin:$TAG" | docker load
+    - name: Load cli-bin image into local docker images (Forked repositories)
+      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
+      run: docker load < image-archives/cli-bin.tar
+    - name: Install CLI
+      run: |
+        # Copy the CLI out of the local cli-bin container.
+        container_id=$(docker create "$DOCKER_REGISTRY/cli-bin:$TAG")
+        docker cp $container_id:/out/linkerd-linux $HOME/.linkerd
+
+        # Validate the CLI version matches the current build tag.
+        [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]]
+    - name: Setup default KinD cluster
+      if: matrix.integration_test != 'custom_domain'
+      uses: engineerd/setup-kind@v0.3.0
+      with:
+        version: "v0.6.1"
+    - name: Setup custom_domain KinD cluster
+      if: matrix.integration_test == 'custom_domain'
+      uses: engineerd/setup-kind@v0.3.0
+      with:
+        config: test/testdata/custom_cluster_domain_config.yaml
+        version: "v0.6.1"
+    - name: Load image archives into the local KinD cluster
+      if: github.event_name == 'push' || !github.event.pull_request.head.repo.fork
+      env:
+        PROXY_INIT_IMAGE_NAME: gcr.io/linkerd-io/proxy-init:v1.3.1
+        PROMETHEUS_IMAGE_NAME: prom/prometheus:v2.15.2
+      run: |
+        # For each container, load the image archive into the KinD cluster.
+        #
+        # `kind load` cannot take input from STDIN, so `<(command)` syntax is
+        # used to load the output into the KinD cluster. Set `DOCKER_HOST` for
+        # a single command, and `docker save` the container from the Packet
+        # host.
+        for image in proxy controller web cni-plugin debug grafana; do
+          kind load image-archive <(DOCKER_HOST=ssh://linkerd-docker docker save "$DOCKER_REGISTRY/$image:$TAG") || tee load_fail &
+        done
+
+        # Wait for `kind load` background processes to complete. Exit early if
+        # any job failed.
+        wait < <(jobs -p)
+        test -f load_fail && exit 1 || true
+
+        # Load proxy-init and prometheus images into KinD while it is
+        # available. Allow these commands to fail since they will be cached
+        # for the next run.
+        kind load image-archive <(DOCKER_HOST=ssh://linkerd-docker docker save $PROXY_INIT_IMAGE_NAME) 2>&1 || true
+        kind load image-archive <(DOCKER_HOST=ssh://linkerd-docker docker save $PROMETHEUS_IMAGE_NAME) 2>&1 || true
+    - name: Load image archives into the local KinD cluster (Forked repositories)
+      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
+      run: |
+        for image in proxy controller web cni-plugin debug grafana; do
+          kind load image-archive image-archives/$image.tar || tee load_fail &
+        done
+
+        # Wait for `kind load` background processes to complete. Exit early if
+        # any job failed.
+        wait < <(jobs -p)
+        test -f load_fail && exit 1 || true
+    - name: Run integration tests
+      run: |
+        # Export `init_test_run` and `*_integration_tests` into the
+        # environment.
+        . bin/_test-run.sh
+
+        init_test_run $HOME/.linkerd
+        ${{ matrix.integration_test }}_integration_tests
diff --git a/.github/workflows/static_checks.yml b/.github/workflows/static_checks.yml
new file mode 100644
index 000000000..d69dd9f5e
--- /dev/null
+++ b/.github/workflows/static_checks.yml
@@ -0,0 +1,59 @@
+name: Static checks
+on:
+  pull_request: {}
+  push:
+    branches:
+    - master
+    tags:
+    - "*"
+jobs:
+  go_dependencies:
+    name: Go dependencies
+    runs-on: ubuntu-18.04
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Dump env
+      run: env | sort
+    - name: Dump GitHub context
+      env:
+        GITHUB_CONTEXT: ${{ toJson(github) }}
+      run: echo "$GITHUB_CONTEXT"
+    - name: Dump job context
+      env:
+        JOB_CONTEXT: ${{ toJson(job) }}
+      run: echo "$JOB_CONTEXT"
+    - name: Validate go deps
+      run: |
+        . bin/_tag.sh
+        for f in $( grep -lR --include=Dockerfile\* go-deps: . ) ; do
+          validate_go_deps_tag $f
+        done
+  go_lint:
+    name: Go lint
+    runs-on: ubuntu-18.04
+    container:
+      image: golang:1.13.4
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Go lint
+      env:
+        GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
+      run: |
+        echo "$GITCOOKIE_SH" | bash
+        bin/lint --verbose
+  go_format:
+    name: Go format
+    runs-on: ubuntu-18.04
+    container:
+      image: golang:1.13.4
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Format
+      env:
+        GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
+      run: |
+        echo "$GITCOOKIE_SH" | bash
+        bin/fmt
diff --git a/.github/workflows/workflow.yml b/.github/workflows/workflow.yml
deleted file mode 100644
index 3374cb5c5..000000000
--- a/.github/workflows/workflow.yml
+++ /dev/null
@@ -1,417 +0,0 @@
-name: CI
-
-on:
-  pull_request: {}
-  push:
-    branches:
-    - master
-    tags:
-    - "*"
-
-# Jobs by event type and dependencies:
-
-# Unit tests for every master/tag push and PR
-#
-# go_dependencies
-# go_unit_tests
-# go_lint
-# js_unit_tests
-
-# All master/tag pushes and PRs
-#
-# docker_build
-# -> kind_integration_tests
-
-# Docker push and cloud integration tests for every master/tag push
-#
-#     -> docker_push
-#        -> cloud_integration_tests
-
-jobs:
-
-  #
-  # Unit tests run for:
-  # - every master push
-  # - every tag push
-  # - every PR
-  #
-
-  go_dependencies:
-    name: Go dependencies
-    runs-on: ubuntu-18.04
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Dump env
-      run: env | sort
-    - name: Dump GitHub context
-      env:
-        GITHUB_CONTEXT: ${{ toJson(github) }}
-      run: echo "$GITHUB_CONTEXT"
-    - name: Dump job context
-      env:
-        JOB_CONTEXT: ${{ toJson(job) }}
-      run: echo "$JOB_CONTEXT"
-    - name: Validate go deps
-      run: |
-        . bin/_tag.sh
-        for f in $( grep -lR --include=Dockerfile\* go-deps: . ) ; do
-          validate_go_deps_tag $f
-        done
-
-  go_unit_tests:
-    name: Go unit tests
-    runs-on: ubuntu-18.04
-    container:
-      image: golang:1.13.4
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Go unit tests
-      env:
-        GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
-      run: |
-        echo "$GITCOOKIE_SH" | bash
-        # TODO: validate bin/protoc-go.sh does not dirty the repo
-        go test -cover -race -v -mod=readonly ./...
-
-  go_lint:
-    name: Go lint
-    runs-on: ubuntu-18.04
-    container:
-      image: golang:1.13.4
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Go lint
-      env:
-        GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
-      run: |
-        echo "$GITCOOKIE_SH" | bash
-        bin/lint --verbose
-
-  go_format:
-    name: Go format
-    runs-on: ubuntu-18.04
-    container:
-      image: golang:1.13.4
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Format
-      env:
-        GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
-      run: |
-        echo "$GITCOOKIE_SH" | bash
-        bin/fmt
-
-  js_unit_tests:
-    name: JS unit tests
-    runs-on: ubuntu-18.04
-    container:
-      image: node:10.16.0-stretch
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Yarn setup
-      run: curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version 1.21.1 --network-concurrency 1
-    - name: JS unit tests
-      run: |
-        export PATH="$HOME/.yarn/bin:$PATH"
-        export NODE_ENV=test
-        bin/web
-        bin/web test --reporters=jest-dot-reporter
-
-  #
-  # Docker build and kind integration tests run for:
-  # - every master push
-  # - every tag push
-  # - every PR
-  #
-
-  docker_build:
-    name: Docker build
-    runs-on: ubuntu-18.04
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Set environment variables from scripts
-      run: |
-        . bin/_tag.sh
-        echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag)
-
-        . bin/_docker.sh
-        echo ::set-env name=DOCKER_REGISTRY::$DOCKER_REGISTRY
-    - name: Setup SSH config for Packet
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
-      run: |
-        mkdir -p ~/.ssh/
-        touch ~/.ssh/id && chmod 600 ~/.ssh/id
-        echo "${{ secrets.DOCKER_SSH_CONFIG }}"  > ~/.ssh/config
-        echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id
-        echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
-        ssh linkerd-docker docker version
-        echo ::set-env name=DOCKER_HOST::ssh://linkerd-docker
-    - name: Build docker images
-      env:
-        DOCKER_TRACE: 1
-      run: |
-        export PATH="`pwd`/bin:$PATH"
-        bin/docker-build
-    - name: Create artifact with CLI and image archives (Forked repositories)
-      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
-      env:
-        ARCHIVES: /home/runner/archives
-      run: |
-        mkdir -p $ARCHIVES
-
-        for image in proxy controller web cni-plugin debug cli-bin grafana; do
-          docker save "$DOCKER_REGISTRY/$image:$TAG" > $ARCHIVES/$image.tar || tee save_fail &
-        done
-
-        # Wait for `docker save` background processes to complete. Exit early
-        # if any job failed.
-        wait < <(jobs -p)
-
-        test -f save_fail && exit 1 || true
-    # `with.path` values do not support environment variables yet, so an
-    # absolute path is used here.
-    #
-    # https://github.com/actions/upload-artifact/issues/8
-    - name: Upload artifact (Forked repositories)
-      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
-      uses: actions/upload-artifact@v1
-      with:
-        name: image-archives
-        path: /home/runner/archives
-
-  kind_integration_tests:
-    strategy:
-      matrix:
-        integration_test: [deep, upgrade, helm, helm_upgrade, custom_domain, external_issuer]
-    needs: [docker_build]
-    name: Integration tests (${{ matrix.integration_test }})
-    runs-on: ubuntu-18.04
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Try to load cached Go modules
-      uses: actions/cache@v1
-      with:
-        path: ~/go/pkg/mod
-        key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
-        restore-keys: |
-          ${{ runner.os }}-go-
-    - name: Set environment variables from scripts
-      run: |
-        . bin/_tag.sh
-        echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag)
-
-        . bin/_docker.sh
-        echo ::set-env name=DOCKER_REGISTRY::$DOCKER_REGISTRY
-    - name: Setup SSH config for Packet
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
-      run: |
-        mkdir -p ~/.ssh/
-        touch ~/.ssh/id && chmod 600 ~/.ssh/id
-        echo "${{ secrets.DOCKER_SSH_CONFIG }}"  > ~/.ssh/config
-        echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id
-        echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
-    - name: Download image archives (Forked repositories)
-      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
-      uses: actions/download-artifact@v1
-      with:
-        name: image-archives
-    - name: Load cli-bin image into local docker images
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
-      run: |
-        # `docker load` only accepts input from STDIN, so pipe the image
-        # archive into the command.
-        #
-        # In order to pipe the image archive, set `DOCKER_HOST` for a single
-        # command and `docker save` the CLI image from the Packet host.
-        DOCKER_HOST=ssh://linkerd-docker docker save "$DOCKER_REGISTRY/cli-bin:$TAG" | docker load
-    - name: Load cli-bin image into local docker images (Forked repositories)
-      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
-      run: docker load < image-archives/cli-bin.tar
-    - name: Install CLI
-      run: |
-        # Copy the CLI out of the local cli-bin container.
-        container_id=$(docker create "$DOCKER_REGISTRY/cli-bin:$TAG")
-        docker cp $container_id:/out/linkerd-linux $HOME/.linkerd
-
-        # Validate the CLI version matches the current build tag.
-        [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]]
-    - name: Setup default KinD cluster
-      if: matrix.integration_test != 'custom_domain'
-      uses: engineerd/setup-kind@v0.3.0
-      with:
-        version: "v0.6.1"
-    - name: Setup custom_domain KinD cluster
-      if: matrix.integration_test == 'custom_domain'
-      uses: engineerd/setup-kind@v0.3.0
-      with:
-        config: test/testdata/custom_cluster_domain_config.yaml
-        version: "v0.6.1"
-    - name: Load image archives into the local KinD cluster
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
-      env:
-        PROXY_INIT_IMAGE_NAME: gcr.io/linkerd-io/proxy-init:v1.3.1
-        PROMETHEUS_IMAGE_NAME: prom/prometheus:v2.15.2
-      run: |
-        # For each container, load the image archive into the KinD cluster.
-        #
-        # `kind load` cannot take input from STDIN, so `<(command)` syntax is
-        # used to load the output into the KinD cluster. Set `DOCKER_HOST` for
-        # a single command, and `docker save` the container from the Packet
-        # host.
-        for image in proxy controller web cni-plugin debug grafana; do
-          kind load image-archive <(DOCKER_HOST=ssh://linkerd-docker docker save "$DOCKER_REGISTRY/$image:$TAG") || tee load_fail &
-        done
-
-        # Wait for `kind load` background processes to complete. Exit early if
-        # any job failed.
-        wait < <(jobs -p)
-        test -f load_fail && exit 1 || true
-
-        # Load proxy-init and prometheus images into KinD while it is
-        # available. Allow these commands to fail since they will be cached
-        # for the next run.
-        kind load image-archive <(DOCKER_HOST=ssh://linkerd-docker docker save $PROXY_INIT_IMAGE_NAME) 2>&1 || true
-        kind load image-archive <(DOCKER_HOST=ssh://linkerd-docker docker save $PROMETHEUS_IMAGE_NAME) 2>&1 || true
-    - name: Load image archives into the local KinD cluster (Forked repositories)
-      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
-      run: |
-        for image in proxy controller web cni-plugin debug grafana; do
-          kind load image-archive image-archives/$image.tar || tee load_fail &
-        done
-
-        # Wait for `kind load` background processes to complete. Exit early if
-        # any job failed.
-        wait < <(jobs -p)
-        test -f load_fail && exit 1 || true
-    - name: Run integration tests
-      run: |
-        # Export `init_test_run` and `*_integration_tests` into the
-        # environment.
-        . bin/_test-run.sh
-
-        init_test_run $HOME/.linkerd
-        ${{ matrix.integration_test }}_integration_tests
-
-  #
-  # Docker push and cloud integration tests run for:
-  # - every master push
-  # - every tag push
-  #
-
-  docker_push:
-    name: Docker push
-    if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
-    runs-on: ubuntu-18.04
-    needs: [go_dependencies, go_unit_tests, go_lint, js_unit_tests, kind_integration_tests]
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Set environment variables from scripts
-      run: |
-        . bin/_tag.sh
-        echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag)
-    - name: Configure gcloud
-      uses: linkerd/linkerd2-action-gcloud@v1.0.0
-      with:
-        cloud_sdk_service_account_key: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }}
-        gcp_project: ${{ secrets.GCP_PROJECT }}
-        gcp_zone: ${{ secrets.GCP_ZONE }}
-    - name: Docker SSH setup
-      run: |
-        mkdir -p ~/.ssh/
-        touch ~/.ssh/id && chmod 600 ~/.ssh/id
-        echo "${{ secrets.DOCKER_SSH_CONFIG }}"  > ~/.ssh/config
-        echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id
-        echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
-        ssh linkerd-docker docker version
-    - name: Push docker images to registry
-      env:
-        DOCKER_HOST: ssh://linkerd-docker
-      run: |
-        export PATH="`pwd`/bin:$PATH"
-        bin/docker-push-deps
-        bin/docker-push $TAG
-        bin/docker-retag-all $TAG master
-        bin/docker-push master
-
-  cloud_integration_tests:
-    name: Cloud integration tests
-    if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
-    runs-on: ubuntu-18.04
-    needs: [docker_push]
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Install linkerd CLI
-      id: install_cli
-      run: |
-        TAG="$(CI_FORCE_CLEAN=1 bin/root-tag)"
-        image="gcr.io/linkerd-io/cli-bin:$TAG"
-        id=$(bin/docker create $image)
-        bin/docker cp "$id:/out/linkerd-linux" "$HOME/.linkerd"
-        $HOME/.linkerd version --client
-        # validate CLI version matches the repo
-        [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]]
-        echo "Installed Linkerd CLI version: $TAG"
-        echo "::set-output name=tag::$TAG"
-    - name: Create GKE cluster
-      uses: linkerd/linkerd2-action-gcloud@v1.0.0
-      with:
-        cloud_sdk_service_account_key: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }}
-        gcp_project: ${{ secrets.GCP_PROJECT }}
-        gcp_zone: ${{ secrets.GCP_ZONE }}
-        create: true
-        name: testing-${{ steps.install_cli.outputs.tag }}-${{ github.run_id }}
-    - name: Run integration tests
-      env:
-        GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
-      run: |
-        export PATH="`pwd`/bin:$PATH"
-        echo "$GITCOOKIE_SH" | bash
-        version="$($HOME/.linkerd version --client --short | tr -cd '[:alnum:]-')"
-        bin/test-run $HOME/.linkerd linkerd-$version
-    - name: CNI tests
-      run: |
-        export TAG="$($HOME/.linkerd version --client --short)"
-        go test -cover -race -v -mod=readonly ./cni-plugin/test -integration-tests
-
-  #
-  # Helm chart artifact deploy run for:
-  # - every tag push
-  #
-
-  chart_deploy:
-    name: Helm chart deploy
-    if: startsWith(github.ref, 'refs/tags')
-    runs-on: ubuntu-18.04
-    needs: [cloud_integration_tests]
-    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: Configure gsutils
-      uses: linkerd/linkerd2-action-gcloud@v1.0.0
-      with:
-        cloud_sdk_service_account_key: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }}
-        gcp_project: ${{ secrets.GCP_PROJECT }}
-        gcp_zone: ${{ secrets.GCP_ZONE }}
-    - name: Edge Helm chart creation and upload
-      if: startsWith(github.ref, 'refs/tags/edge')
-      run: |
-        mkdir -p target/helm
-        gsutil cp gs://helm.linkerd.io/edge/index.yaml target/helm/index-pre.yaml
-        bin/helm-build package
-        gsutil rsync target/helm gs://helm.linkerd.io/edge
-    - name: Stable Helm chart creation and upload
-      if: startsWith(github.ref, 'refs/tags/stable')
-      run: |
-        mkdir -p target/helm
-        gsutil cp gs://helm.linkerd.io/stable/index.yaml target/helm/index-pre.yaml
-        bin/helm-build package
-        gsutil rsync target/helm gs://helm.linkerd.io/stable
diff --git a/cli/cmd/upgrade.go b/cli/cmd/upgrade.go
index 1ae754be7..19470eb24 100644
--- a/cli/cmd/upgrade.go
+++ b/cli/cmd/upgrade.go
@@ -7,6 +7,7 @@ import (
 	"io/ioutil"
 	"os"
 	"strings"
+	"time"
 
 	"github.com/linkerd/linkerd2/pkg/config"
 	"github.com/linkerd/linkerd2/pkg/issuercerts"
@@ -399,7 +400,7 @@ func ensureIssuerCertWorksWithAllProxies(k kubernetes.Interface, cred *tls.Cred)
 		roots, err := tls.DecodePEMCertPool(pod.Anchors)
 
 		if roots != nil {
-			err = cred.Verify(roots, "")
+			err = cred.Verify(roots, "", time.Time{})
 		}
 
 		if err != nil {
@@ -546,7 +547,7 @@ func verifyWebhookTLS(value *charts.TLS, webhook string) error {
 		return err
 	}
 	roots := crt.CertPool()
-	if err := crt.Verify(roots, webhookCommonName(webhook)); err != nil {
+	if err := crt.Verify(roots, webhookCommonName(webhook), time.Time{}); err != nil {
 		return err
 	}
 
diff --git a/pkg/healthcheck/healthcheck.go b/pkg/healthcheck/healthcheck.go
index 27cdd4d38..c8d3a688b 100644
--- a/pkg/healthcheck/healthcheck.go
+++ b/pkg/healthcheck/healthcheck.go
@@ -972,7 +972,7 @@ func (hc *HealthChecker) allCategories() []category {
 					description: "issuer cert is issued by the trust root",
 					hintAnchor:  "l5d-identity-issuer-cert-issued-by-trust-root",
 					check: func(ctx context.Context) error {
-						return hc.issuerCert.Verify(tls.CertificatesToPool(hc.roots), hc.issuerIdentity())
+						return hc.issuerCert.Verify(tls.CertificatesToPool(hc.roots), hc.issuerIdentity(), time.Time{})
 					},
 				},
 			},
diff --git a/pkg/identity/service.go b/pkg/identity/service.go
index 1914832e3..208eefefa 100644
--- a/pkg/identity/service.go
+++ b/pkg/identity/service.go
@@ -29,6 +29,7 @@ const (
 	EnvTrustAnchors  = "LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS"
 	eventTypeSkipped = "IssuerUpdateSkipped"
 	eventTypeUpdated = "IssuerUpdated"
+	eventTypeFailed  = "IssuerValidationFailed"
 )
 
 type (
@@ -113,7 +114,7 @@ func (svc *Service) loadCredentials() (tls.Issuer, error) {
 		return nil, fmt.Errorf("failed to read CA from disk: %s", err)
 	}
 
-	if err := creds.Crt.Verify(svc.trustAnchors, svc.expectedName); err != nil {
+	if err := creds.Crt.Verify(svc.trustAnchors, svc.expectedName, time.Time{}); err != nil {
 		return nil, fmt.Errorf("failed to verify issuer credentials for '%s' with trust anchors: %s", svc.expectedName, err)
 	}
 
@@ -148,7 +149,7 @@ func (svc *Service) ensureIssuerStillValid() error {
 	issuer := *svc.issuer
 	switch is := issuer.(type) {
 	case *tls.CA:
-		return is.Cred.Verify(svc.trustAnchors, svc.expectedName)
+		return is.Cred.Verify(svc.trustAnchors, svc.expectedName, time.Time{})
 	default:
 		return fmt.Errorf("unsupported issuer type. Expected *tls.CA, got %v", is)
 	}
@@ -171,7 +172,9 @@ func (svc *Service) Certify(ctx context.Context, req *pb.CertifyRequest) (*pb.Ce
 	}
 
 	if err := svc.ensureIssuerStillValid(); err != nil {
-		log.Errorf("could not process CSR because of CA cert validation failure: %s", err)
+		log.Errorf("could not process CSR because of CA cert validation failure: %s - CSR Identity : %s", err, reqIdentity)
+		message := fmt.Sprintf("%s - CSR Identity : %s", err.Error(), reqIdentity)
+		svc.recordEvent(v1.EventTypeWarning, eventTypeFailed, message)
 		return nil, err
 	}
 
diff --git a/pkg/issuercerts/issuercerts.go b/pkg/issuercerts/issuercerts.go
index ad1b853f6..50262d7df 100644
--- a/pkg/issuercerts/issuercerts.go
+++ b/pkg/issuercerts/issuercerts.go
@@ -169,7 +169,7 @@ func (ic *IssuerCertData) VerifyAndBuildCreds(dnsName string) (*tls.Cred, error)
 		return nil, err
 	}
 
-	if err := creds.Verify(roots, dnsName); err != nil {
+	if err := creds.Verify(roots, dnsName, time.Time{}); err != nil {
 		return nil, err
 	}
 
diff --git a/pkg/tls/cred.go b/pkg/tls/cred.go
index d46218a6c..d88da77fc 100644
--- a/pkg/tls/cred.go
+++ b/pkg/tls/cred.go
@@ -10,6 +10,7 @@ import (
 	"errors"
 	"fmt"
 	"io/ioutil"
+	"time"
 )
 
 type (
@@ -83,13 +84,21 @@ func (crt *Crt) CertPool() *x509.CertPool {
 }
 
 // Verify the validity of the provided certificate
-func (crt *Crt) Verify(roots *x509.CertPool, name string) error {
+func (crt *Crt) Verify(roots *x509.CertPool, name string, currentTime time.Time) error {
 	i := x509.NewCertPool()
 	for _, c := range crt.TrustChain {
 		i.AddCert(c)
 	}
-	vo := x509.VerifyOptions{Roots: roots, Intermediates: i, DNSName: name}
+	vo := x509.VerifyOptions{Roots: roots, Intermediates: i, DNSName: name, CurrentTime: currentTime}
 	_, err := crt.Certificate.Verify(vo)
+
+	if currentTime.IsZero() {
+		currentTime = time.Now()
+	}
+
+	if crtExpiryError(err) {
+		return fmt.Errorf("%s - Current Time : %s - Invalid before %s - Invalid After %s", err, currentTime, crt.Certificate.NotBefore, crt.Certificate.NotAfter)
+	}
 	return err
 }
 
@@ -221,3 +230,12 @@ func DecodePEMCrt(txt string) (*Crt, error) {
 
 	return &crt, nil
 }
+
+func crtExpiryError(err error) bool {
+	switch v := err.(type) {
+	case x509.CertificateInvalidError:
+		return v.Reason == x509.Expired
+	default:
+		return false
+	}
+}


Test patch:
diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml
new file mode 100644
index 000000000..cc273c448
--- /dev/null
+++ b/.github/workflows/unit_tests.yml
@@ -0,0 +1,40 @@
+name: Unit tests
+on:
+  pull_request: {}
+  push:
+    branches:
+    - master
+    tags:
+    - "*"
+jobs:
+  go_unit_tests:
+    name: Go unit tests
+    runs-on: ubuntu-18.04
+    container:
+      image: golang:1.13.4
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Go unit tests
+      env:
+        GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
+      run: |
+        echo "$GITCOOKIE_SH" | bash
+        # TODO: validate bin/protoc-go.sh does not dirty the repo
+        go test -cover -race -v -mod=readonly ./...
+  js_unit_tests:
+    name: JS unit tests
+    runs-on: ubuntu-18.04
+    container:
+      image: node:10.16.0-stretch
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Yarn setup
+      run: curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version 1.21.1 --network-concurrency 1
+    - name: JS unit tests
+      run: |
+        export PATH="$HOME/.yarn/bin:$PATH"
+        export NODE_ENV=test
+        bin/web
+        bin/web test --reporters=jest-dot-reporter
diff --git a/pkg/tls/cred_test.go b/pkg/tls/cred_test.go
index 576bf942c..a888cc6c0 100644
--- a/pkg/tls/cred_test.go
+++ b/pkg/tls/cred_test.go
@@ -1,7 +1,10 @@
 package tls
 
 import (
+	"crypto/x509"
+	"fmt"
 	"testing"
+	"time"
 )
 
 func newRoot(t *testing.T) CA {
@@ -26,7 +29,7 @@ func TestCrtRoundtrip(t *testing.T) {
 		t.Fatalf("Failed to decode PEM Crt: %s", err)
 	}
 
-	if err := crt.Verify(rootTrust, "endentity.test"); err != nil {
+	if err := crt.Verify(rootTrust, "endentity.test", time.Time{}); err != nil {
 		t.Fatal("Failed to verify round-tripped certificate")
 	}
 }
@@ -47,3 +50,71 @@ func TestCredEncodeCertificateAndTrustChain(t *testing.T) {
 		t.Errorf("Encoded Certificate And TrustChain does not match expected output")
 	}
 }
+
+func TestCrtExpiry(t *testing.T) {
+	root := newRoot(t)
+	rootTrust := root.Cred.Crt.CertPool()
+
+	cred, err := root.GenerateEndEntityCred("expired.test")
+	if err != nil {
+		t.Fatalf("failed to create end entity cred: %s", err)
+	}
+
+	crt, err := DecodePEMCrt(cred.Crt.EncodePEM())
+	if err != nil {
+		t.Fatalf("Failed to decode PEM Crt: %s", err)
+	}
+
+	fakeExpiryError := x509.CertificateInvalidError{Reason: x509.Expired}
+
+	//need to remove seconds and nanoseconds for testing returned error
+	now := time.Now()
+
+	testCases := []struct {
+		currentTime time.Time
+		notBefore   time.Time
+		notAfter    time.Time
+		expected    string
+	}{
+		//cert not valid yet
+		{
+			currentTime: now,
+			notAfter:    now.AddDate(0, 0, 20),
+			notBefore:   now.AddDate(0, 0, 10),
+			expected:    fmt.Sprintf("%s - Current Time : %s - Invalid before %s - Invalid After %s", fakeExpiryError.Error(), now, now.AddDate(0, 0, 10), now.AddDate(0, 0, 20)),
+		},
+		//cert has expired
+		{
+			currentTime: now,
+			notAfter:    now.AddDate(0, 0, -10),
+			notBefore:   now.AddDate(0, 0, -20),
+			expected:    fmt.Sprintf("%s - Current Time : %s - Invalid before %s - Invalid After %s", fakeExpiryError.Error(), now, now.AddDate(0, 0, -20), now.AddDate(0, 0, -10)),
+		},
+		// cert is valid
+		{
+			currentTime: time.Time{},
+			notAfter:    crt.Certificate.NotAfter,
+			notBefore:   crt.Certificate.NotBefore,
+			expected:    "",
+		},
+	}
+
+	for i, tc := range testCases {
+		tc := tc //pin
+		t.Run(fmt.Sprintf("%d", i), func(t *testing.T) {
+			//explicitly kill the certificate
+			crt.Certificate.NotBefore = tc.notBefore
+			crt.Certificate.NotAfter = tc.notAfter
+
+			if err := crt.Verify(rootTrust, "expired.test", tc.currentTime); err != nil {
+				if err.Error() != tc.expected {
+					t.Logf("Returned error : %s\n", err)
+					t.Logf("Expected error : %s\n", tc.expected)
+					t.Fatal("test case failed")
+				}
+			} else {
+				t.Log("no error on verification")
+			}
+		})
+	}
+}