Repository: knative/pkg
Pull request #1333 (2020-05-14 20:11:43)

Add a new "chaosduck" e2e test tool for leaderelection.
This adds a new main package under knative.dev/pkg/leaderelection/chaosduck, which
loads the leader election configuration, and periodically enumerates the leases
in the system namespace and kills a leader pod for each sharded reconciler
under a component.

Based on #1304

knative-prow-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/knative/pkg/pull/1333#" title="Author self-approved">mattmoor</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=knative%2Fpkg).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/knative/pkg/blob/master/OWNERS)~~ [mattmoor]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-pkg-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[codegen/cmd/injection-gen/generators/reconciler_controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file0) | Do not exist | 0.0% | 
[codegen/cmd/injection-gen/generators/reconciler_reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file1) | Do not exist | 0.0% | 
[codegen/cmd/injection-gen/generators/reconciler_reconciler_stub.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file2) | Do not exist | 0.0% | 
[controller/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file3) | 95.2% | 91.3% | -3.9
[leaderelection/config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file4) | 68.8% | 82.5% | 13.7
[leaderelection/context.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file5) | Do not exist | 81.8% | 
[reconciler/leader.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file6) | Do not exist | 100.0% | 
[webhook/certificates/certificates.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file7) | 86.2% | 83.9% | -2.3
[webhook/certificates/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file8) | 100.0% | 87.5% | -12.5
[webhook/configmaps/configmaps.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file9) | 93.9% | 91.3% | -2.6
[webhook/configmaps/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file10) | 100.0% | 91.7% | -8.3
[webhook/psbinding/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file11) | 100.0% | 92.3% | -7.7
[webhook/psbinding/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file13) | 79.6% | 78.7% | -0.9
[webhook/resourcesemantics/conversion/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file14) | 100.0% | 80.0% | -20.0
[webhook/resourcesemantics/conversion/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file15) | 96.4% | 90.3% | -6.1
[webhook/resourcesemantics/defaulting/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file16) | 100.0% | 90.9% | -9.1
[webhook/resourcesemantics/defaulting/defaulting.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file17) | 88.5% | 87.3% | -1.2
[webhook/resourcesemantics/validation/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file18) | 100.0% | 93.8% | -6.2
[webhook/resourcesemantics/validation/reconcile_config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696/artifacts/line-cov.html#file19) | 95.5% | 91.5% | -4.0

knative-prow-robot: @mattmoor: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-pkg-go-coverage | 76c7fb8dff8dd40b6f16eb35509fac19cd6d7e0b | [link](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1261026577324445696) | `/test pull-knative-pkg-go-coverage`

[Full PR test history](https://gubernator.knative.dev/pr/knative_pkg/1333). [Your PR dashboard](https://gubernator.knative.dev/pr/mattmoor).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
knative-prow-robot: @mattmoor: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
knative-prow-robot: @mattmoor: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-pkg-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[controller/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file4) | 92.9% | 91.3% | -1.6
[leaderelection/config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file5) | 56.5% | 72.0% | 15.5
[leaderelection/context.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file6) | Do not exist | 81.8% | 
[reconciler/leader.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file7) | Do not exist | 100.0% | 
[webhook/certificates/certificates.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file8) | 86.2% | 83.9% | -2.3
[webhook/certificates/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file9) | 100.0% | 87.5% | -12.5
[webhook/configmaps/configmaps.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file10) | 93.9% | 91.3% | -2.6
[webhook/configmaps/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file11) | 100.0% | 91.7% | -8.3
[webhook/psbinding/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file12) | 100.0% | 92.3% | -7.7
[webhook/psbinding/psbinding.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file13) | 88.1% | 87.4% | -0.7
[webhook/psbinding/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file14) | 79.6% | 78.7% | -0.9
[webhook/resourcesemantics/conversion/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file15) | 100.0% | 80.0% | -20.0
[webhook/resourcesemantics/conversion/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file16) | 96.4% | 90.3% | -6.1
[webhook/resourcesemantics/defaulting/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file17) | 100.0% | 90.9% | -9.1
[webhook/resourcesemantics/defaulting/defaulting.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file18) | 88.5% | 87.3% | -1.2
[webhook/resourcesemantics/validation/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file19) | 100.0% | 93.8% | -6.2
[webhook/resourcesemantics/validation/reconcile_config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266498503706677249/artifacts/line-cov.html#file20) | 95.5% | 91.5% | -4.0

knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-pkg-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[leaderelection/config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file4) | 56.5% | 72.0% | 15.5
[leaderelection/context.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file5) | Do not exist | 81.8% | 
[reconciler/leader.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file6) | Do not exist | 100.0% | 
[webhook/certificates/certificates.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file7) | 86.2% | 83.9% | -2.3
[webhook/certificates/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file8) | 100.0% | 87.5% | -12.5
[webhook/configmaps/configmaps.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file9) | 93.9% | 91.3% | -2.6
[webhook/configmaps/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file10) | 100.0% | 91.7% | -8.3
[webhook/psbinding/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file11) | 100.0% | 92.3% | -7.7
[webhook/psbinding/psbinding.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file12) | 88.1% | 87.4% | -0.7
[webhook/psbinding/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file13) | 79.6% | 78.7% | -0.9
[webhook/resourcesemantics/conversion/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file14) | 100.0% | 80.0% | -20.0
[webhook/resourcesemantics/conversion/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file15) | 96.4% | 90.3% | -6.1
[webhook/resourcesemantics/defaulting/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file16) | 100.0% | 90.9% | -9.1
[webhook/resourcesemantics/defaulting/defaulting.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file17) | 88.5% | 87.3% | -1.2
[webhook/resourcesemantics/validation/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file18) | 100.0% | 93.8% | -6.2
[webhook/resourcesemantics/validation/reconcile_config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266504418266714112/artifacts/line-cov.html#file19) | 95.5% | 91.5% | -4.0

knative-prow-robot: @mattmoor: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-pkg-unit-tests | 3df015635b11a144bf4110ce2ab8f228390941af | [link](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-unit-tests/1266504418270908417) | `/test pull-knative-pkg-unit-tests`

[Full PR test history](https://gubernator.knative.dev/pr/knative_pkg/1333). [Your PR dashboard](https://gubernator.knative.dev/pr/mattmoor).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-pkg-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[controller/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file4) | 92.9% | 91.3% | -1.6
[leaderelection/config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file5) | 56.5% | 72.0% | 15.5
[leaderelection/context.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file6) | Do not exist | 81.8% | 
[reconciler/leader.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file7) | Do not exist | 100.0% | 
[webhook/certificates/certificates.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file8) | 86.2% | 83.9% | -2.3
[webhook/certificates/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file9) | 100.0% | 87.5% | -12.5
[webhook/configmaps/configmaps.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file10) | 93.9% | 91.3% | -2.6
[webhook/configmaps/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file11) | 100.0% | 91.7% | -8.3
[webhook/psbinding/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file12) | 100.0% | 92.3% | -7.7
[webhook/psbinding/psbinding.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file13) | 88.1% | 87.4% | -0.7
[webhook/psbinding/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file14) | 79.6% | 78.7% | -0.9
[webhook/resourcesemantics/conversion/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file15) | 100.0% | 80.0% | -20.0
[webhook/resourcesemantics/conversion/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file16) | 96.4% | 90.3% | -6.1
[webhook/resourcesemantics/defaulting/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file17) | 100.0% | 90.9% | -9.1
[webhook/resourcesemantics/defaulting/defaulting.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file18) | 88.5% | 87.3% | -1.2
[webhook/resourcesemantics/validation/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file19) | 100.0% | 93.8% | -6.2
[webhook/resourcesemantics/validation/reconcile_config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1266519642898173959/artifacts/line-cov.html#file20) | 95.5% | 91.5% | -4.0

knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-pkg-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[controller/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file4) | 92.9% | 91.3% | -1.5
[leaderelection/config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file5) | 56.5% | 72.0% | 15.5
[leaderelection/context.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file6) | Do not exist | 81.8% | 
[reconciler/leader.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file7) | Do not exist | 100.0% | 
[webhook/certificates/certificates.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file8) | 86.2% | 83.9% | -2.3
[webhook/certificates/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file9) | 100.0% | 87.5% | -12.5
[webhook/configmaps/configmaps.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file10) | 94.3% | 91.8% | -2.5
[webhook/configmaps/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file11) | 100.0% | 91.7% | -8.3
[webhook/psbinding/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file12) | 100.0% | 92.3% | -7.7
[webhook/psbinding/psbinding.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file13) | 88.1% | 87.4% | -0.7
[webhook/psbinding/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file14) | 79.6% | 78.7% | -0.9
[webhook/resourcesemantics/conversion/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file15) | 100.0% | 80.0% | -20.0
[webhook/resourcesemantics/conversion/reconciler.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file16) | 96.4% | 90.3% | -6.1
[webhook/resourcesemantics/defaulting/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file17) | 100.0% | 90.9% | -9.1
[webhook/resourcesemantics/defaulting/defaulting.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file18) | 88.5% | 87.3% | -1.2
[webhook/resourcesemantics/validation/controller.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file19) | 100.0% | 93.8% | -6.2
[webhook/resourcesemantics/validation/reconcile_config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1269371380684558338/artifacts/line-cov.html#file20) | 95.5% | 91.5% | -4.0

knative-prow-robot: @mattmoor: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-pkg-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[leaderelection/config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1281778757933207554/artifacts/line-cov.html#file1) | 80.0% | 78.1% | -1.9

mattmoor: This is now based on: https://github.com/knative/pkg/pull/1476
knative-metrics-robot: The following is the coverage report on the affected files.
Say `/test pull-knative-pkg-go-coverage` to re-run this coverage report

File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
[leaderelection/config.go](https://storage.cloud.google.com/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-go-coverage/1281972533611991042/artifacts/line-cov.html#file1) | 80.0% | 78.1% | -1.9

mattmoor: /hold

Switching holds 🙃 
knative-prow-robot: @mattmoor: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-knative-pkg-unit-tests | 328dab19b36e61cc894a9b092daabfef47470ad0 | [link](https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_pkg/1333/pull-knative-pkg-unit-tests/1282784257357385729) | `/test pull-knative-pkg-unit-tests`

[Full PR test history](https://gubernator.knative.dev/pr/knative_pkg/1333). [Your PR dashboard](https://gubernator.knative.dev/pr/mattmoor).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mattmoor: Alright, this is pretty benign by itself, so I'm gonna let it go in.

/unhold

Patch:
diff --git a/leaderelection/chaosduck/main.go b/leaderelection/chaosduck/main.go
new file mode 100644
index 000000000..6a41937fa
--- /dev/null
+++ b/leaderelection/chaosduck/main.go
@@ -0,0 +1,125 @@
+/*
+Copyright 2020 The Knative Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// The chaosduck binary is an e2e testing tool for leader election, which loads
+// the leader election configuration within the system namespace and
+// periodically kills one of the leader pods for each HA component.
+package main
+
+import (
+	"context"
+	"errors"
+	"log"
+	"strings"
+	"time"
+
+	"golang.org/x/sync/errgroup"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/util/sets"
+	"k8s.io/client-go/kubernetes"
+	kubeclient "knative.dev/pkg/client/injection/kube/client"
+	"knative.dev/pkg/controller"
+	"knative.dev/pkg/injection"
+	"knative.dev/pkg/injection/sharedmain"
+	"knative.dev/pkg/signals"
+	"knative.dev/pkg/system"
+	"knative.dev/pkg/test/ha"
+)
+
+// components is a mapping from component name to the collection of leader pod names.
+type components map[string]sets.String
+
+// buildComponents crawls the list of leases and builds a mapping from component names
+// to the set pod names that hold one or more leases.
+func buildComponents(kc kubernetes.Interface) (components, error) {
+	leases, err := kc.CoordinationV1().Leases(system.Namespace()).List(metav1.ListOptions{})
+	if err != nil {
+		return nil, err
+	}
+
+	cs := components{}
+	for _, lease := range leases.Items {
+		if lease.Spec.HolderIdentity == nil {
+			log.Printf("Found lease %q held by nobody!", lease.Name)
+			continue
+		}
+		pod := strings.SplitN(*lease.Spec.HolderIdentity, "_", 2)[0]
+		deploymentName := ha.ExtractDeployment(pod)
+		if deploymentName == "" {
+			continue
+		}
+
+		set, ok := cs[deploymentName]
+		if !ok {
+			set = sets.NewString()
+			cs[deploymentName] = set
+		}
+		set.Insert(pod)
+	}
+	return cs, nil
+}
+
+// quack will kill one of the components leader pods.
+func quack(ctx context.Context, kc kubernetes.Interface, component string, leaders sets.String) error {
+	tribute, ok := leaders.PopAny()
+	if !ok {
+		return errors.New("this should not be possible, since components are only created when they have components")
+	}
+	log.Printf("Quacking at %q leader %q", component, tribute)
+
+	return kc.CoreV1().Pods(system.Namespace()).Delete(tribute, &metav1.DeleteOptions{})
+}
+
+// frequency is the frequency with which we kill off leaders.
+const frequency = 30 * time.Second
+
+func main() {
+	ctx := signals.NewContext()
+
+	// We don't expect informers to be set up, but we do expect the client to get attached to ctx.
+	ctx, informers := injection.Default.SetupInformers(ctx, sharedmain.ParseAndGetConfigOrDie())
+	if err := controller.StartInformers(ctx.Done(), informers...); err != nil {
+		log.Fatalf("Failed to start informers %v", err)
+	}
+	kc := kubeclient.Get(ctx)
+
+	// Until we are shutdown, build up an index of components and kill
+	// of a leader at the specified frequency.
+	for {
+		select {
+		case <-time.After(frequency):
+			components, err := buildComponents(kc)
+			if err != nil {
+				log.Printf("Error building components: %v", err)
+			}
+			log.Printf("Got components: %#v", components)
+
+			eg, ctx := errgroup.WithContext(ctx)
+			for name, leaders := range components {
+				name, leaders := name, leaders
+				eg.Go(func() error {
+					return quack(ctx, kc, name, leaders)
+				})
+			}
+			if err := eg.Wait(); err != nil {
+				log.Printf("Ended iteration with err: %v", err)
+			}
+
+		case <-ctx.Done():
+			return
+		}
+	}
+}


Test patch:
diff --git a/test/ha/ha.go b/test/ha/ha.go
index 2859cc872..a88e8f173 100644
--- a/test/ha/ha.go
+++ b/test/ha/ha.go
@@ -41,7 +41,7 @@ func countingRFind(wr rune, wc int) func(rune) bool {
 	}
 }
 
-func extractDeployment(pod string) string {
+func ExtractDeployment(pod string) string {
 	if x := strings.LastIndexFunc(pod, countingRFind('-', 2)); x != -1 {
 		return pod[:x]
 	}
@@ -63,7 +63,7 @@ func GetLeaders(t *testing.T, client *test.KubeClient, deploymentName, namespace
 		pod := strings.SplitN(*lease.Spec.HolderIdentity, "_", 2)[0]
 
 		// Deconstruct the pod name and look for the deployment.  This won't work for very long deployment names.
-		if extractDeployment(pod) != deploymentName {
+		if ExtractDeployment(pod) != deploymentName {
 			continue
 		}
 		ret = append(ret, pod)
diff --git a/test/ha/ha_test.go b/test/ha/ha_test.go
index 41b6ca9da..4b460ccde 100644
--- a/test/ha/ha_test.go
+++ b/test/ha/ha_test.go
@@ -20,10 +20,10 @@ import "testing"
 
 func TestExtractDeployment(t *testing.T) {
 	const want = "gke-cluster-michigan-pool-2"
-	if got := extractDeployment("gke-cluster-michigan-pool-2-03f384a0-2zu1"); got != want {
+	if got := ExtractDeployment("gke-cluster-michigan-pool-2-03f384a0-2zu1"); got != want {
 		t.Errorf("Deployment = %q, want: %q", got, want)
 	}
-	if got := extractDeployment("a-b"); got != "" {
+	if got := ExtractDeployment("a-b"); got != "" {
 		t.Errorf("Deployment = %q, want empty string", got)
 	}
 }