Change:
is a bash script linter which could help catch common problems in our (numerous) bash scripts. I know we'd like to reduce our use of bash scripts, but until then I think being stricter about script quality would be helpful. /cc
cc you were talking about this earlier. It has to only check modified lines.
False positives are a problem. For instance, not being able to follow a call through an environment variable causes an error. I would be much more comfortable with a non-blocking way to point out these issues, and leave to it human judgement.
Yeah, I've been interested in doing this for a while, but can be pretty persnickety and I haven't yet taken the time to configure it to be less noisy (or figure out if it can be). It's good, though.
Yeah, it would definitely need to be configured to ignore a lot of types of errors, but I still think it could be used to check for some types of errors. We could start with a pretty minimal set.
I would settle for something that just produced a quick-fix format artifact in the GCS bucket, and then the PR munger reading from it and commenting on the PR. We could re-use that for other linters too.
has been pretty accurate historically.
shellcheck, not spellcheck
reading too many issues in a on
/sig testing /priority awaiting-more-evidence Might be more noise than it's worth, but we do pylint our .py scripts now
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
I'd like to reopen this, and eventually implement (first we need to deal with the failures ...) I've easily found a lot of bugs in test-infra and kubernetes/kubernetes bash with shellcheck. See also /assign
/reopen /remove-lifecycle rotten /priority important-longterm
Reopening this issue. <detailsIn response to : /reopen /remove-lifecycle rotten /priority important-longterm Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We also have non-blocking and optionally-run presubmits, even just having an easy way for reviewers to request a shellcheck result for changed files seems like something we should do. I also think we should fix the existing failures. Glancing through some k/k bash with shellcheck in my editor, these generally seem to be things that are actually wrong.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten coming back to this hopefully
This is done now. Only took two years :^)

Discussion:
<ins> ./build/build-image/rsyncd.sh ./build/common.sh ./build/copy-output.sh ./build/lib/release.sh ./build/make-build-image.sh ./build/make-clean.sh ./build/package-tarballs.sh ./build/release-images.sh ./build/release-in-a-container.sh ./build/release.sh ./build/run.sh ./build/shell.sh ./build/util.sh ./cluster/addons/addon-manager/kube-addons.sh ./cluster/addons/fluentd-elasticsearch/es-image/run.sh ./cluster/addons/fluentd-elasticsearch/fluentd-es-image/run.sh ./cluster/centos/build.sh ./cluster/centos/config-build.sh ./cluster/centos/config-default.sh ./cluster/centos/config-test.sh ./cluster/centos/deployAddons.sh ./cluster/centos/make-ca-cert.sh ./cluster/centos/master/scripts/flannel.sh ./cluster/centos/node/bin/mk-docker-opts.sh ./cluster/centos/node/scripts/flannel.sh ./cluster/centos/util.sh ./cluster/clientbin.sh ./cluster/common.sh ./cluster/gce/config-common.sh ./cluster/gce/config-default.sh ./cluster/gce/config-test.sh ./cluster/gce/delete-stranded-load-balancers.sh ./cluster/gce/gci/configure-helper.sh ./cluster/gce/gci/configure.sh ./cluster/gce/gci/flexvolume_node_setup.sh ./cluster/gce/gci/health-monitor.sh ./cluster/gce/gci/master-helper.sh ./cluster/gce/gci/mounter/stage-upload.sh ./cluster/gce/gci/node-helper.sh ./cluster/gce/gci/shutdown.sh ./cluster/gce/list-resources.sh ./cluster/gce/upgrade-aliases.sh ./cluster/gce/upgrade.sh ./cluster/gce/util.sh ./cluster/get-kube-binaries.sh ./cluster/get-kube-local.sh ./cluster/get-kube.sh ./cluster/images/conformance/run_e2e.sh ./cluster/images/etcd-empty-dir-cleanup/etcd-empty-dir-cleanup.sh ./cluster/juju/prereqs/ubuntu-juju.sh ./cluster/juju/util.sh ./cluster/kube-down.sh ./cluster/kube-up.sh ./cluster/kube-util.sh ./cluster/kubeadm.sh ./cluster/kubectl.sh ./cluster/kubemark/gce/config-default.sh ./cluster/kubemark/iks/config-default.sh ./cluster/kubemark/util.sh ./cluster/local/util.sh ./cluster/log-dump/log-dump.sh ./cluster/pre-existing/util.sh ./cluster/restore-from-backup.sh ./cluster/test-e2e.sh ./cluster/test-network.sh ./cluster/test-smoke.sh ./cluster/update-storage-objects.sh ./cluster/validate-cluster.sh ./hack/benchmark-go.sh ./hack/build-cross.sh ./hack/build-go.sh ./hack/build-ui.sh ./hack/cherry_pick_pull.sh ./hack/dev-build-and-push.sh ./hack/dev-build-and-up.sh ./hack/dev-push-conformance.sh ./hack/dev-push-hyperkube.sh ./hack/e2e-internal/e2e-cluster-size.sh ./hack/e2e-internal/e2e-down.sh ./hack/e2e-internal/e2e-grow-cluster.sh ./hack/e2e-internal/e2e-shrink-cluster.sh ./hack/e2e-internal/e2e-status.sh ./hack/e2e-internal/e2e-up.sh ./hack/e2e-node-test.sh ./hack/generate-bindata.sh ./hack/generate-docs.sh ./hack/get-build.sh ./hack/ginkgo-e2e.sh ./hack/godep-restore.sh ./hack/godep-save.sh ./hack/grab-profiles.sh ./hack/install-etcd.sh ./hack/jenkins/benchmark-dockerized.sh ./hack/jenkins/build.sh ./hack/jenkins/test-dockerized.sh ./hack/jenkins/upload-to-gcs.sh ./hack/jenkins/verify-dockerized.sh ./hack/lib/etcd.sh ./hack/lib/golang.sh ./hack/lib/init.sh ./hack/lib/logging.sh ./hack/lib/protoc.sh ./hack/lib/swagger.sh ./hack/lib/test.sh ./hack/lib/util.sh ./hack/lib/version.sh ./hack/list-feature-tests.sh ./hack/local-up-cluster.sh ./hack/make-rules/build.sh ./hack/make-rules/clean.sh ./hack/make-rules/cross.sh ./hack/make-rules/helpers/cache_go_dirs.sh ./hack/make-rules/make-help.sh ./hack/make-rules/test-cmd.sh ./hack/make-rules/test-e2e-node.sh ./hack/make-rules/test-integration.sh ./hack/make-rules/test-kubeadm-cmd.sh ./hack/make-rules/test.sh ./hack/make-rules/update.sh ./hack/make-rules/verify.sh ./hack/make-rules/vet.sh ./hack/print-workspace-status.sh ./hack/run-in-gopath.sh ./hack/test-go.sh ./hack/test-integration.sh ./hack/test-update-storage-objects.sh ./hack/update-all.sh ./hack/update-api-reference-docs.sh ./hack/update-bazel.sh ./hack/update-codegen.sh ./hack/update-generated-device-plugin-dockerized.sh ./hack/update-generated-device-plugin.sh ./hack/update-generated-docs.sh ./hack/update-generated-kms-dockerized.sh ./hack/update-generated-kms.sh ./hack/update-generated-kubelet-plugin-registration-dockerized.sh ./hack/update-generated-kubelet-plugin-registration.sh ./hack/update-generated-pod-resources-dockerized.sh ./hack/update-generated-pod-resources.sh ./hack/update-generated-protobuf-dockerized.sh ./hack/update-generated-protobuf.sh ./hack/update-generated-runtime-dockerized.sh ./hack/update-generated-runtime.sh ./hack/update-generated-swagger-docs.sh ./hack/update-godep-licenses.sh ./hack/update-gofmt.sh ./hack/update-openapi-spec.sh ./hack/update-staging-godeps-dockerized.sh ./hack/update-staging-godeps.sh ./hack/update-swagger-spec.sh ./hack/update-translations.sh ./hack/update-workspace-mirror.sh ./hack/verify-all.sh ./hack/verify-api-groups.sh ./hack/verify-api-reference-docs.sh ./hack/verify-bazel.sh ./hack/verify-boilerplate.sh ./hack/verify-cli-conventions.sh ./hack/verify-codegen.sh ./hack/verify-description.sh ./hack/verify-generated-device-plugin.sh ./hack/verify-generated-docs.sh ./hack/verify-generated-files-remake.sh ./hack/verify-generated-files.sh ./hack/verify-generated-kms.sh ./hack/verify-generated-kubelet-plugin-registration.sh ./hack/verify-generated-pod-resources.sh ./hack/verify-generated-protobuf.sh ./hack/verify-generated-runtime.sh ./hack/verify-generated-swagger-docs.sh ./hack/verify-godep-licenses.sh ./hack/verify-godeps.sh ./hack/verify-gofmt.sh ./hack/verify-golint.sh ./hack/verify-govet.sh ./hack/verify-import-boss.sh ./hack/verify-imports.sh ./hack/verify-linkcheck.sh ./hack/verify-no-vendor-cycles.sh ./hack/verify-openapi-spec.sh ./hack/verify-pkg-names.sh ./hack/verify-readonly-packages.sh ./hack/verify-spelling.sh ./hack/verify-staging-godeps.sh ./hack/verify-staging-meta-files.sh ./hack/verify-swagger-spec.sh ./hack/verify-symbols.sh ./hack/verify-test-featuregates.sh ./hack/verify-test-images.sh ./hack/verify-test-owners.sh ./hack/verify-typecheck.sh ./pkg/kubectl/cmd/edit/testdata/record_testcase.sh ./pkg/util/verify-util-pkg.sh ./plugin/pkg/admission/imagepolicy/gencerts.sh ./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/update-codegen.sh ./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/verify-codegen.sh ./staging/src/k8s.io/apiextensions-apiserver/hack/build-image.sh ./staging/src/k8s.io/apiextensions-apiserver/hack/update-codegen.sh ./staging/src/k8s.io/apiextensions-apiserver/hack/verify-codegen.sh ./staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/testcerts/gencerts.sh ./staging/src/k8s.io/apiserver/pkg/util/webhook/gencerts.sh ./staging/src/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc/testdata/gen.sh ./staging/src/k8s.io/apiserver/plugin/pkg/authorizer/webhook/gencerts.sh ./staging/src/k8s.io/code-generator/generate-groups.sh ./staging/src/k8s.io/code-generator/generate-internal-groups.sh ./staging/src/k8s.io/code-generator/hack/update-codegen.sh ./staging/src/k8s.io/code-generator/hack/verify-codegen.sh ./staging/src/k8s.io/csi-api/hack/update-codegen.sh ./staging/src/k8s.io/csi-api/hack/verify-codegen.sh ./staging/src/k8s.io/kube-aggregator/hack/build-image.sh ./staging/src/k8s.io/kube-aggregator/hack/local-up-kube-aggregator.sh ./staging/src/k8s.io/kube-aggregator/hack/register-all-apis-from.sh ./staging/src/k8s.io/kube-aggregator/hack/update-codegen.sh ./staging/src/k8s.io/kube-aggregator/hack/verify-codegen.sh ./staging/src/k8s.io/metrics/hack/update-codegen.sh ./staging/src/k8s.io/metrics/hack/verify-codegen.sh ./staging/src/k8s.io/node-api/hack/update-codegen.sh ./staging/src/k8s.io/node-api/hack/verify-codegen.sh ./staging/src/k8s.io/sample-apiserver/hack/build-image.sh ./staging/src/k8s.io/sample-apiserver/hack/update-codegen.sh ./staging/src/k8s.io/sample-apiserver/hack/verify-codegen.sh ./staging/src/k8s.io/sample-controller/hack/update-codegen.sh ./staging/src/k8s.io/sample-controller/hack/verify-codegen.sh ./test/cmd/apply.sh ./test/cmd/apps.sh ./test/cmd/authorization.sh ./test/cmd/batch.sh ./test/cmd/certificate.sh ./test/cmd/core.sh ./test/cmd/crd.sh ./test/cmd/create.sh ./test/cmd/diff.sh ./test/cmd/discovery.sh ./test/cmd/generic-resources.sh ./test/cmd/get.sh ./test/cmd/initializers.sh ./test/cmd/legacy-script.sh ./test/cmd/node-management.sh ./test/cmd/old-print.sh ./test/cmd/proxy.sh ./test/cmd/rbac.sh ./test/cmd/request-timeout.sh ./test/cmd/run.sh ./test/cmd/save-config.sh ./test/cmd/storage.sh ./test/cmd/template-output.sh ./test/cmd/version.sh ./test/e2e_node/conformance/run_test.sh ./test/e2e_node/environment/setup_host.sh ./test/e2e_node/gubernator.sh ./test/e2e_node/jenkins/conformance/conformance-jenkins.sh ./test/e2e_node/jenkins/copy-e2e-image.sh ./test/e2e_node/jenkins/e2e-node-jenkins.sh ./test/e2e_node/jenkins/ubuntu-14.04-nvidia-install.sh ./test/images/image-util.sh ./test/images/pets/redis-installer/on-start.sh ./test/images/pets/zookeeper-installer/install.sh ./test/images/pets/zookeeper-installer/on-start.sh ./test/images/volume/gluster/run_gluster.sh ./test/images/volume/iscsi/create_block.sh ./test/images/volume/nfs/run_nfs.sh ./test/images/volume/rbd/bootstrap.sh ./test/images/volume/rbd/create_block.sh ./test/images/volume/rbd/mon.sh ./test/images/volume/rbd/osd.sh ./test/integration/ipamperf/test-performance.sh ./test/integration/scheduler_perf/test-performance.sh ./test/kubemark/common/util.sh ./test/kubemark/configure-kubectl.sh ./test/kubemark/gce/util.sh ./test/kubemark/iks/shutdown.sh ./test/kubemark/iks/startup.sh ./test/kubemark/iks/util.sh ./test/kubemark/master-log-dump.sh ./test/kubemark/pre-existing/util.sh ./test/kubemark/resources/start-kubemark-master.sh ./test/kubemark/run-e2e-tests.sh ./test/kubemark/start-kubemark.sh ./test/kubemark/stop-kubemark.sh ./third_party/forked/shell2junit/sh2ju.sh ./third_party/intemp/intemp.sh ./third_party/multiarch/qemu-user-static/register/qemu-binfmt-conf.sh ./third_party/multiarch/qemu-user-static/register/register.sh </ins>

<ins> #!/usr/bin/env bash # Copyright 2018 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" # upstream shellcheck latest stable image as of January 10th, 2019 SHELLCHECK_IMAGE="koalaman/shellcheck-alpine:v0.6.0@sha256:7d4d712a2686da99d37580b4e2f45eb658b74e4b01caf67c1099adc294b96b52" SHELLCHECK_CONTAINER="k8s-shellcheck" # disabled lints disabled=( # this lint dissalows non-constant source, which we use extensively 1090 # this lint prefers command -v to which, they are not the same 2230 ) # comma separate for passing to shellcheck join_by() { local IFS="$1"; shift; echo "$*"; } SHELLCHECK_DISABLED="$(join_by , "${disabled[@]}")" readonly SHELLCHECK_DISABLED cd "${KUBE_ROOT}" # find all shell scripts excluding ./_*, ./.git/*, ./vendor*, # and anything git-ignored all_shell_scripts=() while IFS=$'n' read -r script; do git check-ignore -q "$script" || all_shell_scripts+=("$script"); done < <(find . -name "*.sh"  -not (  -path ./_*      -o  -path ./.git*   -o  -path ./vendor*     )) # make sure known failures are sorted failure_file="${KUBE_ROOT}/hack/.shellcheck_failures" if ! diff -u "${failure_file}" <(LC_ALL=C sort "${failure_file}"); then { echo echo "hack/.shellcheck_failures is not in alphabetical order. Please sort it:" echo echo "  LC_ALL=C sort -o hack/.shellcheck_failures hack/.shellcheck_failures" echo } >&2 false fi # load known failure files failing_files=() while IFS=$'n' read -r script; do failing_files+=("$script"); done < <(cat "${failure_file}") # TODO(bentheelder): we should probably move this and the copy in verify-golint.sh # to one of the bash libs array_contains () { local seeking=$1; shift # shift will iterate through the array local in=1 # in holds the exit status for the function for element; do if [[ "$element" == "$seeking" ]]; then in=0 # set in to 0 since we found it break fi done return $in } # creates the shellcheck container for later user create_container () { # TODO(bentheelder): this is a performance hack, we create the container with # a sleep MAX_INT32 so that it is effectively paused. # We then repeatedly exec to it to run each shellcheck, and later rm it when # we're done. # This is incredibly much faster than creating a container for each shellcheck # call ... docker run --name "${SHELLCHECK_CONTAINER}" -d --rm -v "${KUBE_ROOT}:${KUBE_ROOT}" -w "${KUBE_ROOT}" --entrypoint="sleep" "${SHELLCHECK_IMAGE}" 2147483647 } # removes the shellcheck container remove_container () { docker rm -f "${SHELLCHECK_CONTAINER}" &> /dev/null || true } # remove any previous container, ensure we will attempt to cleanup on exit, # and create the container remove_container trap remove_container EXIT if ! output="$(create_container 2>&1)"; then { echo "Failed to create shellcheck container with output: " echo "" echo "${output}" } >&2 exit 1 fi # lint each script, tracking failures errors=() not_failing=() for f in "${all_shell_scripts[@]}"; do set +o errexit failedLint=$(docker exec -t "${SHELLCHECK_CONTAINER}" shellcheck --exclude="${SHELLCHECK_DISABLED}" "${f}") set -o errexit array_contains "${f}" "${failing_files[@]}" && in_failing=$? || in_failing=$? if [[ -n "${failedLint}" ]] && [[ "${in_failing}" -ne "0" ]]; then errors+=( "${failedLint}" ) fi if [[ -z "${failedLint}" ]] && [[ "${in_failing}" -eq "0" ]]; then not_failing+=( "${f}" ) fi done # Check to be sure all the packages that should pass lint are. if [ ${#errors[@]} -eq 0 ]; then echo 'Congratulations!  All shell files have been linted.' else { echo "Errors from shellcheck:" for err in "${errors[@]}"; do echo "$err" done echo echo 'Please review the above warnings. You can test via "./hack/verify-shellcheck"' echo 'If the above warnings do not make sense, you can exempt this package from shellcheck' echo 'checking by adding it to hack/.shellcheck_failures (if your reviewer is okay with it).' echo } >&2 false fi if [[ ${#not_failing[@]} -gt 0 ]]; then { echo "Some packages in hack/.shellcheck_failures are passing shellcheck. Please remove them." echo for f in "${not_failing[@]}"; do echo "  $f" done echo } >&2 false fi # Check that all failing_packages actually still exist gone=() for f in "${failing_files[@]}"; do array_contains "$f" "${all_shell_scripts[@]}" || gone+=( "$f" ) done if [[ ${#gone[@]} -gt 0 ]]; then { echo "Some files in hack/.shellcheck_failures do not exist anymore. Please remove them." echo for f in "${gone[@]}"; do echo "  $f" done echo } >&2 false fi </ins>