Code change:
<ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Source this script in the Jenkins "Execute shell" build action to have all # test artifacts and the console log uploaded at the end of the test run. # For example, you might use the following line as the first command: # mkdir -p _tmp/ && curl -fsS --retry 3 -o _tmp/upload-to-gcs.sh  #   "https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/upload-to-gcs.sh"  #   && source _tmp/upload-to-gcs.sh # Note that this script requires the Jenkins shell binary to be set to bash, not # the system default (which may be dash on Debian-based systems). # Also, for magicfile support to work correctly, the "file" utility must be # installed. set -o errexit set -o nounset set -o pipefail set -o xtrace if [[ ${JOB_NAME} =~ -pull- ]]; then : ${JENKINS_GCS_LOGS_PATH:="gs://kubernetes-jenkins/pr-logs/${ghprbActualCommit:-unknown}"} else : ${JENKINS_GCS_LOGS_PATH:="gs://kubernetes-jenkins/logs"} fi : ${JENKINS_UPLOAD_TO_GCS:="y"} # Upload the build log and all test artifacts (under _artifacts) to GCS when # JENKINS_UPLOAD_TO_GCS is set to y. # We intentionally ignore gsutil errors since we don't want failed uploads to # fail the entire test run (#13548). function upload_logs_to_gcs() { if [[ ! ${JENKINS_UPLOAD_TO_GCS:-} =~ ^[yY]$ ]]; then return fi local -r artifacts_path="${WORKSPACE}/_artifacts" local -r gcs_job_path="${JENKINS_GCS_LOGS_PATH}/${JOB_NAME}" local -r gcs_build_path="${gcs_job_path}/${BUILD_NUMBER}" local -r gcs_acl="public-read" for upload_attempt in $(seq 3); do echo "Uploading to ${gcs_build_path} (attempt ${upload_attempt})" if [[ -d "${artifacts_path}" ]]; then gsutil -m -q -o "GSUtil:use_magicfile=True" cp -a "${gcs_acl}" -r -c  -z log,txt,xml "${artifacts_path}" "${gcs_build_path}/artifacts" || continue fi # Remove ANSI escape sequences from the console log before uploading. local -r filtered_console_log="${WORKSPACE}/console-log.txt" sed -r "s/x1B[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g"  "${JENKINS_HOME}/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log"  > "${filtered_console_log}" gsutil -q cp -a "${gcs_acl}" -z txt "${filtered_console_log}" "${gcs_build_path}/" || continue # Mark this build as the latest completed. { echo "BUILD_NUMBER=${BUILD_NUMBER}" echo "GIT_COMMIT=${GIT_COMMIT:-}" } | gsutil -q -h "Content-Type:text/plain" -h "Cache-Control:private, max-age=0, no-transform"  cp -a "${gcs_acl}" - "${gcs_job_path}/latest-build.txt" || continue break  # all uploads succeeded if we hit this point done } # Automatically upload logs to GCS on exit or timeout. trap upload_logs_to_gcs EXIT SIGTERM SIGINT </ins>

Related discussion:
text: Thanks for looking into it and ! Is it easy to increase the retry count? Can we do it (I agree that 5 doesn't seem low, but we still observe errors). Did you talk to GCS people? I wouldn't expect the number of errors to be such big...
No, the retry count is unfortunately not configurable. I have not yet spoke to the GCS people, but plan to.
^
did we ever follow up with the GCS folks? I'm working on a PR to upload test artifacts/logs all within itself.
sorry, no.
- why did you close this? IIUC the PR is still not merged.
Yes, sorry, you're right. I'll assign this one to to close once the PR is merged.
I enabled the new script for -scalability, but it will take a few hours to verify that it works as intended. I haven't disabled the plugin yet, but assuming everything works as intended, I'll do that during the day PDT Friday.
- thanks!
- this doesn't work at all - in fact the cluster doesn't even start after this change
- I reverted your change Can you please experiment on something different than -scalability - if this doesn't work it blocks merge bot for ~2.5 hours which is how long a single run lasts. Can you please experiment on something that is not blocking merge-bot first? FYI:
Ugh, sorry about that. It'd been working fine on a number of builds, but all of those cleaned their workspaces before running tests. should fix the issue, and I'll test it on one of the flaky builds first.