Change:
In our scalability tests it sometimes happens that all tests are passing, but the run is marked red as "failure". This is because uploading logs to GCS sometimes failed with error: I've seen 2 examples during last 24 hours: For now it's just misleading. However, since scalability suite is now pretty stable, I would like to add it to list of projects that our merge bot requires to be green to merge anything. Unfortunately this looks like a blocker for it. Any thoughts how to fix that? cc
cc
Another example of it:
Yes, I think this is a good idea. It should be relatively trivial to mask the uploading error from Jenkins (in the script). We do ideally need some sort of warning to alert us if/when the uploads are failing repeatedly (which could be done in a separate PR). Ideally it would email the oncall, or automatically (re-)open and assign a github ticket. do you have time to look at this next week? Anyone else? If not, I will. Tentatively assigning to myself, although hoping that someone else will pick it up.
We're using a Jenkins plugin, not the e2e scripts, to upload build results to GCS. There's no way that I can find to make it not fail the build if an upload fails. Side note: the actual error message is a 401 error from GCS which seems to suggest an actual issue in the underlying infrastructure somewhere, which probably shouldn't be overlooked, but I have no idea where to start to try to figure which team to notify.
That said, it should be pretty straightforward to write a script to upload the artifacts and build log, rather than relying on the plugin.
Yes - that would be great if you could do it...
Just bumping it up - it happened again few times over the weekend and also today:
I looked at the existing plugin that we're using, and it does indeed retry (5 times!) Also searched the logs and found a lot of GCS errors: $ grep GoogleJsonResponseException: uniq -c 50 401 Unauthorized 22 500 Internal Server Error 12 503 Service Unavailable So I don't really think that there is much more that we can do here other than help GCS to track down their errors, and increase the retry count (although 5 with exponential backoff already seems rather excessive).
Thanks for looking into it and ! Is it easy to increase the retry count? Can we do it (I agree that 5 doesn't seem low, but we still observe errors). Did you talk to GCS people? I wouldn't expect the number of errors to be such big...
No, the retry count is unfortunately not configurable. I have not yet spoke to the GCS people, but plan to.
^
did we ever follow up with the GCS folks? I'm working on a PR to upload test artifacts/logs all within itself.
sorry, no.
- why did you close this? IIUC the PR is still not merged.
Yes, sorry, you're right. I'll assign this one to to close once the PR is merged.
I enabled the new script for -scalability, but it will take a few hours to verify that it works as intended. I haven't disabled the plugin yet, but assuming everything works as intended, I'll do that during the day PDT Friday.
- thanks!
- this doesn't work at all - in fact the cluster doesn't even start after this change
- I reverted your change Can you please experiment on something different than -scalability - if this doesn't work it blocks merge bot for ~2.5 hours which is how long a single run lasts. Can you please experiment on something that is not blocking merge-bot first? FYI:
Ugh, sorry about that. It'd been working fine on a number of builds, but all of those cleaned their workspaces before running tests. should fix the issue, and I'll test it on one of the flaky builds first.

Discussion:
<ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Source this script in the Jenkins "Execute shell" build action to have all # test artifacts and the console log uploaded at the end of the test run. # For example, you might use the following line as the first command: # mkdir -p _tmp/ && curl -fsS --retry 3 -o _tmp/upload-to-gcs.sh  #   "https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/upload-to-gcs.sh"  #   && source _tmp/upload-to-gcs.sh # Note that this script requires the Jenkins shell binary to be set to bash, not # the system default (which may be dash on Debian-based systems). # Also, for magicfile support to work correctly, the "file" utility must be # installed. set -o errexit set -o nounset set -o pipefail set -o xtrace if [[ ${JOB_NAME} =~ -pull- ]]; then : ${JENKINS_GCS_LOGS_PATH:="gs://kubernetes-jenkins/pr-logs/${ghprbActualCommit:-unknown}"} else : ${JENKINS_GCS_LOGS_PATH:="gs://kubernetes-jenkins/logs"} fi : ${JENKINS_UPLOAD_TO_GCS:="y"} # Upload the build log and all test artifacts (under _artifacts) to GCS when # JENKINS_UPLOAD_TO_GCS is set to y. # We intentionally ignore gsutil errors since we don't want failed uploads to # fail the entire test run (#13548). function upload_logs_to_gcs() { if [[ ! ${JENKINS_UPLOAD_TO_GCS:-} =~ ^[yY]$ ]]; then return fi local -r artifacts_path="${WORKSPACE}/_artifacts" local -r gcs_job_path="${JENKINS_GCS_LOGS_PATH}/${JOB_NAME}" local -r gcs_build_path="${gcs_job_path}/${BUILD_NUMBER}" local -r gcs_acl="public-read" for upload_attempt in $(seq 3); do echo "Uploading to ${gcs_build_path} (attempt ${upload_attempt})" if [[ -d "${artifacts_path}" ]]; then gsutil -m -q -o "GSUtil:use_magicfile=True" cp -a "${gcs_acl}" -r -c  -z log,txt,xml "${artifacts_path}" "${gcs_build_path}/artifacts" || continue fi # Remove ANSI escape sequences from the console log before uploading. local -r filtered_console_log="${WORKSPACE}/console-log.txt" sed -r "s/x1B[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g"  "${JENKINS_HOME}/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log"  > "${filtered_console_log}" gsutil -q cp -a "${gcs_acl}" -z txt "${filtered_console_log}" "${gcs_build_path}/" || continue # Mark this build as the latest completed. { echo "BUILD_NUMBER=${BUILD_NUMBER}" echo "GIT_COMMIT=${GIT_COMMIT:-}" } | gsutil -q -h "Content-Type:text/plain" -h "Cache-Control:private, max-age=0, no-transform"  cp -a "${gcs_acl}" - "${gcs_job_path}/latest-build.txt" || continue break  # all uploads succeeded if we hit this point done } # Automatically upload logs to GCS on exit or timeout. trap upload_logs_to_gcs EXIT SIGTERM SIGINT </ins>