Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help?: No What keywords did you search in Kubernetes issues before filing this one? : copy-to-staging Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature Request / Refactoring. <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Environment: Cloud provider or hardware configuration: Google Cloud (gce) What happened: Whenever I execute cluster/kube- it copies my tar files to google cloud, even if the files haven't changed. These files are a few hundred megs and can take a long time to upload. What you expected to happen: I'd like copy-to-staging function to check whether a matching tar already exist, and avoid uploading it again in this situation. How to reproduce it: execute cluster/kube- Here is the current behavior: cluster/kube- ... Starting cluster in us-central1-b using provider gce ... calling verify-prereqs ... calling verify-kube-binaries ... calling kube-up Project: <PROJECT-NAMEZone: us-central1-b +++ Staging server tars to Google Storage: gs://kubernetes-staging-<PROJECT-NAME/kubernetes-devel +++ kubernetes-server-linux- uploaded (sha1 = <HASH) Anything else we need to know: I have written the code to deal with this issue, and I'll submit a related PR shortly.
- PR coming...
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig cluster-lifecycle
/assign

Discussion:
fi } <ins> # Use gsutil to get the md5 hash for a particular tar function gsutil_get_tar_md5() { # location_tar could be local or in the cloud # local tar_location example ./_output/release-tars/kubernetes-server-linux-amd64.tar.gz # cloud tar_location example gs://kubernetes-staging-PROJECT/kubernetes-devel/kubernetes-server-linux-amd64.tar.gz local -r tar_location=$1 #parse the output and return the md5 hash #the sed command at the end removes whitespace local -r tar_md5=$(gsutil hash -h -m ${tar_location} 2>/dev/null | grep "Hash (md5):" | awk -F ':' '{print $2}' | sed 's/^[[:space:]]*//g') echo "${tar_md5}" } </ins> # Copy a release tar and its accompanying hash. function copy-to-staging() { local -r staging_path=$1 local -r gs_url=$2 local -r tar=$3 local -r hash=$4 <ins> local -r basename_tar=$(basename ${tar}) #check whether this tar alread exists and has the same hash #if it matches, then don't bother uploading it again #remote_tar_md5 checks the remote location for the existing tarball and its md5 #staging_path example gs://kubernetes-staging-PROJECT/kubernetes-devel #basename_tar example kubernetes-server-linux-amd64.tar.gz local -r remote_tar_md5=$(gsutil_get_tar_md5 "${staging_path}/${basename_tar}") if [[ -n ${remote_tar_md5} ]]; then #local_tar_md5 checks the remote location for the existing tarball and its md5 hash #tar example ./_output/release-tars/kubernetes-server-linux-amd64.tar.gz local -r local_tar_md5=$(gsutil_get_tar_md5 "${tar}") if [[ "${remote_tar_md5}" == "${local_tar_md5}" ]]; then echo "+++ ${basename_tar} uploaded earlier, cloud and local file md5 match (md5 = ${local_tar_md5})" return 0 fi fi </ins> echo "${hash}" > "${tar}.sha1" gsutil -m -q -h "Cache-Control:private, max-age=0" cp "${tar}" "${tar}.sha1" "${staging_path}" gsutil -m acl ch -g all:R "${gs_url}" "${gs_url}.sha1" >/dev/null 2>&1 <del> echo "+++ $(basename ${tar}) uploaded (sha1 = ${hash})" </del> <ins> echo "+++ ${basename_tar} uploaded (sha1 = ${hash})" </ins> } # Given the cluster zone, return the list of regional GCS release # bucket suffixes for the release in preference order. GCS doesn't # give us an API for this, so we hardcode it.