Change:
cc
Assigning to per test-
Flaked again
The flake actually happened when pausing the deployment. Changing issue title.
When , sometimes one of the deployments isn't paused. cc
I can't access the internal Google logs to examine the cause, nor can I reproduce this issue locally. Is there any additional information that you can provide to help troubleshoot the flake?
I've gone ahead and attempted a fix for this in
We should fix this in eventually
the external links are here ():

Discussion:
function kubectl-with-retry() { ERROR_FILE="${KUBE_TEMP}/kubectl-error" <ins> preserve_err_file=${PRESERVE_ERR_FILE-false} </ins> for count in $(seq 0 3); do kubectl "$@" 2> ${ERROR_FILE} || true if grep -q "the object has been modified" "${ERROR_FILE}"; then

rm "${ERROR_FILE}" sleep $((2**count)) else <del> rm "${ERROR_FILE}" </del> <ins> if [ "$preserve_err_file" != true ] ; then rm "${ERROR_FILE}" fi </ins> break fi done

kube::test::get_object_assert 'rc busybox1' "{{$rc_replicas_field}}" '1' # Command output_message=$(! kubectl scale --current-replicas=1 --replicas=2 -f hack/testdata/recursive/rc --recursive 2>&1 "${kube_flags[@]}") <del> # Post-condition: busybox0 & busybox replication controllers are scaled to 2 # replicas, and since busybox2 is malformed, it should error </del> <ins> # Post-condition: busybox0 & busybox1 replication controllers are scaled to 2 replicas, and since busybox2 is malformed, it should error </ins> kube::test::get_object_assert 'rc busybox0' "{{$rc_replicas_field}}" '2' kube::test::get_object_assert 'rc busybox1' "{{$rc_replicas_field}}" '2' kube::test::if_has_string "${output_message}" "Object 'Kind' is missing"

kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" <del> ### Rollback a deployment </del> <ins> ### Rollout on multiple deployments recursively </ins> # Pre-condition: no deployments exist kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" '' # Command

! kubectl create -f hack/testdata/recursive/deployment --recursive "${kube_flags[@]}" kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" 'nginx0-deployment:nginx1-deployment:' kube::test::get_object_assert deployment "{{range.items}}{{$deployment_image_field}}:{{end}}" "${IMAGE_NGINX}:${IMAGE_NGINX}:" <del> ## Rollback to revision 1 - should be no-op </del> <ins> ## Rollback the deployments to revision 1 recursively </ins> output_message=$(! kubectl rollout undo -f hack/testdata/recursive/deployment --recursive --to-revision=1 2>&1 "${kube_flags[@]}") <ins> # Post-condition: nginx0 & nginx1 should be a no-op, and since nginx2 is malformed, it should error </ins> kube::test::get_object_assert deployment "{{range.items}}{{$deployment_image_field}}:{{end}}" "${IMAGE_NGINX}:${IMAGE_NGINX}:" kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" <del> ## Pause the deployment output_message=$(! kubectl rollout pause -f hack/testdata/recursive/deployment --recursive 2>&1 "${kube_flags[@]}") </del> <ins> ## Pause the deployments recursively PRESERVE_ERR_FILE=true kubectl-with-retry rollout pause -f hack/testdata/recursive/deployment --recursive "${kube_flags[@]}" output_message=$(cat ${ERROR_FILE}) # Post-condition: nginx0 & nginx1 should both have paused set to true, and since nginx2 is malformed, it should error </ins> kube::test::get_object_assert deployment "{{range.items}}{{.spec.paused}}:{{end}}" "true:true:" kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" <del> ## Resume the deployment output_message=$(! kubectl rollout resume -f hack/testdata/recursive/deployment --recursive 2>&1 "${kube_flags[@]}") </del> <ins> ## Resume the deployments recursively kubectl-with-retry rollout resume -f hack/testdata/recursive/deployment --recursive "${kube_flags[@]}" output_message=$(cat ${ERROR_FILE}) # Post-condition: nginx0 & nginx1 should both have paused set to nothing, and since nginx2 is malformed, it should error </ins> kube::test::get_object_assert deployment "{{range.items}}{{.spec.paused}}:{{end}}" "<no value>:<no value>:" kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" <ins> ## Retrieve the rollout history of the deployments recursively </ins> output_message=$(! kubectl rollout history -f hack/testdata/recursive/deployment --recursive 2>&1 "${kube_flags[@]}") <ins> # Post-condition: nginx0 & nginx1 should both have a history, and since nginx2 is malformed, it should error </ins> kube::test::if_has_string "${output_message}" "nginx0-deployment" kube::test::if_has_string "${output_message}" "nginx1-deployment" kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" # Clean up <ins> unset PRESERVE_ERR_FILE rm "${ERROR_FILE}" </ins> ! kubectl delete -f hack/testdata/recursive/deployment --recursive "${kube_flags[@]}" --grace-period=0 sleep 1 <del> ### Rollback a resource that cannot be rolled back (replication controller) </del> <ins> ### Rollout on multiple replication controllers recursively - these tests ensure that rollouts cannot be performed on resources that don't support it </ins> # Pre-condition: no replication controller exists kube::test::get_object_assert rc "{{range.items}}{{$id_field}}:{{end}}" '' # Command <del> # Create replication controllers (revision 1) recursively from directory of YAML files </del> <ins> # Create replication controllers recursively from directory of YAML files </ins> ! kubectl create -f hack/testdata/recursive/rc --recursive "${kube_flags[@]}" kube::test::get_object_assert rc "{{range.items}}{{$id_field}}:{{end}}" 'busybox0:busybox1:' # Command <del> ## Rollback to revision 1 - should be no-op </del> <ins> ## Attempt to rollback the replication controllers to revision 1 recursively </ins> output_message=$(! kubectl rollout undo -f hack/testdata/recursive/rc --recursive --to-revision=1 2>&1 "${kube_flags[@]}") <ins> # Post-condition: busybox0 & busybox1 should error as they are RC's, and since busybox2 is malformed, it should error </ins> kube::test::if_has_string "${output_message}" 'no rollbacker has been implemented for {"" "ReplicationController"}' kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" <del> ## Pause the deployment </del> <ins> ## Attempt to pause the replication controllers recursively </ins> output_message=$(! kubectl rollout pause -f hack/testdata/recursive/rc --recursive 2>&1 "${kube_flags[@]}") <ins> # Post-condition: busybox0 & busybox1 should error as they are RC's, and since busybox2 is malformed, it should error </ins> kube::test::if_has_string "${output_message}" 'error when pausing "hack/testdata/recursive/rc/busybox.yaml' kube::test::if_has_string "${output_message}" 'error when pausing "hack/testdata/recursive/rc/rc/busybox.yaml' kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" <del> ## Resume the deployment </del> <ins> ## Attempt to resume the replication controllers recursively </ins> output_message=$(! kubectl rollout resume -f hack/testdata/recursive/rc --recursive 2>&1 "${kube_flags[@]}") <ins> # Post-condition: busybox0 & busybox1 should error as they are RC's, and since busybox2 is malformed, it should error </ins> kube::test::if_has_string "${output_message}" 'error when resuming "hack/testdata/recursive/rc/busybox.yaml' kube::test::if_has_string "${output_message}" 'error when resuming "hack/testdata/recursive/rc/rc/busybox.yaml' kube::test::if_has_string "${output_message}" "Object 'Kind' is missing"