Change:
# See the License for the specific language governing permissions and # limitations under the License. <del> # Provide reasonable default for running the end-to-end tests against a recent # stable release and then again after upgrading it to a version built from head. </del> <ins> # This runs e2e, does a checked upgrade of the MASTER using the locally built release, then runs e2e again. </ins> go run "$(dirname $0)/e2e.go" -v -build -up -version="v0.14.0" -test -check_version_skew=false if [ $? -eq 0 ]; then <del> echo "Tests on initial version succeeded. Proceeding with push and second set of tests." go run "$(dirname $0)/e2e.go" -v -push -version="" -test -check_version_skew=false </del> <ins> echo "Tests on initial version succeeded. Running the checked master upgrade." go run "$(dirname $0)/e2e.go" -v --test --test_args='--ginkgo.focus=Skipped.*Clustersupgrade.*gce-upgrade' -check_version_skew=false </ins> else echo "Tests on initial version failed. Skipping tests on second version." fi <ins> echo "Master upgrade complete. Running e2e on the upgraded cluster." go run "$(dirname $0)/e2e.go" -v -build -up -version="" -test -check_version_skew=false </ins> exit $?

Discussion:
Forked from . For testing master upgrades, we will need a test harness that: a cluster at some known version (e.g. latest release) some amount of work on the cluster Running e2es would be ok, but e2e tests tend to try and clean up after themselves and therefore a test that is better suited would be one that keeps running during the upgrade the master to a new version (e.g. latest release running in jenkins) For GCE, we should use for this part (and fix it as necessary) that the cluster is still in a sane state (from the previous tests) e2e tests against the upgraded cluster. /cc
I volunteer a workload based on Elasticsearch.
I forked to discuss the workload part. This one is for the harness (Jenkins?) itself. Feel free to assign to yourself.
are you available to work on this?
is pretty busy right now getting e2e's to run on every PR pre-merge, so that we don't keep wasting so much time tracking down regressions. I'd like him to stay focussed on that until it's done, hopefully mid next week.
Yeah sorry I didn't update the issue, I checked with him a couple of days ago and he said the same, so I was leaving unassigned for now. I am hoping can work on this but haven't discussed with him yet.
Is also interested in working on it. I just discussed it with him.
Provisionally assigning to
FYI
cc
What's the status here?
The test is set up in Jenkins. It currently picks up , deploys it to a cluster, runs E2E to verify stability, then runs the upgrade GCE script to upgrade to latest_ci, and another E2E to verify cluster afterwards. The test is not passing yet, but it works, so we can close this bug.
Where is it failing? If you "upgrade" instead to just again, does it pass?
For v17.1 to v17.1 it fails at: "line 71: set-master-htpasswd: command not found" This should be fixed in v18.0
do you think you could add what is described in this comment namely adding "invoke kube-push and run e2e again" ?
Yep, I'll take a crack at it today.
Done.

Added and to Jenkins GCE upgrade test.
Thanks a lot !
+1, thanks!
I think we can close this. If you agree, please close it.
Thanks for the work here! My understanding from comment at that of Robby's initial comment, the tests now do 1., 3., and 5., but not 2. or 4.? If the e2es clean up after themselves and we're testing upgrading an empty cluster, are we vulnerable to upgrades causing existing pods/services/etc to become broken in some way? Or restarted all pods? Or maybe turning this around: if we had these tests for borgmaster rollouts, would you be content? :)
Agh, sorry, I wasn't paying attention. I can see from 's description in that we run e2e before and after upgrade, but not during. IIUC you're suggesting we should be running some test during the upgrade. Unfortunately we never got around to doing (and is out until a week and a half from now), so we don't have a lot of options. I think there are two choices (i) Run e2e during upgrade in addition to before and after (ii) Write a new correctness test (maybe can do it?) that we would start right before the upgrade and would run during the entirety of the upgrade. Would (i) be sufficient? BTW I didn't understand step 4: "Verifies that the cluster is still in a sane state (from the previous tests)" -- what does "sane state" mean? Does it mean verify that the upgrade actually happened (i.e. new versions are running)?
Certainly happy to jump on this on Monday when I return assuming is OK with it. I think I asked this question on another thread, but don't we also want to check that the upgrade actually occurred? Otherwise the "NOP" operation would satisfy the test? This seems to suggest that we want something running that spans the lifetime of the two clusters and applies a probe to check for the dy/dx.
The issue with e2es is that they start from an empty cluster state and return it to an empty cluster state. I think making sure they pass before and after is necessary, but it'd be even better to test more things. What are the invariants we want to ensure through an upgrade process?

Something like: components are actually running the new versions, as described. pods/services/whatevers created before the upgrade still exist and are running afterwards? others? If so, that would mean step 2 creates a bunch of pods/services/whatevers, ensures they're in a sane state (accessible via network, running, ...?), and step 4 then verifies they're still accessible, running, etc. Step 4 probably also verifies components are the new version. And maybe also that e.g. pods didn't get restarted or evicted or deleted? Sorry for the super-high-level vagueness. I think I'm proposing that someone think about what's needed to ensure a good upgrade experience, maybe document it if the existing issues don't capture that, and then start implementing paths to achieve that good experience along with the tests that will verify it holds true across future versions? Re: v1 explicitly, I'm fine with a cut whereever, though I think we'd get a lot of mileage out of doing tests to verify behavior first and then iteratively improve capabilities over time?
SGTM. Thanks for taking over this issue. For reference, 's plan was to take his load test and just add a final step where you create a service and then use it to test connectivity to all of the pods. The load test is having problems, though (and it's probably better not to conflate load testing and correctness testing right now), so it would probably be better if you were to just do something simple from scratch (it could still have the same basic idea, I guess -- e.g. create a replication controller and a service, verify you can connect to the pods through the service).
offered to work on this and was OK with that (and it seems an especially good idea since just did the etcd-killing e2e test), so reassigning to Mike. Thanks, Mike!
Hmm, I don't think this should have been closed. I'm adding my comment from below.
The following could come as follow-up PRs / work, but are necessary to [x] [ ] Validate that the upgrade actually happened. We want to ensure that the new software is really running, otherwise--as pointed out in --a no-op would satisfy the "upgrade". [x] [ ] Have the webserver util create a replication controller rather than just a pod.

At a minimum, we want to validate the core k8s resources, and I think pod/rc/svc are the minimum. We'd have to make sure the services test still works, and we'd probably want to pull this code out of services and into util. [x] [ ] Do a bit more validation. I think we want to check, e.g., more than just the name of the service. Think of it this way: if you were running a cluster with a bunch of resources, what are all of the properties of your resources that you'd expect to hold after an upgrade? I think we need a bit deeper introspection. [x] [ Make the script () only run stable e2es when it runs e2es before and after the upgrade. We also don't want to run the 'restart' e2e test as that does another upgrade. This will hopefully be solved soon by having the necessary ginkgo test arg strings checked into github (cc so we can just that and use it in the script. [x] [ Set up a Jenkins job to basically run what does. We'll have to see how this is best done given the current Jenkins script. The answer might be to have multiple "step" jobs for Jenkins that trigger each other. [x] [ Tweak above code and config until Jenkins really runs green. [x] [ ] Add GKE support for the above The above, in my mind, will close out . The following would be extra credit (after the above are fixed): [ ] Validate more kubernetes resources (secretes, volumes, persistent volumes, ...?) can be created before an upgrade, (possibly) work during an upgrade, and (definitely) are correct after an upgrade. I'll also extend the above for nodes.
Do you have an ETA on the remaining work for this issue?
I updated the checklist; I can get a PR for the last item above out today.