nathanmarz/storm — pull request #741 by xiaokang (2013-11-12 17:49:20+00:00)
Title: storm topology online update

This PR is for #540. The core code modification is about 300 lines and many others are generated by thrift.

main points:

1. interface changes (compatible with old versions)
1.1 zk add :topology-version and :update-duration-sec fields to StormBase:status map
1.2 zk add :topology-version to executor heartbeat
1.3 worker local state add versions dir to storm worker's current running topology-version
1.4 nimbus add updateTopology interface
1.5 add topology-version field to storm.thrift three struct: TopologySummary ExecutorSummary TopologyInfo

2. update process
2.1 storm client run "storm jar xxxx -c topology.update=true" to invoke topology update process
2.2 storm client upload new jar file to nimbus
2.3 storm client call nimbus updateTopology interface
2.4 nimbus check the new topology and replace stormdist/storm-id dir
2.5 nimbus update StormBase in zk, set :topology-version(for destination version) and :update-duration-sec(for all workers update process duration) fields to StormBase:status map
2.6 supervisors check zk StormBase and do the update work if topology's local version is not the same with zk version
2.6.1 sync-supervisor download the latest code from nimbus to local stormdist/topology-version dir
2.6.2 each supervisor schedule the topology's worker update at a rand(expect-max-update-time) time point
2.6.3 sync-process check local worker version, if it is not the same with sync-supervisor downloaded version and update time point reached, set worker state to a new :update state
2.6.4 sync-process kill workers in :update state as normally
2.6.5 sync-process restart killed worker as normally, expect that read topology and conf from stormdist/topology-version dir
2.6.6 new worker heartbeat to zk with new topology-version, it can be displayed on web ui to check update progress