gc3-uzh-ch/elasticluster — pull request #111 by jwm (2014-10-01 20:42:28+00:00)
Title: avoid pickle assertion failure on something in boto

I started playing with elasticluster this morning, and experienced an assertion failure when it tried to write the pickled cluster state. It seems to fail on something in the boto instances, and it seems reasonable (and backwards compatible, since the pickle format ID id stored in the data stream) to use the highest protocol available, which avoids the assertion failure.

Starting cluster `aws-slurm` with 2 compute nodes.
Starting cluster `aws-slurm` with 1 frontend nodes.
(this may take a while...)
Traceback (most recent call last):
File "/home/jwm/.bin/env/elasticluster/bin/elasticluster", line 9, in
load_entry_point('elasticluster==1.1-dev', 'console_scripts', 'elasticluster')()
File "/home/jwm/.bin/env/elasticluster/local/lib/python2.7/site-packages/elasticluster-1.1_dev-py2.7.egg/elasticluster/main.py", line 175, in main
app.run()
File "/home/jwm/.bin/env/elasticluster/local/lib/python2.7/site-packages/pyCLI-2.0.3-py2.7.egg/cli/app.py", line 245, in run
return self.post_run(returned)
File "/home/jwm/.bin/env/elasticluster/local/lib/python2.7/site-packages/pyCLI-2.0.3-py2.7.egg/cli/app.py", line 241, in run
returned = self.main(*args)
File "/home/jwm/.bin/env/elasticluster/local/lib/python2.7/site-packages/elasticluster-1.1_dev-py2.7.egg/elasticluster/main.py", line 160, in main
return self.params.func()
File "/home/jwm/.bin/env/elasticluster/local/lib/python2.7/site-packages/elasticluster-1.1_dev-py2.7.egg/elasticluster/subcommands.py", line 70, in __call__
return self.execute()
File "/home/jwm/.bin/env/elasticluster/local/lib/python2.7/site-packages/elasticluster-1.1_dev-py2.7.egg/elasticluster/subcommands.py", line 187, in execute
cluster.start(min_nodes=min_nodes)
File "/home/jwm/.bin/env/elasticluster/local/lib/python2.7/site-packages/elasticluster-1.1_dev-py2.7.egg/elasticluster/cluster.py", line 334, in start
self.repository.save_or_update(self)
File "/home/jwm/.bin/env/elasticluster/local/lib/python2.7/site-packages/elasticluster-1.1_dev-py2.7.egg/elasticluster/repository.py", line 197, in save_or_update
pickle.dump(cluster, storage)
File "/usr/lib/python2.7/pickle.py", line 1370, in dump
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.7/pickle.py", line 405, in save_reduce
self.memoize(obj)
File "/usr/lib/python2.7/pickle.py", line 244, in memoize
assert id(obj) not in self.memo
AssertionError