dfm/emcee — Issue #13: multiprocessing pickling error
Author: joeyrichar
Created: 2012-02-22 20:53:53+00:00
State: open

I run into the following error when setting the threads to > 1:

>>> sampler = emcee.EnsembleSampler(nwalkers , 14 , lnpost , args =['12071006', priors, orderedkeys ], threads = 2)
>>> pos , prob , state = sampler.run_mcmc(p0 , 100)

Exception in thread Thread-3:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/threading.py", line 505, in run
self.__target(*self.__args, **self.__kwargs)
File "/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/multiprocessing/pool.py", line 313, in _handle_tasks
put(task)
PicklingError: Can't pickle : attribute lookup __builtin__.instancemethod failed

I am defining lnpostfn as a static function within my module. Curiously, I am able to pickle the lnpostfn that I use in the sampler as well as the data object that I call within the lnpostfn. So what is the multiprocessing module trying to pickle?

Here's a simplified version of my lnpostfn:

import ooplc

def lnpost(theta, name, prior, keys):
# lnprior is defined as a static function within my module
logprior = lnprior(theta, prior, keys) # compute prior

# binary is a class inside the ooplc module
binary = ooplc.binary('bin2', name, override=theta, params='new2', timegrid='coarse')
model = binary.ldF_of_t

data = binary.kepler_lcdata1
data_err = np.sqrt(binary.kepler_lcdata1**2 + theta[0]**2)

loglikelihood = sum( -0.5 * ( (binary.ldF_of_t - data) / data_err )**2)
return logprior + loglikelihood

Any insights? I do not fully understand what multiprocessing.pool is trying to pickle.