klbostee/dumbo — Issue #50: Options duplicated when given to Hadoop
Author: andrix
Created: 2012-01-23 18:22:39+00:00
State: open

Hi Klbostee,

I got bitten by this problem when running a script. I'm using the last dumbo version, but I see the same problem in 0.21.29. I could find using PDB that the duplicates appear here : uhttps://github.com/klbostee/dumbo/blob/master/dumbo/core.py#L384.

I workaround it with this : http://paste.pocoo.org/show/539376/. I think the best is that the options be a set of (key, value) instead of a list, but I don't know if the options can hold some non hashable value on it.

Well, what do you think? I think there is some room to improve options handling, but better to know well from you a bit more about its values.

Btw, I'm running this line (I have to change the names, sorry) on the cluster:

python2.5 -m dumbo.cmd start mylib.mymodule.script -cachefile hdfspath/to/myfile.db#myfile.db -param locale=en-gb -param workdir=/some/other/hdfspath -param dataversion=2012-01-23T01-59-59.797550 -input /data/input/2012/01/20 -output /data/output/2012/01/20/transformed -hadoop system

Well, thanks,
Andrés