Package: mister
For all your medium data needs!Mister attempts to make running a map/reduce job approachable.When you’ve got data that isn’t really big and so you’re not quite ready
to distribute the data across a gazillian machines and stuff but would
still like an answer in a reasonable amount of time.5 minute getting startedMister needs you to define three methods:prepare(get the data
ready to be run across multiple processes),map(actually do
something with the chunks of data fromprepare), andreduce(mash all the values returned frommaptogether).Thereducemethodprepare(self,count,*args,**kwargs)Thecountis the number of processes the job will be run across, and*argsand**kwargsis whatever is passed into your child class’s__init__method. Thepreparemethod returnscountrows
containing a tuple((),{})of the arguments that will be passed to
eachmapprocess.Themapmethodmap(self,*args,**kwargs)The*argsand**kwargsare whatever was returned fromprepare. Themapmethod returns whatever you wantreduceto
use to merge all the data together.Thereducemethodreduce(self,output,value)Theoutputis the global aggregation of all thevaluearguments
thereducemethod has seen. Basically, whatever you return from onereducecall will be passed back into the nextreducecall asoutput. Thevalueargument is whatever the recently finishedmapcall returned.Bringing it all togetherSo let’s bring it all together in ourMrHelloWorldjob, first let’s
get the skeleton in place:frommisterimportBaseMisterclassMrHelloWorld(BaseMister):defprepare(self,count,*args,**kwargs):passdefmap(self,*args,**kwargs):passdefreduce(self,output,value):passNow let’s flesh out thepreparemethod:defprepare(self,count,name):# we're just going to return the number and the name we pass inforxinrange(count):yield([x,name],{})And ourmapmethod:defmap(self,x,name):return"Process{}says 'hello{}'".format(x,name)Finally, ourreducemethod:defreduce(self,output,value):ifoutputisNone:output=[]output.append(value)returnoutputRunning our job:mr=MrHelloWorld("Alice")output=mr.run()print(output)will result in:[
    "Process 1 says 'hello Alice'",
    "Process 0 says 'hello Alice'",
    "Process 2 says 'hello Alice'",
    "Process 3 says 'hello Alice'",
    "Process 4 says 'hello Alice'",
    "Process 5 says 'hello Alice'",
    "Process 6 says 'hello Alice'",
    "Process 7 says 'hello Alice'",
    "Process 8 says 'hello Alice'",
    "Process 9 says 'hello Alice'",
    "Process 10 says 'hello Alice'"
]Congrats, you just ran a map/reduce job, you are now an AI and a ML
engineer, remember me when you’re famous!Another ExampleI think word counting is the traditional map/reduce example? So here it
is:importosimportreimprotmathfromcollectionsimportCounterfrommisterimportBaseMisterclassMrWordCount(BaseMister):defprepare(self,count,path):"""prepare segments the data for the map() method"""size=os.path.getsize(path)length=int(math.ceil(size/count))start=0forxinrange(count):kwargs={}kwargs["path"]=pathkwargs["start"]=startkwargs["length"]=lengthstart+=lengthyield(),kwargsdefmap(self,path,start,length):"""all the magic happens right here"""output=Counter()withopen(path)asfp:fp.seek(start,0)words=fp.read(length)# I don't compensate for word boundaries because exampleforwordinre.split(r"\s+",words):output[word]+=1returnoutputdefreduce(self,output,count):"""take all the return values from map() and aggregate them to the final value"""ifnotoutput:output=Counter()output.update(count)returnoutput# let's count the biblepath="./testdata/bible-kjv.txt"mr=MrWordCount(path)wordcounts=mr.run()print(wordcounts.most_common(10))On my computer, the asynchronous code above runs about 3x faster than
its syncronous equivalent below:importrefromcollectionsimportCounterpath="./testdata/bible-kjv.txt"output=Counter()withopen(path)asfp:words=fp.read()forwordinre.split(r"\s+",words):output[word]+=1print(wordcounts.most_common(10))InstallationTo install, use Pip:$ pip install misterOr, to grab the latest and greatest:$ pip install --upgrade git+https://github.com/Jaymon/mister#egg=mister