Package: tinysort
Experimental in-memory Pythonic MapReduce inspired bySpotify’s luigi framework.The Word Count ExampleCurrently the only MapReduce implementation is in-memory and serial, but an
implementation with parallelized map and reduce phases will be added.fromcollectionsimportCounterimportjsonimportreimportsysfromtinymr.memoryimportMemMapReduceclassWordCount(MemMapReduce):"""
    The go-to MapReduce example.

    Don't worry, a better example is on its way:
    https://github.com/geowurster/tinymr/issues/11
    """# Counting word occurrence does not benefit from sorting post-map or# post-reduce and our `final_reducer()` doesn't care about key order# so we can disable sorting for a speed boost.sort_map=Falsesort_reduce=Falsesort_final_reduce=Falsedef__init__(self):"""
        Stash a regex to strip off punctuation so we can grab it later.
        """self.pattern=re.compile('[\W_]+')defmapper(self,line):"""
        Take a line of text from the input file and figure out how many
        times each word appears.

        An alternative, simpler implementation would be:

            def mapper(self, item):
                for word in item.split():
                    word = self.pattern.sub('', word)
                    if word:
                        yield word.lower(), 1

        This simpler example is more straightforward, but holds more data
        in-memory.  The implementation below does more work per line but
        potentially has a smaller memory footprint.  Like anything
        MapReduce the implementation benefits greatly from knowing a lot
        about the input data.
        """# Normalize all words to lowercaseline=line.lower().strip()# Strip off punctuationline=[self.pattern.sub('',word)forwordinline]# Filter out empty stringsline=filter(lambdax:x!='',line)# Return an iterable of `(word, count)` pairsreturnCounter(line).items()defreducer(self,key,values):"""
        Just sum the number of times each word appears in the entire
        dataset.

        At this phase `key` is a word and `values` is an iterable producing
        all of the values for that word from the map phase.  Something like:

            key = 'the'
            values = (1, 1, 2, 2, 1)

        The word `the` appeared once on 3 lines and twice on two lines for
        a total of `7` occurrences, so we yield:

            ('the', 7)
        """yieldkey,sum(values)defoutput(self,pairs):"""
        Normally this phase is where the final dataset is written to disk,
        but since we're operating in-memory we just want to re-structure as
        a dictionary.

        `pairs` is an iterator producing `(key, iter(values))` tuples from
        the reduce phase, and since we know that we only produced one key
        from each reduce we want to extract it for easier access later.
        """return{k:tuple(v)[0]fork,vinpairs}wc=WordCount()withopen('LICENSE.txt')asf:out=wc(f)print(json.dumps(out,indent=4,sort_keys=True))Truncated output:{"a":1,"above":2,"advised":1,"all":1,"and":8,"andor":1}Word Count WorkflowInternally, the workflow looks like this:Input data:$head-10LICENSE.txtNew BSD License

Copyright (c) 2015, Kevin D. Wurster
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.MapCount occurrences of each word in every line.# Input lineline='Copyright (c) 2015, Kevin D. Wurster'# Sanitized wordswords=['Copyright','c','2015','Kevin','D','Wurster']# Return tuples with word as the first element and count as the secondpairs=[('Copyright',1),('c',1),('2015',1),('Kevin',1),('D',1),('Wurster',1)]PartitionOrganize all of the(word, count)pairs byword.  Thewordkeys are
kept at this point in case the data is sorted.  Sorting grabs the second to last
key, so the data could be partitioned on one key and sorted on another with(word, sort, count).  The second to last key is used for sorting so the keys
that appear below match thewordonly because asortkey was not given.Words that appear in the input text on multiple lines have multiple(word, count)pairs.  Acountof2would indicate a word that
appeared twice on a single line, but our input data does not have this
condition.  Truncated output below.  The dictionary values are lists containing
tuples to allow for a sort key, which is explained elsewhere.{'2015':[(1,)]'above':[(1,)]'all':[(1,)]'and':[(1,),(1,),(1,)]'are':[(1,),(1,)]'binary':[(1,)]'bsd':[(1,)]'c':[(1,)]'code':[(1,)]}ReduceSumcountfor eachword.# The ``reducer()`` receives a key and an iterator of valueskey='the'values=(1,1,1)defreducer(key,values):yieldkey,sum(values)PartitionThe reducer does not _have_ to produces the same key it was given, so the data
is partitioned by key again, which is superfluous for this wordcount example.
Again the keys are kept in case the data is sorted and only matchwordbecause an optionalsortkey was not given.  Truncated output below.{'2015':[(1,)]'above':[(1,)]'all':[(1,)]'and':[(3,)]'are':[(2,)]'binary':[(1,)]'bsd':[(1,)]'c':[(1,)]'code':[(1,)]}OutputThe default implementation is to return(key, iter(values))pairs from thefinal_reducer(), which would look something like:values=[('the',(3,)),('in',(1,),]But a dictionary is much more useful, and we know that we only got a single
value for eachwordin the reduce phase, so we can extract that value
and produce a dictionary.return{k:tuple(v)[0]fork,vinvalues}Thetuple()call is included because the data in thevaluekey is
_always_ an iterable object but _may_ be an iterator.  Callingtuple()expands the iterable and lets us get the first element.Developing$gitclonehttps://github.com/geowurster/tinymr.git$cdtinymr$pipinstall-e.\[dev\]$py.testtests--covtinymr--cov-reportterm-missingLicenseSeeLICENSE.txtChangelogSeeCHANGES.md