Package: bounter
Bounter -- Counter for large datasetsBounter is a Python library, written in C, for extremely fast probabilistic counting of item frequencies in massive datasets. UnlikeCounter, it uses only a small fixed memory footprint.Why Bounter?Bounter lets you count how many times an item appears, similar to Python's built-indictorCounter:frombounterimportbountercounts=bounter(size_mb=1024)# use at most 1 GB of RAMcounts.update([u'a','few',u'words',u'a',u'few',u'times'])# count item frequenciesprint(counts[u'few'])# query the counts2However, unlikedictorCounter, Bounter can process huge collections where the items would not even fit in RAM. This commonly happens in Machine Learning and NLP, with tasks likedictionary buildingorcollocation detectionthat need to estimate counts of billions of items (token ngrams) for their statistical scoring and subsequent filtering.Bounter implements approximative algorithms using optimized low-level C structures, to avoid the overhead of Python objects. It lets you specify the maximum amount of RAM you want to use. In the Wikipedia example below, Bounter uses 31x less memory compared toCounter.Bounter is also marginally faster than the built-indictandCounter, so wherever you can represent youritems as strings(both byte-strings and unicode are fine, and Bounter works in both Python2 and Python3), there's no reason not to use Bounter instead except:When not to use Bounter?Beware, Bounter is only a probabilistic frequency counter and cannot be relied on for exact counting. (You can't expect a data structure with finite size to hold infinite data.)
Example of Bounter failing:frombounterimportbounterbounts=bounter(size_mb=1)bounts.update(str(i)foriinrange(10000000))bounts['100']0Please useCounterordictwhen such exact counts matter. When they don't matter, like in most NLP and ML applications with huge datasets, Bounter is a very good alternative.InstallationBounter has no dependencies beyond Python >= 2.7 or Python >= 3.3 and a C compiler:pipinstallbounter# install from PyPIOr, if you prefer to install from thesource tar.gz:pythonsetup.pytest# run unit testspythonsetup.pyinstallHow does it work?No magic, just some clever use of approximative algorithms and solid engineering.In particular, Bounter implements three different algorithms under the hood, depending on what type of "counting" you need:Cardinality estimation: "How many unique items are there?"frombounterimportbountercounts=bounter(need_counts=False)counts.update(['a','b','c','a','b'])print(counts.cardinality())# cardinality estimation3print(counts.total())# efficiently accumulates counts across all items5This is the simplest use case and needs the least amount of memory, by using theHyperLogLog algorithm(built on top of Joshua Andersen'sHLLcode).Item frequencies: "How many times did this item appear?"frombounterimportbountercounts=bounter(need_iteration=False,size_mb=200)counts.update(['a','b','c','a','b'])print(counts.total(),counts.cardinality())# total and cardinality still work(5L,3L)print(counts['a'])# supports asking for counts of individual items2This uses theCount-min Sketch algorithmto estimate item counts efficiently, in afixed amount of memory. See theAPI docsfor full details and parameters.As a further optimization, Count-min Sketch optionally support alogarithmic probabilistic counter:bounter(need_iteration=False): default option. Exact counter, no probabilistic counting. Occupies 4 bytes (max value 2^32) per bucket.bounter(need_iteration=False, log_counting=1024): an integer counter that occupies 2 bytes. Values up to 2048 are exact; larger values are off by +/- 2%. The maximum representable value is around 2^71.bounter(need_iteration=False, log_counting=8): a more aggressive probabilistic counter that fits into just 1 byte. Values up to 8 are exact and larger values can be off by +/- 30%. The maximum representable value is about 2^33.Such memory vs. accuracy tradeoffs are sometimes desirable in NLP, where being able to handle very large collections is more important than whether an event occurs exactly 55,482x or 55,519x.Full item iteration: "What are the items and their frequencies?"frombounterimportbountercounts=bounter(size_mb=200)# default version, unless you specify need_items or need_countscounts.update(['a','b','c','a','b'])print(counts.total(),counts.cardinality())# total and cardinality still work(5L,3)print(counts['a'])# individual item frequency still works2print(list(counts))# iterator returns keys, just like Counter[u'b',u'a',u'c']print(list(counts.iteritems()))# supports iterating over key-count pairs, etc.[(u'b',2L),(u'a',2L),(u'c',1L)]Stores the keys (strings) themselves in addition to the total cardinality and individual item frequency (8 bytes). Uses the most memory, but supports the widest range of functionality.This option uses a custom C hash table underneath, with optimized string storage. It will remove its low-count objects when nearing the maximum alotted memory, instead of expanding the table.For more details, see theAPI docstringsor read theblog.Example on the English WikipediaLet's count the frequencies of all bigrams in the English Wikipedia corpus:withsmart_open('wikipedia_tokens.txt.gz')aswiki:forlineinwiki:words=line.decode().split()bigrams=zip(words,words[1:])counter.update(u' '.join(pair)forpairinbigrams)print(counter[u'czech republic'])42099The Wikipedia dataset contained 7,661,318 distinct words across 1,860,927,726 total words, and 179,413,989 distinct bigrams across 1,857,420,106 total bigrams. Storing them in a naive built-indictwould consume over 31 GB RAM.To test the accuracy of Bounter, we automatically extractedcollocations(common multi-word expressions, such as "New York", "network license", "Supreme Court" or "elementary school") from these bigram counts.We compared the set of collocations extracted from Counter (exact counts, needs lots of memory) vs Bounter (approximate counts, bounded memory) and present the precision and recall here:AlgorithmTime to buildMemoryPrecisionRecallF1 scoreCounter(built-in)32m 26s31 GB100%100%100%bounter(size_mb=128, need_iteration=False, log_counting=8)19m 53s128 MB95.02%97.10%96.04%bounter(size_mb=1024)17m 54s1 GB100%99.27%99.64%bounter(size_mb=1024, need_iteration=False)19m 58s1 GB99.64%100%99.82%bounter(size_mb=1024, need_iteration=False, log_counting=1024)20m 05s1 GB100%100%100%bounter(size_mb=1024, need_iteration=False, log_counting=8)19m 59s1 GB97.45%97.45%97.45%bounter(size_mb=4096)16m 21s4 GB100%100%100%bounter(size_mb=4096, need_iteration=False)20m 14s4 GB100%100%100%bounter(size_mb=4096, need_iteration=False, log_counting=1024)20m 14s4 GB100%99.64%99.82%Bounter achieves a perfect F1 score of 100% at 31x less memory (1GB vs 31GB), compared to a built-inCounterordict. It is also 61% faster.Even with just 128 MB (250x less memory), its F1 score is still 96.04%.SupportUseGithub issuesto report bugs, and ourmailing listfor general discussion and feature ideas.Bounteris open source software released under theMIT license.Copyright (c) 2017RaRe Technologies