Package: madoka
madokaMadoka is an implementation of a Count-Min sketch data structure for summarizing data streams.String-int pairs in a Madoka-Sketch may take less memory than in a standard Python dict, Counter, Redis.Counting error rate is about 0.0911 %More details are described inBenchmark.ipynbThis module is based onmadokaC++ library.NOTE: Madoka-Sketch does not have index of keys. so Madoka-Sketch can not dump all keys such as Python dict’sdict.keys(). However, when setkparameter to costructer,most_commonmethod (returns key and value as many ask) is available.Contributions are welcome!Installation$ pip install madokaClassMadoka has some classes having same interface. These classes are vary in value data type. So you can choose for your purpose.For example, if you wants to count float data, it’s preferable to choose CroquisFloat class or CroquisDouble class.Sketch
- storing unsigned long long (64bit) and fast implementationCroquisFloat
- storing float (32bit)CroquisDouble
- storing double (64bit)CroquisUint8
- storing unsigned char (8bit)CroquisUint16
- storing unsigned short (16bit)CroquisUint32
- storing unsigned int (32bit)CroquisUint64
- storing unsigned long long (64bit)UsageFrom here, I will describe about Sketch class.
But, Croquis classes have also same interfaces mostly.
So you can use other classes by the same way as Sketch class.
In that case, you should replace to intended class from “Sketch”.Create a new sketch>>>importmadoka>>>sketch=madoka.Sketch()Sketch madoka.Sketch([width=1048576, max_value=35184372088831, path=’’, flags=0, seed=0, k=5])widthis a size of register. If you are worrying about gap, you should increasewidthvalue. The largerwidthis, the fewer mistakes madoka makes in estimating value. But, the largerwidthis, the larger memory consumption is.Permission ofpathshould be 644kmeans Top-K used bymost_commonmethod. if you don’t want to usemost_commonmethod, then I recommend to setk=0so it is slightly fast.madoka.Sketch()callsmadoka.Sketch.create(), so you don’t have to explicitly callcreate()in initializationIncrement a key value>>>sketch['mami']+=1or>>>sketch.inc('mami')int inc(key[, key_length=0])Note thatkey_lengthis automatically determined when not givingkey_length. Thus, the order of parameters differs from original madoka C++ library.Add a value to the current key value>>>sketch['mami']+=6or>>>sketch.add('mami',6)int add(key, value[, key_length=0])Note thatkey_lengthis automatically determined when not givingkey_length. Thus, the order of parameters differs from original madoka C++ library.Update a key value>>>sketch['mami']=6or>>>sketch.set('mami',6)void set(key, value[, key_length=0])Note thatset()does nothing when the given value is not greater than the current key value.Also note that the new value is saturated when the given value is greater than the upper limit.Additionally note thatkey_lengthis automatically determined when not givingkey_length. Thus, the order of parameters differs from original madoka C++ library.Get a key value>>>sketch['mami']or>>>sketch.get('mami')int get(key[, key_length=0])Note thatkey_lengthis automatically determined when not givingkey_length. Thus, the order of parameters differs from original madoka C++ library.Get all values>>>sketch.values()generator<int> values()Note that processing time increases according to sketch’s width. But this method may be slow, so I recommend setting width to less than 1000000 when creating sketch.Save a sketch to a file>>>sketch.save('example.madoka')void save(path)Permission ofpathshould be 644Load a sketch from a file>>>sketch.load('example.madoka')void load(path)Permission ofpathshould be 644Clear a sketch>>>sketch.clear()void clear()Delete all key-value pairs. It differs fromcreate()in maintaining current settings.Initialize a sketch with settings change>>>sketch.create()void create([width=0, max_value=0, path=NULL, flags=0, seed=0])Permission of file given topathshould be 644Copy a sketch>>>sketch.copy(othersketch)void copy(Sketch)Merge two sketches>>>sketch+=other_sketchor>>>sketch.merge(othersketch)void merge(Sketch[, lhs_filter=None, rhs_filter=None])lhs_filter is applied for self.sketch, rhs_filter is applied for given sketchShrink a sketch>>>sketch.shrink(sketch,width=1000)void shrink(Sketch[, width=0, max_value=0, filter=None, path=None, flags=0])When width > 0, width must be less than source sketchPermission ofpathshould be 644Get summed sketch>>>summed_sketch=sketch+other_sketchCreate summed sketch, So it does not break original sketchesGet summed sketch by dict>>>summed_sketch=sketch+{'mami':1,'kyoko':2}Create summed sketch, So it does not break original sketchesCheck whether sketch contains key value>>>'mami'insketchGet inner product of two sketches>>>sketch.inner_product(other_sketch)list<float> inner_product(Sketch)Returns [inner product, square length of left hands sketch (float), square length of right hands sketch (float)]Get median value>>>sketch['madoka']=1>>>sketch['mami']=2>>>sketch['sayaka']=3>>>sketch['kyouko']=4>>>sketch['homura']=5>>>sketch.median()# => 3int or float median()Apply filter into all values>>>sketch.filter(lambdax:x+1)void filter(Callable[, apply_zerovalue=False])If apply_zerovalue = True, filter_method is applied also 0 values (It may be slow) (from version 0.6 or later)Note that processing time increases according to sketch’s width. If you feel this method is slow, I recommend setting width to less than 1000000 when creating sketchSet values from dict>>>sketch.fromdict({'mami':14,'madoka':13})or>>>sketch+={'mami':14,'madoka':13}void fromdict(dict)Get most common keys>>>sketch.most_common()generator most_common([k=5])returns key-value pair as many askNote that this method is required to setkparameter in constructer.LicenseWrapper code is licensed under New BSD License.BundledmadokaC++ library is licensed under the Simplified BSD License.CHANGES0.7.1 (2018-02-11)Fix bug for Python 2.70.7 (2018-02-11)Support Python 3.5 - 3.7Unsupport Python 2.6Addmost_common()function (requiring to givek=nparameter to constructer)Addmedian()functionFixed a bug that madoka::FILE_PRELOAD does not work0.6 (2014-11-23)Support Python 3.4Improve processing time ofinner_product()Fixshrink()method bugChangefilter()methods paramSupport with-statementImplement increment-add from dict
- (e.g.) summed_sketch = sketch + dict; sketch += dict0.5 (2014-04-08)Add Croquis classes handling some data types (e.g., float, uint8)Given length=True to inner product method, returns also square length of both left hands and right hands sketchAdd fromdict method0.4 (2014-03-30)Implement dict-like interface (e.g., sketch[‘key’])Add filter() methodAdd values() method for dumping all values0.3 (2014-03-14)Key length is automatically determined when it is not givenRemove filter functionSlightly decreasing amount of memory usage0.2 (2013-10-12)Simplify the step of creating new sketch.0.1 (2013-10-11)Initial release.