Package: mercury-settrie
Mercury SettrieWhat is this?TLDR: A SetTrie is a container of sets that performs efficient subset and superset queries.Settrieis a Python library implemented in C++ to create, update and query SetTrie objects.Highly efficient C++ implementation.100% pythonic interface: objects are serializable, use iterators and support native Python sets.What problems is Settrie good for?Settriewas born from the need of a better implementation of the algorithm for our recommender system. It has direct application to text
indexing for search in logarithmic time of documents containing a set of words. Think of a collection of documents as a container of
the set of words that each document has. Finding all the documents that contain a set of words is finding the superset of the set of words
in the query. A SetTrie will give you the answer --the list of document names-- in logarithmic time. The data structure is also used in
auto-complete applications.What data size can Settrie tackle?Settrie is a C++ implementation with a Python interface. It is single threaded and can seamlessly operate over really large collections
of sets. Note that the main structure is a tree and a tree node is exactly 20 bytes, a billion nodes is 20 Gb, of course plus some
structures to store identifiers, etc. Note that the tree is compressing the documents by sharing the common parts and documents are already
compressed by considering them a set of words. An of-the-shelf computer can store in RAM a representation of terabytes of documents and
query result in much less than typing speed.How does this implementation compare to other Python implementations?It is about 200 times faster and 20 times more memory efficient that a pure Python implementation.Try it without any installation in Google ColabThe API is very easy to use. You can see this benchmark notebook for reference.Benchmark: Comparing two Settrie implementationsInstallpipinstallmercury-settrieUsagefromsettrieimportSetTrie# Create a SetTrie objectstt=SetTrie()# Insert some setsstt.insert({2,3},'id1')stt.insert({2,3,4.4},'id2')stt.insert({'Mon','Tue'},'days')# Find id by setprint(stt.find({2,3}))# Find ids of all supersetsforidinstt.supersets({2,3}):print(id)# Find ids of all subsetsforidinstt.subsets({2,3}):print(id)# Nested iteration over the sets and elementsforstinstt:print(st.id)foreinst.elements:print('  ',e)# Store as a pickle file fileimportpicklewithopen('my_settrie.pickle','wb')asf:pickle.dump(stt,f)# Load from a pickle filewithopen('my_settrie.pickle','rb')asf:tt=pickle.load(f)# Check that they are identicalfort,stinzip(tt,stt):assertt.id==st.idforet,estinzip(t.elements,st.elements):assertet==est# Remove sets by idstt.remove('id2')stt.remove('days')# After many .remove() calls, the tree has nodes marked as dirty,# calling .purge() removes them completely and frees RAM.stt.purge()Clone and set up a development environment to work with itTo work with Settrie command line or develop Settrie, you can set up an environment with git, gcc, make and the following tools:catch2 (Already included in source code)doxygen 1.9.5 or better (to render C++ documentation)mkdocs 1.4.2 or better (to render Python documentation)swig 4.0.2python 3.x with appropriate paths to python.h (see Makefile)gitclonehttps://github.com/BBVA/mercury-settrie.gitcdmercury-settrie/src

makeMake without arguments gives help. Try all the options. Everything should work assuming the tools are installed.DocumentationPython APIC++ APIAlgorithm description: Index Data Structure for Fast Subset and Superset QueriesLicenseApache License
                   Version 2.0, January 2004
                http://www.apache.org/licenses/

     Copyright 2021-23, Banco de Bilbao Vizcaya Argentaria, S.A.

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0