Package: nanoset
nanoset.pyA memory-optimized wrapper for Python sets likely to be empty.⏱️ TL;DRSave up to 85% of the memory used by emptysetinstances in your code with
a single line of code:fromnanosetimportPicoSetasset🚩 Table of ContentsOverviewAbout Python memory usageSimple example usecaseImplementationStatisticsInstallingAPI ReferenceLicense🗺️ Overview🐏 About Python memory usagePython is a great programming language (fight me), but sometimes you start
questioning why it does things in certain ways. Since Python 2.3, the standard
library provides thesetcollection, which is a specialized container for membership testing. On the
contrary to the ubiquitouslistcollection,setis not ordered (or, more accurately,does not let you access
the order it stores the elements in). The other feature ofsetis that just
like its mathematical counterpart, it does not allow duplicates, which is very
useful for some algorithms. However,sets are memory-expensive:>>>importsys>>>sys.getsizeof(list())56>>>sys.getsizeof(set())216An empty set takes more than three times the memory of an empty list! For some
data structures or objects with a lot ofsetattributes, they can quickly
cause a very large part of the memory to be used for nothing. This is even more
sad when you are used toRust, where mostcollectionsallocate lazily.This is wherenanosetcomes to the rescue:>>>importnanoset>>>sys.getsizeof(nanoset.NanoSet())48>>>sys.getsizeof(nanoset.PicoSet())32Actually, that's a lie, but keep reading.💡 Simple example usecaseLet's imagine we are building an ordered graph data structure, where we may
want to storetaxonomic data,
or any other kind of hierarchy. We can simply define the graphs and its nodes
with the two following classes:classGraph:root:Nodenodes:Dict[str,Node]classNode:neighbors:Set[node]This makes adding an edge and querying for an edge existence between two nodes
anO(1)operation, and iterating over all the nodes anO(n)operation, which
is mot likely what we want here. We usesetand notlistbecause we want to
avoid storing an edge in duplicate, which is a sensible choice. But now let's
look at thestatisticsof theNCBITaxonproject, the
database for Organismal Classification developed by the US National Center for
Biotechnology Information:Metrics
    Number of classes*: 1595237
    Number of individuals: 0
    Number of properties: 0
    Classes without definition: 1595237
    Classes without label: 0
    Average number of children: 12
    Classes with a single child: 40319
    Maximum number of children: 41761
    Classes with more than 25 children: 0
    Classes with more than 1 parent: 0
    Maximum depth: 38
    Number of leaves**: 1130671According to these, we are going to have1,130,671leaves for a total of1,595,237nodes, which means70.8%of empty sets. Now you may think:Ok, I got this. But in this case, I just need a special case for leaves, where
instead of storing an empty set ofneighbors, I store a reference toNonewhen that set would be empty. I can then replace that reference with an actual
set only when I want to add new edges from that node. Problem solved!Well, glad we are on the same level: this is whatnanosetdoes for you!🔨 ImplementationActually, it's not magic at all. Just imagine a classNanoSetthat works as
aproxyto an actual Pythonsetit wraps, but which is only allocated when some data
actually needs to be stored:classNanoSet(collections.abc.Set):def__init__(self,iterable=None):self.inner=NoneifiterableisNoneelseset(iterable)defadd(self,element):ifself.innerisNone:self.inner=set()self.inner.add(element)# ... the rest of the `set` API ...That's about it! However, doing it like so in Python would not be super
efficient, as the resulting object would be48bytes. Usingslots, this can be
reduced to40bytes, which is on par to what we get withNanoSet.Note that these values are only when the inner set is empty!When actually
allocating the set to store our values, we allocate an additional232bytes
of data. This means that usingNanoSetcreates an overhead, since a
non-empty set will now weigh264bytes (248bytes forPicoSet).Well, I was way better off with my approach of storingOptional[Set]everywhere then, I don't want to pay any additional cost for nonempty sets!Sure. But that would mean changing your whole code. And actually, you may not
gain that much memory from doing that compared to usingnanoset, since the
only time the wrapper performs badly is when you have a load factor of more than
90%. Furthermore, just to give you some perspective,sys.getsizeof(1)is28bytes.By the way, you didn't mentionPicoSet. How did you manage to get that down
to32bytes, when a slotted Python object can't be less that40bytes?Easy:PicoSetis basicallyNanoSet, but without an implementation of theGarbage Collector protocol.
This saves us8bytes of object memory, but comes with a drawback: the
garbage collector cannot see the set allocatedinsidethePicoSet. This
does not change anything for execution, but debugging with a memory profiler
will be harder. Here is an example where we allocate1,000,000singletons
first withNanoSet, then withPicoSet, usingguppy3to check the heap:>>>l=[nanoset.NanoSet({x})forxinrange(1000000)]>>>guppy.hpy().heap()Partitionofasetof3036763objects.Totalsize=304996526bytes.IndexCount%Size%Cumulative%Kind(class/dictofclass)01000044332161052487121610524871set1100000033480000001626410524887nanoset.NanoSet...311308716880330085140499list...>>>l=[nanoset.PicoSet({x})forxinrange(1000000)]>>>guppy.hpy().heap()Partitionofasetof1036905objects.Totalsize=44998965bytes.IndexCount%Size%Cumulative%Kind(class/dictofclass)010000009632000000713200000071nanoset.PicoSet19608712752243271275289list...On the second run, we have about the same order of allocated memory, saving16 MB(16bytes saved by switching fromNanoSettoPicoSettimes1,000,000instances). However, the garbage collector has no idea where
some of the memory is, becausePicoSethides the sets it allocates (this is
fine: it will be deallocated along with thePicoSet).As such, I'd advise avoiding usingPicoSetwhen debugging, which can be done
easily with Python's__debug__flag:if__debug__:fromnanosetimportNanoSetassetelse:fromnanosetimportPicoSetassetThis will causePicoSetto be used instead ofNanoSetwhen running Python
with the-Oflag.📈 StatisticsOkay, so let's do some maths. WithS = 232the size of an allocated set,sthe size of the wrapper (48forNanoSet,32forPicoSet), thexpercentage of nonempty sets in our data structure, the relative size
of our sets is:if we're usingset:S * x / (S * x) = 100%(we use that as a reference)if we're usingnanoset:((S + s) * x + s * (100 - x)) / (S * x)This gives us the following graph, which shows how much memory you can save
depending of the ratio of empty sets you have at runtime:If we get back to our NCBITaxon example, we have a total of1,595,237nodes
and1,130,671leaves, which means that by using sets we are allocating1,595,237 * 232 = 328.6 MiBof memory simply forsetafter the whole
taxonomy is loaded. If we useNanoSethowever, we
can reduce this to168.7 MiB, or even to144.4 MiBwithPicoSet!We just saved about 45% memory just by usingNanoSetin place ofset.🔧 InstallingThis module is implemented in Rust, but nativePython wheelsare compiled for the following platforms:Windows x86-64: CPython 3.5, 3.6, 3.7, 3.8Linux x86-64: CPython 3.5, 3.6, 3.7, 3.8OSX x86-64: CPython 3.5, 3.6, 3.7, 3.8If you platform is not among these, you will need aworking Rustnightlytoolchainas well as thesetuptools-rustlibrary installed to build the extension module.Then, simply install withpip:$pipinstall--usernanoset📖 API ReferenceWell, this is a comprehensive wrapper forset, so you can just read thestandard library documentation.
Except for some very particular edge-cases,NanoSetandPicoSetboth pass thesettest suiteofCPython.There are however things youcan'tdo:Subclassing aPicoSetor aNanoSet.Weakrefing aPicoSetor aNanoSet.Checking for membership in a plainsetorfrozensetwith implicit
conversion tofrozenset.Creating adictfrom aPicoSetor aNanoSetwithout rehashing keys.📜 LicenseThis library is provided under the open-sourceMIT license.