Package: objsize
objsizeTheobjsizePython package allows for the exploration and
measurement of an object’s complete memory usage in bytes, including its
child objects. This process, often referred to as deep size calculation,
is achieved through Python’s internal Garbage Collection (GC) mechanism.Theobjsizepackage is designed to ignore shared objects, such asNone, types, modules, classes, functions, and lambdas, because they
are shared across many instances. One of the key performance features ofobjsizeis that it avoids recursive calls, ensuring a faster and
safer execution.Key FeaturesTraverse objects’ subtreeCalculates the size of objects, including nested objects (deep size), in bytesExclude non-exclusive objectsExclude specified objects subtreeProvides flexibility by allowing users to define custom handlers for:Object’s size calculationObject’s referents (i.e., its children)Object filter (skip specific objects)DocumentationobjsizeTraversal over Python's objects subtree and calculating the total size of the subtree (deep size).Installpipinstallobjsize==0.7.0Basic UsageCalculate the size of the object including all its members in bytes.>>>importobjsize>>>objsize.get_deep_size(dict(arg1='hello',arg2='world'))340It is possible to calculate the deep size of multiple objects by passing
multiple arguments:>>>objsize.get_deep_size(['hello','world'],dict(arg1='hello',arg2='world'),{'hello','world'})628Complex Dataobjsizecan calculate the size of an object’s entire subtree in
bytes regardless of the type of objects in it, and its depth.Here is a complex data structure, for example, that include a self
reference:my_data=list(range(3)),list(range(3,6))classMyClass:def__init__(self,x,y):self.x=xself.y=yself.d={'x':x,'y':y,'self':self}def__repr__(self):returnf"{self.__class__.__name__}()"my_obj=MyClass(*my_data)We can calculatemy_objdeep size, including its stored data.>>>objsize.get_deep_size(my_obj)724We might want to ignore non-exclusive objects such as the ones stored inmy_data.>>>objsize.get_deep_size(my_obj,exclude=[my_data])384Or simply letobjsizedetect that automatically:>>>objsize.get_exclusive_deep_size(my_obj)384Non Shared Functions or Classesobjsizefilters functions, lambdas, and classes by default since
they are usually shared among many objects. For example:>>>method_dict={"identity":lambdax:x,"double":lambdax:x*2}>>>objsize.get_deep_size(method_dict)232Some objects, however, as illustrated in the above example, have unique
functions not shared by other objects. Due to this, it may be useful to
count their sizes. You can achieve this by providing an alternative
filter function.>>>objsize.get_deep_size(method_dict,filter_func=objsize.shared_object_filter)986Notes:The default filter function isobjsize.traverse.shared_object_or_function_filter().When usingobjsize.traverse.shared_object_filter(), shared functions and
lambdas are also counted, but builtin functions are still excluded.Special CasesSome objects handle their data in a way that prevents Python’s GC from
detecting it. The user can supply a special way to calculate the actual
size of these objects.Case 1:torchUsing a simple calculation of the object size won’t work fortorch.Tensor.>>>importtorch>>>objsize.get_deep_size(torch.rand(200))72So the user can define its own size calculation handler for such cases:importobjsizeimportsysimporttorchdefget_size_of_torch(o):# `objsize.safe_is_instance` catches `ReferenceError` caused by `weakref` objectsifobjsize.safe_is_instance(o,torch.Tensor):returnsys.getsizeof(o)+(o.element_size()*o.nelement())else:returnsys.getsizeof(o)Then use it as follows:>>>objsize.get_deep_size(...torch.rand(200),...get_size_func=get_size_of_torch...)872The above approach may neglect the object’s internal structure. The user
can helpobjsizeto find the object’s hidden storage by supplying it
with its own referent and filter functions:importobjsizeimportgcimporttorchdefget_referents_torch(*objs):# Yield all native referentsyield fromgc.get_referents(*objs)foroinobjs:# If the object is a torch tensor, then also yield its storageiftype(o)==torch.Tensor:yieldo.untyped_storage()# `torch.dtype` is a common object like Python's types.MySharedObjects=(*objsize.SharedObjectOrFunctionType,torch.dtype)deffilter_func(o):returnnotobjsize.safe_is_instance(o,MySharedObjects)Then use these as follows:>>>objsize.get_deep_size(...torch.rand(200),...get_referents_func=get_referents_torch,...filter_func=filter_func...)928Case 2:weakrefUsing a simple calculation of the object size won’t work forweakref.proxy.>>>fromcollectionsimportUserList>>>o=UserList([0]*100)>>>objsize.get_deep_size(o)1032>>>importweakref>>>o_ref=weakref.proxy(o)>>>objsize.get_deep_size(o_ref)72To mitigate this, you can provide a method that attempts to fetch the
proxy’s referents:importweakrefimportgcdefget_weakref_referents(*objs):yield fromgc.get_referents(*objs)foroinobjs:iftype(o)inweakref.ProxyTypes:try:yieldo.__repr__.__self__exceptReferenceError:passThen use it as follows:>>>objsize.get_deep_size(o_ref,get_referents_func=get_weakref_referents)1104After the referenced object will be collected, then the size of the
proxy object will be reduced.>>>delo>>>gc.collect()>>># Wait for the object to be collected>>>objsize.get_deep_size(o_ref,get_referents_func=get_weakref_referents)72Object Size SettingsTo avoid repeating the input settings when handling the special cases
above, you can use theObjSizeSettingsclass.>>>torch_objsize=objsize.ObjSizeSettings(...get_referents_func=get_referents_torch,...filter_func=filter_func,...)>>>torch_objsize.get_deep_size(torch.rand(200))928>>>torch_objsize.get_deep_size(torch.rand(300))1328SeeObjSizeSettingsfor the
list of configurable parameters.TraversalA user can implement its own function over the entire subtree using the
traversal method, which traverses all the objects in the subtree.>>>foroinobjsize.traverse_bfs(my_obj):...print(o)...MyClass(){'x': [0, 1, 2], 'y': [3, 4, 5], 'd': {'x': [0, 1, 2], 'y': [3, 4, 5], 'self': MyClass()}}[0, 1, 2][3, 4, 5]{'x': [0, 1, 2], 'y': [3, 4, 5], 'self': MyClass()}210543Similar to before, non-exclusive objects can be ignored.>>>foroinobjsize.traverse_exclusive_bfs(my_obj):...print(o)...MyClass(){'x': [0, 1, 2], 'y': [3, 4, 5], 'd': {'x': [0, 1, 2], 'y': [3, 4, 5], 'self': MyClass()}}{'x': [0, 1, 2], 'y': [3, 4, 5], 'self': MyClass()}AlternativePympleralso supports
determining an object deep size viapympler.asizeof(). There are two
main differences betweenobjsizeandpympler.objsizehas additional features:Traversing the object subtree: iterating all the object’s
descendants one by one.Excluding non-exclusive objects. That is, objects that are also
referenced from somewhere else in the program. This is true for
calculating the object’s deep size and for traversing its
descendants.objsizehas a simple and robust implementation with significantly
fewer lines of code, compared topympler. The Pympler
implementation uses recursion, and thus have to use a maximal depth
argument to avoid reaching Python’s max depth.objsize, however,
uses BFS which is more efficient and simple to follow. Moreover, the
Pympler implementation carefully takes care of any object type.objsizearchives the same goal with a simple and generic
implementation, which has fewer lines of code.License: BSD-3Copyright (c) 2006-2023, Liran Funaro.
All rights reserved.Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.Neither the name of the copyright holder nor the
names of its contributors may be used to endorse or promote products
derived from this software without specific prior written permission.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS”
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.