Package: pandasbox
PandasBoxContainers for finding Python objects by attribute. Backed by Pandas.pip install pandasbox⚠️ This project is not being maintained. It's unclear if it's actually got a use case.Usagefrom pandasbox import PandasBox
tb = PandasBox(
    [{'item': 1, 'size': 1000, 'shape': 'square'}],  # provide a list of objects or dicts 
    {'size': int, 'shape': str})                     # specify attributes to store
tb.find('size >= 1000 and shape == "square"')        # find by attribute valueThe objects can be any container ofclass,dataclass,namedtuple, ordictobjects.There are two classes available.PandasBox: SQLite-backed container. Faster when finding a few of your objects (< 10%).PandasBox: Pandas-backed container. Faster when finding many of your objects (>= 10%).You canadd(),add_many(),update(), andremove()items from a PandasBox or PandasBox.How it worksWhen you do:PandasBox(list_of_objects, on={'size': int, 'shape': string})orPandasBox(...)A table or dataframe is created with 3 columns:sizeshapePython object referenceOnfind(), a query will run to find the matching objects.Only the relevant attributes of the object are copied into the table. The rest of the object remains in memory.An ideal use case is when you have "heavy" objects containing images / audio / large texts, plus some small
metadata fields that you want to find by. Just make a PandasBox or PandasBox on the metadata, and use it to find
the object without needing to serialize / deserialize the heavy stuff.Thepandasboxcontainers are especially good when finding by<and>. If you only need==, considerfiltered-- it is based on dict lookups which are faster in that case.APIThe API is largely the same across PandasBox and PandasBox. The only difference is initialization.PandasBox InitPandasBox(
        objs: Optional[Iterable[Any]] = None,
        on: Optional[Dict[str, Any]] = None,
        index: Optional[List[ Union[Tuple[str], str]]] = None
)Creates a PandasBox.objsis optional. It can be any container of class, dataclass, dict, or namedtuple objects.onis required. It specifies the attributes and types to index. The allowed types are float, int, bool, and str.indexspecifies the indices to create on the SQLite table. If unspecified, a single-column index is made on each
attribute.Theindexparameter is the key to getting good performance. A multi-column index can often speed upfind()operations.index=[('a', 'b', 'c'), 'd']will create a multi-column index on(a, b, c)and a single-column index
ond.  Conversely, some columns such as those containing string data may perform better without an index.SeeSQLite index documentationfor more insights.PandasBox InitCreates a PandasBox.PandasBox(
        objs: Optional[Iterable[Any]] = None,
        on: Optional[Dict[str, Any]] = None,
        : Optional[List[ Union[Tuple[str], str]]] = None
)Other API functionsThe remaining functions are the same for both PandasBox and PandasBox; exceptions as noted.find()find(where: Optional[str]) -> Listfinds objects matching the query string inwhere.Examples:tb.find('b == True and string == "okay"')tb.find('(x == 0 and y >= 1000.0) or x == 9')Ifwhereis unspecified, all objects in the container are returned.The syntax ofwhereis nearly identical between pandas and sqlite. Exceptions:In sqlite, usefind('x is null')/find('x is not null').In pandas, usefind('x != x')to match nulls, orfind('x == x')for non-nulls.Sqlite accepts either=or==for equality; pandas accepts only==.Consult the syntax forSQLite queriesorpandas queriesas needed.add(), add_many()add(obj:Any)
add_many(objs:Iterable[Any])Theadd()method adds a single object. If you have many objects, it is much faster toadd_many()than it is to
calladd()on each one.If an added object is missing an attribute, the object will still be added. The missing attribute will be given aNonevalue.update()update(self, obj: Any, updates: Dict[str, Any])updates attributes of a single object in the index.updatesis a dict containing the new values for each changed attribute, e.g.{'x': 5.5, 'b': True}.If you change an indexed object's attributes without callingupdate(), the PandasBox will be out of sync and
return inaccurate results.update()will changes both the value in the PandasBox table and the object's value.Update is fast (less than 1 ms), it's O(log n) in both sqlite and pandas.remove()remove(self, obj: Any)removes an object.Remove is fast (less than 1ms) in SQLite but slower (tens of ms) in Pandas.
This is because removing an item requires rebuilding arrays there.Container methodsYou can do the usual container things:Length:len(tb)Contains:obj in tbIteration:for obj in tb: ...PerformanceBaselineSqliteBoxPandasBoxFind 1 item0.9s0.2ms43.1msFind 10 items0.9s0.7ms44.9msFind 100 items1.0s1.9ms43.8msFind 1K items1.0s6.7ms43.9msFind 10K items1.1s27.2ms47.6msFind 100K items1.2s0.18s88.3msFind 1M items1.7s1.37s0.24sFind 10M items2.9s10.6s0.45sThis is a benchmark on random range queries against a dataset of 10 million (10^7) objects indexed on two numeric
fields.Baselineis a Python list comprehension.This is the same data as a graph, showing relative speedup. Each line is divided bybaseline.
Note that both axis labels are powers of 10. So10^3on the Y-axis indicates a 1000X speedup.Seeexamplesfor more performance tests.