Package: pyodb
PyODBPython Object DataBase (PyODB) is a SQLite3 ORM library aiming to be as simple to use as possible.
This library is supposed to be used for testing, in small projects, when wanting to export complex
python data in a well structured format or for local inter-process data caching.Basics:PyODB uses the python built-in SQLite3 implementation as database and has no external dependencies.PyODB can take any (non-primitive) type you pass into it, extracts its members and creates a
database schema representing the type. Sub-types of the main type are also extracted recursively.Saves instances of known types into the database and loads them using basic filtering options.Also provides a caching module 'PyODBCache' which may be used for inter-process data caching.When using multiprocessing excessively race conditions may occur.All data (cache & non-cache) can persist or be deleted upon closing of the process.SetupBasic UsageMore in-depth examplesLimitationsHow it worksConverting Primitive TypesConverting Custom TypesSimple custom typeCustom types contained in a collection or dictDynamic type definitionsSetupThere are two main ways to install the package. The simplest version is using pip:pipinstallpyodbSince there are no external dependencies this should work without problem.Another way is to install the package from source. To do this you have to pull or download this
repository and move into the package folder. Once inside the folder you can install the package:pipinstall.Basic UsageA very basic usage example is using the library to save a simple custom type and load it elsewhere.To do this a custom class with some members is needed. We will use the example class below:classMyType:some_data:list[str]some_number:int|Nonedef__init__(self,number:int):self.some_data=["Hello","World"]self.some_number=numberdef__repr__(self)->str:returnf"MyType Number:{self.some_number}"Next you can simply import pyodb and add MyType and try saving some instances:frompyodbimportPyODB# Create PyODB instance with persistent data and type-definitionspyodb=PyODB(persistent=True)# Add type and save some instancespyodb.add_type(MyType)pyodb.save(MyType(1))pyodb.save_multiple([MyType(2),MyType(3),MyType(4),MyType(5)])Now let's say you need the data in another process or in another python program altogether.Create a new instance - types are re-loaded from the database since persistent was set to TrueThis time, when the process exits, the data will be deleted because persistent is False by defaultpyodb=PyODB()Now you can get a Selector instance which is used to select and filter data loaded from the database.select=pyodb.select(MyType)## filter loaded instances by some_number > 2select.gt(some_number=2)## Load results into resultresult=select.all()print(result)The same select can also be written as a one-liner.result=pyodb.select(MyType).gt(some_number=2).all()print(result)You can easily delete some saved entries with the Deleter using filters like the Selector.deleted=pyodb.delete(MyType).gt(some_number=2).commit()print(f"Deleted{deleted}entries")# Count remaining entriescount=pyodb.select(MyType).count()print(f"{count}entries remaining")You can also scrap all data from the database keeping the table definitions usingclear().pyodb.clear()Known types can also be shown and removed.
Removing types does have some restrictions. For more information look at thePyODB Examples.print(pyodb.known_types)pyodb.remove_type(MyType)print(pyodb.known_types)As you can see loading and removing data is quite simple. Basic filtering is also possible in both
select and delete.More in-depth examplesBelow are links to two documents containing more comprehensive examples of different functions.
They also contain best practices regarding performance and some possible error cases.For PyODB examples please refer toPyODB examplesFor PyODBCache examples please refer toCaching examplesLimitationsWhat this library can store:Primitive Datatypes (int, float, str, bool)ListsDictionariesCustom TypesWhat this library cannot store:Fields containing functions (maybe in the future...)Fields that are dependent on external states or state changesImportant Notice:Storing of types imported from external libraries should be avoided.
Depending of how deep an library type is nested one single datapoint could lead to the creation of
dozens of tables. One for every nested sub-type. Additionally there may be some fields containing
stateful connections or other illegal members.As example -> Storing aPoolManagerfromurllib3would result in these tables:PoolManagerUrlProxyConfigSSLContextSSLObjectSSLSocketSSLSession...ConnectionPoolLifoQueueQueue...This is still a very basic example. As a rule of thumb; The higher level the library the deeper the
objects are nested on average.To prevent this themax recursion depthshould not be set above 3. Using more than this is
discouraged because of performance reasons as well. You can also exclude certain members by
defining the__odb_members__dict containing all wanted class members.How it worksWhen a python class is added to the PyODB schema the classes are converted to SQL Tables and
inserted into the database schema.IMPORTANT: If a class changes between executions the table retains the old definition.
It is advisable to reset the database manually in case it was persisted.The UML Diagram below shows what such a conversion looks like:Converting Primitive TypesPrimitive python datatypes are simply converted to SQL datatypes.SQL DatatypePython DatatypesINTEGERintREALfloatNUMERICboolTEXTstrBLOB (pickled)bytes, list, dict, tuple, setConverting Custom TypesThere is a certain amount of scenarios when converting custom types. Below is a list of
all handled scenarios and the corresponding rules that deal with them. All of these scenarios are
also depicted in the example diagram above.Henceforth Origin references the original class whose type is parsed into a SQL Table.Simple custom typeWhen a simple custom type is contained by the Origin a table is created for the sub-type as
well. The sub-type's table uses a_parent_column which references the_id_of the
Origin and a_parent_table_column referencing the type/table-name of the Origin.When an Origin instance is inserted the sub-type instance is saved in it's own table. The table
which the instance is saved in is then referenced by the Origin's sub-type column. As example:Here are the columns of an imaginary origin type:Column NameTypeContent_id_TEXTOriginIDsubtypeTEXTMySubTypeAnd here are the columns of the imaginary sub-type:Column NameTypeContent_parent_TEXTOriginID_parent_tableTEXTOriginCustom types contained in a collection or dictBecause of performance considerations and to keep my sanity all collection types and dicts are
converted tobytesobjects via pickle and then saved to the Database as BLOBs.Collections and dicts can therefore not be loaded by other programs besides python unlike all other
datatypes.Dynamic type definitionsDynamic type definitions (Union) of custom types can also be saved by PyODB.
Only primitive datatypes must be unambiguous.The only exception is None. Data may always be defined with None as an option.This is allowed:classPerson:name:strdob:intinfo:str|Nonecard:CreditCard|BankcardThis is not:classSquare:length:int|floatImportant Notice:A table will be created for every possible Datatype!