Package: dblite
dbliteSimple query interface to SQL databases.Supports SQLite and Postgres.Full API documentation available athttps://suurjaak.github.io/dblite.InstallationUsageQueriesName quotingAdapters and convertersRow factoriesObject-relational mappingSQLitePostgresAPIDependenciesInstallationpipinstalldbliteUsageimportdblitedblite.init(":memory:")# Open SQLite in-memory databasedblite.executescript("CREATE TABLE test (id INTEGER PRIMARY KEY, val TEXT)")dblite.insert("test",val=None)foriinrange(5):dblite.insert("test",{"val":i})dblite.fetchone("test",id=1)# Queries return dictionariesdblite.fetchall("test",order="val",limit=3)dblite.update("test",{"val":None},val=2)dblite.fetchall("test",val=("IN",range(3)))dblite.delete("test",id=5)dblite.executescript("DROP TABLE test")Provides a simple context manager for transactions:# dblite.init("sqlite path" or {..postgres opts..})withdblite.transaction()astx:tx.insert("test",val="will be rolled back")tx.update("test",{"val":"will be rolled back"},id=0)raisedblite.Rollback# Rolls back uncommitted actions and exits blocktx.insert("test",val="this will never be reached")print("continuing, Rollback does not propagate out of managed context")withdblite.transaction(commit=False)astx:tx.insert("test",val="will be committed")tx.commit()# Commits uncommitted actionstx.insert("test",val="will be rolled back")tx.rollback()# Rolls back uncommitted actionstx.insert("test",val="will be rolled back automatically by Transaction")Queries directly on the Database object use autocommit mode:
every action query gets committed immediately.Database instances are usable as context managers:withdblite.init("my.sqlite")asdb:# File will be closed on exiting blockdb.executescript("CREATE TABLE test (id INTEGER PRIMARY KEY, val TEXT)")db.insert("test",id=1,val="value")The first Database instance created for engine is cached per engine,
consecutiveinit()calls with no connection options yield the cached instance.# Create default database for SQLitedblite.init(":memory:")# All module-level queries use the very first createddblite.fetchall("sqlite_master")# Create default database for Postgresdblite.init("postgresql://user@localhost/mydb")# All module-level queries use the very first database createddblite.fetchone("sqlite_master")# Access the second default Database by engine namedblite.init(engine="postgres").fetchall("information_schema.columns")# Grab references to eitherdb1=dblite.init(engine="sqlite")db2=dblite.init(engine="postgres")QueriesColumns toSELECTcan be a string, or a sequence of strings:# Result: SELECT *dblite.fetchone("test")dblite.fetchone("test","*")# Result: SELECT iddblite.fetchone("test","id")dblite.fetchone("test",["id"])# Result: SELECT id, valdblite.fetchone("test","id, val")dblite.fetchone("test",["id","val"])# Can be arbitrary SQL expressions, invoking functions and assigning aliasesdblite.fetchone("test","COUNT(*) AS total")Keyword arguments are added toWHEREclause, or toVALUESclause forINSERT:myid=dblite.insert("test",val="lorem")dblite.update("test",{"val":"lorem ipsum"},id=myid)dblite.fetchone("test",id=myid)dblite.delete("test",val="lorem ipsum")WHEREclause supports simple equality match, binary operators,
collection lookups ("IN","NOT IN"), raw SQL strings, or
arbitrary SQL expressions. Used SQL needs to be supported by the underlying engine.dblite.fetchall("test",val="ciao")dblite.fetchall("test",where={"id":("<",10)})dblite.fetchall("test",id=("IN",list(range(5))))dblite.fetchall("test",val=("!=",None))dblite.fetchall("test",val=("IS NOT",None))dblite.fetchall("test",val=("LIKE","%a%"))dblite.fetchall("test",where=[("LENGTH(val)",(">",4))])dblite.fetchall("test",where=[("EXPR",("LENGTH(val) > ?",[4]))])dblite.fetchall("test",where=[("EXPR",("val = ? OR id > ? or id < ?",[0,1,2]))])WHEREarguments areAND-ed together,ORneeds subexpressions:# Result: WHERE (id < 1 OR id > 2) AND val = 3dblite.fetchall("test",where=[("id < ? OR id > ?",[1,2]),("val",3)])Argument for key-value parameters, likeWHEREorVALUES,
can be a dict, or a sequence of key-value pairs:# Result: SET val = 'done' WHERE id = 1dblite.update("test",values={"val":"done"},where=[("id",1)])Argument for sequence parameters, likeGROUP BY,ORDER BY, orLIMIT,
can be an iterable sequence like list or tuple, or a single value.# Result: SELECT * FROM test GROUP BY valdblite.fetchall("test",group="val")# Result: SELECT * FROM test GROUP BY id, valdblite.fetchall("test",group="id, val")dblite.fetchall("test",group=("id","val"))# Result: SELECT * FROM test ORDER BY iddblite.fetchall("test",order="id")dblite.fetchall("test",order="id ASC")dblite.fetchall("test",order=("id",True))# Result: SELECT * FROM test ORDER BY id ASC val DESCdblite.fetchall("test",order="id, val DESC")dblite.fetchall("test",order=["id",("val",False)])dblite.fetchall("test",order=[("id",True),("val",False)])dblite.fetchall("test",order=[("id","ASC"),("val","DESC")])# Result: SELECT * FROM test LIMIT 2 OFFSET 0dblite.fetchall("test",limit=2)dblite.fetchall("test",limit=(2,0))dblite.fetchall("test",limit=(2,-1))dblite.fetchall("test",limit=(2,None))# Result: SELECT * FROM test LIMIT 2 OFFSET 10dblite.fetchall("test",limit=(2,10))# Result: SELECT * FROM test OFFSET 10dblite.fetchall("test",limit=(-1,10))dblite.fetchall("test",limit=(None,10))Name quotingTable and column name strings are not quoted automatically. Names with whitespace
or non-alphanumeric characters or reserved words can be quoted withDatabase.quote()andTransaction.quote():withdblite.init("my.sqlite")asdb:db.executescript("CREATE TABLE test (id INTEGER PRIMARY KEY,%sTEXT)"%db.quote("my column"))db.insert("test",{"id":1,db.quote("my column"):"value"})forrowindb.select("test"):print(row["my column"])Note that in Postgres, quoted identifiers are case-sensitive.Table and column names that were given as data classes and class members,arequoted automatically if their values need escaping,
see name quoting in objects.Adapters and convertersProvides options to register custom adapters and converters,
to auto-adapt Python types to database types in query parameters,
and to auto-convert database types to Python types in query results.dblite.init(":memory:")dblite.register_adapter(json.dumps,(dict,list,tuple))dblite.register_converter(json.loads,"JSON")dblite.executescript("CREATE TABLE test (id INTEGER PRIMARY KEY, data JSON)")dblite.insert("test",id=1,data={"some":{"nested":["data",1,2]}})dblite.fetchone("test")# `data` is auto-converted to Python dictionaryRow factoriesA custom row factory can be specified, to return results as desired type instead of dictionaries.defkvfactory(cursor,row):# Returns row as [(colname, value), ].returnlist(zip([c[0]forcincursor.description],row))dblite.init(":memory:")dblite.register_row_factory(kvfactory)dblite.executescript("CREATE TABLE test (id INTEGER PRIMARY KEY, val TEXT)")forrowindblite.select("sqlite_master"):print(row)# Prints [("type", "table"), ("name", "test"), ..]Row factory can also be specified per Database:db=dblite.init(":memory:")db.row_factory=lambdacursor,row:rowdb.executescript("CREATE TABLE test (id INTEGER PRIMARY KEY, val TEXT)")forrowindb.select("sqlite_master"):print(row)# Prints ("table", "test", ..)Database row factory overrides the globally registered factory, if any.Object-relational mappingdblite uses dictionaries as rows by default, but can just as easily operate with
various types of data classes and objects, using classes in place of table names
and objects in place of data dictionaries.And if data attributes have been declared as properties on the class,
the class properties can be used directly in dblite in place of column names,
e.g. forORDER BYclause.(Such data descriptor properties are automatically available forpropertyattributes, classes with__slots__, and namedtuples).Data classesschema="CREATE TABLE devices (id INTEGER PRIMARY KEY, name TEXT)"classDevice(object):def__init__(self,id=None,name=None):self._id=idself._name=namedefget_id(self):returnself._iddefset_id(self,id):self._id=idid=property(get_id,set_id)defget_name(self):returnself._namedefset_name(self,name):self._name=namename=property(get_name,set_name)Device.__name__="devices"# cls.__name__ will be used as table namedblite.init(":memory:").executescript(schema)device=Device(name="lidar")device.id=dblite.insert(Device,device)device.name="solid-state lidar"dblite.update(Device,device,{Device.id:device.id})device=dblite.fetchone(Device,Device.id,where=device)print(device.name)# Will be None as we only selected Device.idfordeviceindblite.fetchall(Device,order=Device.name):print(device.id,device.name)dblite.delete(Device,device)It is also possible to use very simple data classes with no declared properties.schema="CREATE TABLE devices (id INTEGER PRIMARY KEY, name TEXT)"classDevice(object):def__init__(self,id=None,name=None):self.id=idself.name=nameDevice.__name__="devices"# cls.__name__ will be used as table namedblite.init(":memory:").executescript(schema)device=Device(name="lidar")device.id=dblite.insert(Device,device)device.name="solid-state lidar"dblite.update(Device,device,id=device.id)device=dblite.fetchone(Device,"id",where=device)print(device.name)# Will be None as we only selected Device.idfordeviceindblite.fetchall(Device,order="name"):print(device.id,device.name)dblite.delete(Device,device)Classes with__slots__schema="CREATE TABLE devices (id INTEGER PRIMARY KEY, name TEXT)"classDevice(object):__slots__=("id","name")def__init__(self,id=None,name=None):self.id=idself.name=nameDevice.__name__="devices"# cls.__name__ will be used as table namedblite.init(":memory:").executescript(schema)device=Device(name="lidar")device.id=dblite.insert(Device,device)device.name="solid-state lidar"dblite.update(Device,device,id=device.id)device=dblite.fetchone(Device,Device.id,where=device)print(device.name)# Will be None as we only selected Device.idfordeviceindblite.fetchall(Device,order=Device.name):print(device.id,device.name)dblite.delete(Device,device)namedtupleWorks with bothcollections.namedtupleandtyping.NamedTuple.schema="CREATE TABLE devices (id INTEGER PRIMARY KEY, name TEXT)"Device=collections.namedtuple("devices",("id","name"))dblite.init(":memory:").executescript(schema)device=Device(id=None,name="lidar")device_id=dblite.insert(Device,device)device=Device(id=device_id,name="solid-state lidar")dblite.update(Device,device,{Device.id:device_id})device=dblite.fetchone(Device,Device.id,where=device)print(device.name)# Will be None as we only selected Device.idfordeviceindblite.fetchall(Device,order=Device.name):print(device.id,device.name)dblite.delete(Device,device)Name quoting in objectsdblite automatically quotes table and column names in queries when using objects as arguments.schema='CREATE TABLE "restaurant bookings" ("group" TEXT, "table" TEXT, "when" TIMESTAMP, "PATRON" BOOLEAN)'Booking=collections.namedtuple("_",("group","table","when","patron"))Booking.__name__="restaurant bookings"dblite.init(":memory:").executescript(schema)booking1=Booking("Squirrel Charity","Table 16",datetime.datetime(2022,12,30,20,30),False)booking2=Booking("The Three Henrys","Table 23",datetime.datetime(2022,12,30,19,00),True)dblite.insert(Booking,booking1)dblite.insert(Booking,booking2)forbookingindblite.fetchall(Booking,order=Booking.when):print(booking.when,booking.group,booking.table,booking.patron)For more thorough examples on using objects, see test/test_orm.py.In Postgres, schema definition is looked up from the database to ensure properly cased
names in queries, as cased names for Postgres tables and columns must use the declared form.
If there is no exact match for the Python name in database, falls back to lower-case name,
or if name is lower-case, falls back to cased name if database has a single matching cased name.SQLiteSQLite connection parameter needs to be a valid path or a path-like object,
or the special":memory:"for transient in-memory database.Connection flags default tocheck_same_thread=False, detect_types=sqlite3.PARSE_DECLTYPES,
can be overridden on init:dblite.init("/path/to/my.db",detect_types=False)Note that SQLite connections do not support multiple concurrent isolated transactions,
transaction state is shared per connection. To mitigate this, Transaction contexts
in SQLite default to exclusive access:dblite.init(":memory:")withdblite.transaction()astx:print("Entering another Transaction with-block will block until this exits.")This can be overridden forSELECT-only transactions:dblite.init(":memory:")withdblite.transaction(exclusive=False)astx:print("Will only be doing SELECT queries, no need for exclusion.")tx.fetchall("test")PostgresPostgres connection parameters can be:Postgres URI scheme"postgresql://user:pass@hostname:port/dbname?parameter1=val1&.."Postgres keyword-value format"user=myuser password=mypass host=myhost port=myport dbname=myname .."dictionary of connection options{"user": "myuser", "host": "myhost", ..}# These are all equivalent:dblite.init("postgresql://myuser@myhost/mydb")dblite.init("user=myuser host=myhost dbname=mydb")dblite.init({"user":"myuser","host":"myhost","dbname":"mydb"})Postgres connection parameters can also be specified in OS environment,
via standard Postgres environment variables likePGUSERandPGPASSWORD.By default uses a pool of 1..4 connections per Database.withdblite.init("host=localhost user=postgres dbname=mydb",maxconn=1):print("Use a pool of only 1 connection.")withdblite.init("host=localhost user=postgres dbname=mydb",minconn=4,maxconn=8):print("Use a pool of 4..8 connections.")Postgres transactions can specify database table schema name up front:dblite.init("host=localhost user=postgres dbname=mydb")withdblite.transaction(schema="information_schema")astx:forrowintx.fetchall("columns",table_schema="public",order="table_name, ordinal_position"):print(row["table_name"],row["column_name"],row["data_type"])Postgres transactions support server-side cursors for iterative data access,
fetching and materializing rows in batches:dblite.init("host=localhost user=postgres dbname=bigdata")withTransaction(lazy=True)astx:# Can only run a single queryfori,rowinenumerate(tx.select("some really huge table")):print("Processing row #%s"%i)# Can also specify size of fetched batches (default is 2000 rows)withTransaction(lazy=True,itersize=100)astx:fori,rowinenumerate(tx.select("some really huge table")):print("Processing row #%s"%i)Note thatexecutescript()in Postgres forces an internal reload of schema metadata,
allowinginsert()to return inserted primary key value for newly created tables,
and query parameters to be auto-cast to expected column types.APINameDescriptiondblite.init()returns a openeddblite.Databaseobject, the first created if no options givendblite.fetchall()runsSELECT, returns all rowsdblite.fetchone()runsSELECT, returns a single row, orNonedblite.insert()INSERTa single row into table, returns inserted IDdblite.insertmany()INSERTmultiple rows into table, returns a list of inserted IDsdblite.select()runsSELECT, returns cursordblite.update()UPDATEtable, returns affected row countdblite.delete()DELETEfrom table, returns affected row countdblite.execute()executes SQL with arguments, returns cursordblite.executemany()executes SQL against all parameter sequencesdblite.executescript()executes SQL as a script of one or more SQL statementsdblite.close()closes the database and all pending transactions, if opendblite.transaction()returnsdblite.Transactioncontext managerdblite.register_adapter()registers function to auto-adapt given Python types to database types in query parametersdblite.register_converter()registers function to auto-convert given database types to Python in query resultsdblite.register_row_factory()registers function to produce query results as custom typedblite.DatabaseDatabase.fetchall()runsSELECT, returns all rowsDatabase.fetchone()runsSELECT, returns a single row, orNoneDatabase.insert()INSERTa single row into table, returns inserted IDDatabase.insertmany()INSERTmultiple rows into table, returns a list of inserted IDsDatabase.select()runsSELECT, returns cursorDatabase.update()UPDATEtable, returns affected row countDatabase.delete()DELETEfrom table, returns affected row countDatabase.execute()executes SQL with arguments, returns cursorDatabase.executemany()executes SQL against all parameter sequencesDatabase.executescript()executes SQL as a script of one or more SQL statementsDatabase.transaction()returnsdblite.Transactioncontext managerDatabase.open()opens database connection if not already openDatabase.close()closes the database and all pending transactions, if openDatabase.closedwhether database is not openDatabase.cursordatabase engine cursor objectDatabase.row_factorycustom row factory, asfunction(cursor, row tuple)Database.ENGINEunderlying database engine,"sqlite"for SQLite3 and"postgres"for PostgreSQLdblite.TransactionTransaction.fetchall()runsSELECT, returns all rowsTransaction.fetchone()runsSELECT, returns a single row, orNoneTransaction.insert()INSERTa single row into table, returns inserted IDTransaction.insertmany()INSERTmultiple rows into table, returns a list of inserted IDsTransaction.select()runsSELECT, returns cursorTransaction.update()UPDATEtable, returns affected row countTransaction.delete()DELETEfrom table, returns affected row countTransaction.execute()executes SQL with arguments, returns cursorTransaction.executemany()executes SQL against all parameter sequencesTransaction.executescript()executes SQL as a script of one or more SQL statementsTransaction.commit()commits pending actions, if anyTransaction.rollback()rolls back pending actions, if anyTransaction.close()closes the transaction, performing commit or rollback as specifiedTransaction.closedwhether transaction is not openTransaction.cursordatabase engine cursor objectTransaction.databasetransactionDatabaseinstanceDatabase.ENGINEunderlying database engine,"sqlite"for SQLite3 and"postgres"for PostgreSQLDependenciesPython 3 or Python 2.7six (https://pypi.org/project/six)If using Postgres:psycopg2 (https://pypi.org/project/psycopg2)