Package: cs.sqlalchemy-utils
Assorted utility functions to support working with SQLAlchemy.Latest release 20230612:Use cs.fileutils.lockfile context manager instead of makelockfile.Rename arranged_session to orchestrated_session.Some tweaks around connection closes, still edging towards a good work practice for easily doing short lived stuff (for cs.ebooks.calibre).Functionauto_session(function)Decorator to run a function in a session if one is not presupplied.
The functionfunctionruns within a transaction,
nested if the session already exists.Seewith_sessionfor details.ClassBasicTableMixinUseful methods for most tables.Functionfind_json_field(column_value, field_name, *, infill=False)Descend a JSONable Python objectcolumn_valuetofield_name.
Returncolumn_value(possibly infilled),final_field,final_field_name.This supports database row columns which are JSON columns.Parameters:column_value: the original value of the columnfield_name: the field within the column to locateinfill: optional keyword parameter, defaultFalse.
If true,column_valueand its innards will be filled in asdicts
to allow deferencing thefield_name.Thefield_nameis astrconsisting of a period ('.') separated sequence of field parts.
Each field part becomes a key to index the column mapping.
These keys are split into the leading field parts
and the final field part,
which is returned asfinal_field_nameabove.Thefinal_fieldreturn value above
is the mapping within whichfinal_field_valuemay lie
and wherefinal_field_valuemay be set.
Note: it may not be present.If a leading key is missing andinfillis true
the corresponding part of thecolumn_valueis set to an empty dictionary
in order to allow deferencing the leading key.
This includes the case whencolumn_valueitself isNone,
which is why thecolumn_valueis part of the return.If a leading key is missing andinfillis false
this function will raise aKeyErrorfor the portion of thefield_namewhich failed.Examples:>>> find_json_field({'a':{'b':{}}}, 'a.b')
({'a': {'b': {}}}, {'b': {}}, 'b')
>>> find_json_field({'a':{}}, 'a.b')
({'a': {}}, {}, 'b')
>>> find_json_field({'a':{'b':{}}}, 'a.b.c.d')
Traceback (most recent call last):
    ...
KeyError: 'a.b.c'
>>> find_json_field({'a':{'b':{}}}, 'a.b.c.d', infill=True)
({'a': {'b': {'c': {}}}}, {}, 'd')
>>> find_json_field(None, 'a.b.c.d')
Traceback (most recent call last):
    ...
KeyError: 'a'
>>> find_json_field(None, 'a.b.c.d', infill=True)
({'a': {'b': {'c': {}}}}, {}, 'd')Functionget_json_field(column_value, field_name, *, default=None)Return the value offield_namefromcolumn_valueor a defaault if the field is not present.Parameters:column_value: the original value of the columnfield_name: the field within the column to locatedefault: default value to return if the field is not present,
default:NoneExamples:>>> get_json_field({'a': 1}, 'a')
1
>>> get_json_field({'b': 1}, 'a')
>>> get_json_field({'a': {}}, 'a.b')
>>> get_json_field({'a': {'b': 2}}, 'a.b')
2ClassHasIdMixinInclude an "id"Columnas the primary key.Functionjson_column(*da, **dkw)Class decorator to declare a virtual column name on a table
where the value resides inside a JSON column of the table.Parameters:cls: the class to annotateattr: the virtual column name to present as a row attributejson_field_name: the field within the JSON column
used to store this value,
default the same asattrjson_column_name: the name of the associated JSON column,
default'info'default: the default value returned by the getter
if the field is not present,
defaultNoneExample use:Base = declarative_base()
...
@json_column('virtual_name', 'json.field.name')
class TableClass(Base):
  ...This annotates the class with a.virtual_nameproperty
which can be accessed or set,
accessing or modifying the associated JSON column
(in this instance, the columninfo,
accessinginfo['json']['field']['name']).Functionlog_level(func, a, kw, level=None)Temporarily set the level of the default SQLAlchemy logger tolevel.
Yields the logger.NOTE: this is not MT safe - competing Threads can mix log levels up.ClassORM(cs.resources.MultiOpenMixin, cs.context.ContextManagerMixin)A convenience base class for an ORM class.This defines a.Baseattribute which is a newDeclarativeBaseand provides various Session related convenience methods.
It is also aMultiOpenMixinsubclass
supporting nested open/close sequences and use as a context manager.MethodORM.__init__(self, db_url, serial_sessions=None):
Initialise the ORM.Ifserial_sessionsis true (defaultTruefor SQLite,Falseotherwise)
then allocate a lock to serialise session allocation.
This might be chosen with SQL backends which do not support
concurrent sessions such as SQLite.In the case of SQLite there's a small inbuilt timeout in
an attempt to serialise transactions but it is possible to
exceed it easily and recovery is usually infeasible.
Instead we use theserial_sessionsoption to obtain a
mutex before allocating a session.Functionorm_auto_session(method)Decorator to run a method in a session derived fromself.ormif a session is not presupplied.
Intended to assist classes with a.ormattribute.Seewith_sessionfor details.Functionproxy_on_demand_field(*da, **dkw)A decorator to provide a field value on demand
via a functionfield_func(self,db_row,session=session).Example:@property
@proxy_on_demand_field
def formats(self,db_row,*,session):
    """ A mapping of Calibre format keys to format paths
        computed on demand.
    """
    return {
        fmt.format:
        joinpath(db_row.path, f'{fmt.name}.{fmt.format.lower()}')
        for fmt in db_row.formats
    }FunctionRelationProxy(relation, columns: Union[str, Tuple[str], List[str]], *, id_column: Optional[str] = None, orm=None)Construct a proxy for a row from a relation.Parameters:relation: an ORM relation for which this will be a proxycolumns: a list of the column names to cache,
or a space separated string of the column namesid_column: options primary key column name,
default fromBasicTableMixin.DEFAULT_ID_COLUMN:'id'orm: the ORM, default fromrelation.ormThis is something of a workaround for applications which dip
briefly into the database to obtain information instead of
doing single long running transactions or sessions.
Instead of keeping the row instance around, which might want
to load related data on demand after its source session is
expired, we keep a proxy for the row with cached values
and refetch the row at need if further information is required.Typical use is to construct this proxy class as part
of the__init__of a larger class which accesses the database
as part of its operation.
The example below is based oncs.ebooks.calibre.CalibreTree:def __init__(self, calibrepath):
  super().__init__(calibrepath)
  # define the proxy classes
  class CalibreBook(RelationProxy(self.db.books, [
      'author',
      'title',
  ])):
    """ A reference to a book in a Calibre library.
    """
    @typechecked
    def __init__(self, tree: CalibreTree, dbid: int, db_book=None):
      self.tree = tree
      self.dbid = dbid
    ... various other CalibreBook methods ...
  self.CalibreBook = CalibreBook

def __getitem__(self, dbid):
  return self.CalibreBook(self, dbid, db_book=db_book)Functionset_json_field(column_value, field_name, value, *, infill=False)Set a newvalueforfield_nameofcolumn_value.
Return the newcolumn_value.Parameters:column_value: the original value of the columnfield_name: the field within the column to locatevalue: the value to store asfield_nameinfill: optional keyword parameter, defaultFalse.
If true,column_valueand its innards will be filled in asdicts
to allow deferencing thefield_name.As withfind_json_field,
a trueinfillmay modifycolumn_valueto providefield_namewhich is why this function returns the newcolumn_value.Examples:>>> set_json_field({'a': 2}, 'a', 3)
{'a': 3}
>>> set_json_field({'a': 2, 'b': {'c': 5}}, 'b.c', 4)
{'a': 2, 'b': {'c': 4}}
>>> set_json_field({'a': 2}, 'b.c', 4)
Traceback (most recent call last):
    ...
KeyError: 'b'
>>> set_json_field({'a': 2}, 'b.c', 4, infill=True)
{'a': 2, 'b': {'c': 4}}
>>> set_json_field(None, 'b.c', 4, infill=True)
{'b': {'c': 4}}ClassSQLAState(cs.threads.ThreadState, _thread._local)Thread local state for SQLAlchemy ORM and session.Functionusing_session(orm=None, session=None)A context manager to prepare an SQLAlchemy session
for use by a suite.Parameters:orm: optional reference ORM,
an object with a.session()method for creating a new session.
Default: if needed, obtained from the globalstate.orm.session: optional existing session.
Default: the globalstate.sessionif notNone,
otherwise created byorm.session().If a new session is created, the new session and reference ORM
are pushed onto the globalsstate.sessionandstate.ormrespectively.If an existing session is reused,
the suite runs within a savepoint fromsession.begin_nested().Functionwith_orm(function, *a, orm=None, **kw)Callfunctionwith the suppliedormin the shared state.Functionwith_session(function, *a, orm=None, session=None, **kw)Callfunction(*a,session=session,**kw), creating a session if required.
The functionfunctionruns within a transaction,
nested if the session already exists.
If a new session is created
it is set as the default session in the shared state.This is the inner mechanism of@auto_sessionandORM.auto_session.Parameters:function: the function to calla: the positional parametersorm: optional ORM class with a.session()context manager method
such as theORMbase class supplied by this module.session: optional existing ORM sessionkw: other keyword arguments, passed tofunctionOne oformorsessionmust be notNone; ifsessionisNonethen one is made fromorm.session()and used as
a context manager.Thesessionis also passed tofunctionas
the keyword parametersessionto support nested calls.Release LogRelease 20230612:Use cs.fileutils.lockfile context manager instead of makelockfile.Rename arranged_session to orchestrated_session.Some tweaks around connection closes, still edging towards a good work practice for easily doing short lived stuff (for cs.ebooks.calibre).Release 20230212:
ORM.init: drop case_sensitive, no longer supported?Release 20220606:BasicTableMixin: provide DEFAULT_ID_COLUMN='id', by_id() has new optional id_column parameter.RelationProxy factory to make base classes which proxy a relation, for circumstances where you want to minimise access to the db itself.ORM.engine_keywords: turn on echo mode only if "SQL" in $DEBUG.Release 20220311:
Many updates and small fixes.Release 20210420:ORM: drop .Session from docstring, no longer used.Rename ORM.sessionmaker to ORM._sessionmaker, not for public use.ORM: replace session with arranged_session, which allocates a session in conformance with ORM.serial_sessions (serial sessions are used with SQLite).Drop @ORM.auto_session and @ORM.orm_method decorators, no longer used.SQLAState.new_session: use orm.arranged_session(), use begin_nested(); SQLAState.auto_session: use begin_nested().Release 20210322:
Delete escaped debug code which issued a RuntimeError.Release 20210321:Default session support, particularly though an ORM's .sqla_state per-Thread state object - this allows removal of a lot of plumbing and @auto_session decoration.Support for serialised sessions, for db backend where only one session may be active at a time; this brings easy support for multithreaded SQLite access.Release 20210306:Rename _state to state, making it public.Some other internal changes.Release 20201025:New BasicTableMixin and HasIdMixin classes with useful methods and a typicalidColumn respectively.Assorted fixes and improvements.Release 20190830.1:
Have the decorators set .module.Release 20190830:
@json_column: small docstring improvement.Release 20190829:Bugfix @json_column setter: mark the column as modified for the ORM.New push_log_level context manager and @log_level decorator to temporarily change the SQLAlchemy logging handler level.Release 20190812:Make ORM a MultiOpenMixin.get_json_field: use forgottendefaultparameter.Other minor changes.Release 20190526:Support for virtual columns mapped to a JSON column interior value:New functions find_json_field, get_json_field, set_json_field.New decorator @json_column for declaritive_base tables.Release 20190517:Make ORM._Session private session factory the public ORM.Session factory for external use.with_session: preexisting sessions still trigger a session.begin_nested, removes flush/commit tension elsewhere.Release 20190403:Rename @ORM.orm_auto_session to @ORM.auto_session.New @orm_auto_session decorator for methods of objects with a .orm attribute.Release 20190319.1:
Initial release. ORM base class, @auto_session decorator.