Package: cs.sqltags
Simple SQL based tagging
and the associatedsqltagscommand line script,
supporting both tagged named objects and tagged timestamped log entries.Latest release 20240201.1:
Release with the "sqltags" script.Compared tocs.fstagsand its associatedfstagscommand,
this is oriented towards large numbers of items
not naturally associated with filesystem objects.My initial use case is an activity log
(unnamed timestamped tag sets)
but I'm also using it for ontologies
(named tag sets containing metadata).Many basic tasks can be performed with thesqltagscommand line utility,
documented under theSQLTagsCommandclass below.See theSQLTagsORMdocumentation for details about how data
are stored in the database.
See theSQLTagSetdocumentation for details of how various
tag value types are supported.ClassBaseSQLTagsCommand(cs.cmdutils.BaseCommand, SQLTagsCommandsMixin, cs.tagset.TagsCommandMixin)Common features for commands oriented around anSQLTagsdatabase.Command line usage:Usage: basesqltags [-f db_url] subcommand [...]
  -f db_url SQLAlchemy database URL or filename.
            Default from $SQLTAGS_DBURL (default '~/var/sqltags.sqlite').
  Subcommands:
    dbshell
      Start an interactive database shell.
    edit criteria...
      Edit the entities specified by criteria.
    export [-F format] [{tag[=value]|-tag}...]
      Export entities matching all the constraints.
      -F format Specify the export format, either CSV or FSTAGS.
    find [-o output_format] {tag[=value]|-tag}...
      List entities matching all the constraints.
      -o output_format
                  Use output_format as a Python format string to lay out
                  the listing.
                  Default: {localtime} {headline}
    help [-l] [subcommand-names...]
      Print the full help for the named subcommands,
      or for all subcommands if no names are specified.
      -l  Long help even if no subcommand-names provided.
    import [{-u|--update}] {-|srcpath}...
      Import CSV data in the format emitted by "export".
      Each argument is a file path or "-", indicating standard input.
      -u, --update  If a named entity already exists then update its tags.
                    Otherwise this will be seen as a conflict
                    and the import aborted.
    init
      Initialise the database.
      This includes defining the schema and making the root metanode.
    log [-c category,...] [-d when] [-D strptime] {-|headline} [tags...]
      Record entries into the database.
      If headline is '-', read headlines from standard input.
      -c categories
        Specify the categories for this log entry.
        The default is to recognise a leading CAT,CAT,...: prefix.
      -d when
        Use when, an ISO8601 date, as the log entry timestamp.
      -D strptime
        Read the time from the start of the headline
        according to the provided strptime specification.
    orm define_schema
      Runs the ORM's `define_schema()` method, which creates missing tables
      and entity 0 if missing.
    shell
      Run a command prompt via cmd.Cmd using this command's subcommands.
    tag {-|entity-name} {tag[=value]|-tag}...
      Tag an entity with multiple tags.
      With the form "-tag", remove that tag from the direct tags.
      A entity-name named "-" indicates that entity-names should
      be read from the standard input.Functionglob2like(glob: str) -> strConvert a filename glob to an SQL LIKE pattern.Functionmain(argv=None)Command line mode.ClassPolyValue(PolyValue, builtins.tuple, PolyValued)Anamedtuplefor the polyvalues used in anSQLTagsORM.We express various types in SQL as one of 3 columns:float_value: forfloats andints which round trip withfloatstring_value: forstrstructured_value: a JSON transcription of any other typeThis allows SQL indexing of basic types.Note that becausestrgets stored instring_valuethis leaves us free to use "bare string" JSON to serialise
various nonJSONable types.TheSQLTagSetsclass has ato_polyvaluefactory
which produces aPolyValuesuitable for the SQL rows.
NonJSONable types such asdatetimeare converted to astrbut stored in thestructured_valuecolumn.
This should be overridden by subclasses as necessary.On retrieval from the database
the tag rows are converted to Python values
by theSQLTagSets.from_polyvaluemethod,
reversing the process above.ClassPolyValueColumnMixin(PolyValued)A mixin for classes with(float_value,string_value,structured_value)columns.
This is used by theTagsandTagMultiValuesrelations insideSQLTagsORM.ClassPolyValuedA mixin for classes with(float_value,string_value,structured_value)columns.Functionprefix2like(prefix: str, esc='\\') -> strConvert a prefix string to an SQL LIKE pattern.ClassSQLParameters(SQLParameters, builtins.tuple)The parameters required for constructing queries
or extending queries with JOINs.Attributes:criterion: the source criterion, usually anSQTCriterionsubinstancealias: an alias of the source table for use in queriesentity_id_column: theentitiesid column,alias.idif the alias is ofentities,alias.entity_idif the alias is oftagsconstraint: a filter query based onaliasClassSQLTagBasedTest(cs.tagset.TagBasedTest, cs.tagset.TagBasedTest, builtins.tuple, SQTCriterion, cs.tagset.TagSetCriterion, cs.deco.Promotable)Acs.tagset.TagBasedTestextended with a.sql_parametersmethod.ClassSQLTagProxiesA proxy for the tags supporting Python comparison =>SQLParameters.Example:sqltags.tags.dotted.name.here == 'foo'ClassSQLTagProxyAn object based on aTagname
which produces anSQLParameterswhen compared with some value.Example:>>> sqltags = SQLTags('sqlite://')
>>> sqltags.init()
>>> # make a SQLParameters for testing the tag 'name.thing'==5
>>> sqlp = sqltags.tags.name.thing == 5
>>> str(sqlp.constraint)
'tags_1.name = :name_1 AND tags_1.float_value = :float_value_1'
>>> sqlp = sqltags.tags.name.thing == 'foo'
>>> str(sqlp.constraint)
'tags_1.name = :name_1 AND tags_1.string_value = :string_value_1'ClassSQLTags(cs.tagset.BaseTagSets, cs.resources.MultiOpenMixin, cs.context.ContextManagerMixin, collections.abc.MutableMapping, collections.abc.Mapping, collections.abc.Collection, collections.abc.Sized, collections.abc.Iterable, collections.abc.Container, cs.deco.Promotable)A class using an SQL database to store itsTagSets.ClassSQLTagsCommand(BaseSQLTagsCommand, cs.cmdutils.BaseCommand, SQLTagsCommandsMixin, cs.tagset.TagsCommandMixin)sqltagsmain command line utility.Command line usage:Usage: sqltags [-f db_url] subcommand [...]
  -f db_url SQLAlchemy database URL or filename.
            Default from $SQLTAGS_DBURL (default '~/var/sqltags.sqlite').
  Subcommands:
    dbshell
      Start an interactive database shell.
    edit criteria...
      Edit the entities specified by criteria.
    export [-F format] [{tag[=value]|-tag}...]
      Export entities matching all the constraints.
      -F format Specify the export format, either CSV or FSTAGS.
    find [-o output_format] {tag[=value]|-tag}...
      List entities matching all the constraints.
      -o output_format
                  Use output_format as a Python format string to lay out
                  the listing.
                  Default: {localtime} {headline}
    help [-l] [subcommand-names...]
      Print the full help for the named subcommands,
      or for all subcommands if no names are specified.
      -l  Long help even if no subcommand-names provided.
    import [{-u|--update}] {-|srcpath}...
      Import CSV data in the format emitted by "export".
      Each argument is a file path or "-", indicating standard input.
      -u, --update  If a named entity already exists then update its tags.
                    Otherwise this will be seen as a conflict
                    and the import aborted.
    init
      Initialise the database.
      This includes defining the schema and making the root metanode.
    list [entity-names...]
      List entities and their tags.
    log [-c category,...] [-d when] [-D strptime] {-|headline} [tags...]
      Record entries into the database.
      If headline is '-', read headlines from standard input.
      -c categories
        Specify the categories for this log entry.
        The default is to recognise a leading CAT,CAT,...: prefix.
      -d when
        Use when, an ISO8601 date, as the log entry timestamp.
      -D strptime
        Read the time from the start of the headline
        according to the provided strptime specification.
    ls [entity-names...]
      List entities and their tags.
    orm define_schema
      Runs the ORM's `define_schema()` method, which creates missing tables
      and entity 0 if missing.
    shell
      Run a command prompt via cmd.Cmd using this command's subcommands.
    tag {-|entity-name} {tag[=value]|-tag}...
      Tag an entity with multiple tags.
      With the form "-tag", remove that tag from the direct tags.
      A entity-name named "-" indicates that entity-names should
      be read from the standard input.ClassSQLTagSet(cs.obj.SingletonMixin, cs.tagset.TagSet, builtins.dict, cs.dateutils.UNIXTimeMixin, cs.lex.FormatableMixin, cs.lex.FormatableFormatter, string.Formatter, cs.mappings.AttrableMappingMixin)A singletonTagSetattached to anSQLTagsinstance.As with theTagSetsuperclass,
tag values can be any Python type.
However, because we are storing these values in an SQL database
it is necessary to provide a conversion facility
to prepare those values for storage.The database schema is described in theSQLTagsORMclass;
in short we directly supportNone,floatandstr,ints which round trip withfloat,
andlist,tupleanddictwhose contents transcribe to JSON.ints which are too large to round trip withfloatare treated as an extended"bigint"type
using the scheme described below.Because the ORM has distinctfloatandstrcolumns to support indexing,
there will be no plain strings in the remaining JSON blob column.
Therefore we support other types by providing functions
to convert each type to astrand back,
and an associated "type label" which will be prefixed to the string;
the resulting string is stored in the JSON blob.The default mechanism is based on the following class attributes and methods:TYPE_JS_MAPPING: a mapping of a type label string
to a 3 tuple of(type,to_str,from_str)being the extended type,
a function to convert an instance tostrand a function to convert astrto an instance of this typeto_js_str: a method accepting(tag_name,tag_value)and returningtag_valueas astr;
the default implementation looks up the type oftag_valueinTYPE_JS_MAPPINGto locate the correspondingto_strfunctionfrom_js_str: a method accepting(tag_name,js)which uses the leading type label prefix from thejsto look up the correspondingfrom_strfunction
fromTYPE_JS_MAPPINGand use it on the tail ofjsThe defaultTYPE_JS_MAPPINGhas mappings for:"bigint": conversions forint"date": conversions fordatetime.date"datetime": conversions fordatetime.datetimeSubclasses wanting to augument theTYPE_JS_MAPPINGshould prepare their own with code such as:class SubSQLTagSet(SQLTagSet,....):
    ....
    TYPE_JS_MAPPING=dict(SQLTagSet.TYPE_JS_MAPPING)
    TYPE_JS_MAPPING.update(
      typelabel=(type, to_str, from_str),
      ....
    )ClassSQLTagsORM(cs.sqlalchemy_utils.ORM, cs.resources.MultiOpenMixin, cs.context.ContextManagerMixin, cs.dateutils.UNIXTimeMixin)The ORM for anSQLTags.The current implementation uses 3 tables:entities: this has a NULLablenameandunixtimeUNIX timestamp;
this is unique pernameif the name is not NULLtags: this has anentity_id,nameand a value stored
in one of three columns:float_value,string_valueandstructured_valuewhich is a JSON blob;
this is unique per(entity_id,name)tag_subvalues: this is a broken out version oftagswhenstructured_valueis a sequence or mapping,
breaking out the values one per row;
this exists to support "tag contains value" lookupsTag values are stored as follows:None: all 3 columns are set toNULLfloat: stored infloat_valueint: if theintround trips tofloatthen it is stored infloat_value,
otherwise it is stored instructured_valuewith the type label"bigint"str: stored instring_valuelist,tuple,dict: stored instructured_value;
if these containers contain unJSONable content there will be troubleother types, such asdatetime:
these are converted to strings with identifying type label prefixes
and stored instructured_valueThefloat_valueandstring_valuecolumns
allow us to provide indices for these kinds of tag values.The type label scheme takes advantage of the fact that actualstrs
are stored in thestring_valuecolumn.
Because of this, there will be no actual strings instructured_value.
Therefore, we can convert nonJSONable types tostrand store them here.The scheme used is to provide conversion functions to convert types
tostrand back, and an associated "type label" prefix.
For example, we store adatetimeas the ISO format of thedatetimewith"datetime:"prefixed to it.The actual conversions are kept with theSQLTagSetclass
(or any subclass).
This ORM receives the 3-tuples of SQL ready values
from that class as thePolyValuenamedtupleand does not perform any conversion itself.
The conversion process is described inSQLTagSet.ClassSQTCriterion(cs.tagset.TagSetCriterion, cs.deco.Promotable)Subclass ofTagSetCriterionrequiring an.sql_parametersmethod
which returns anSQLParametersproviding the information required
to construct an sqlalchemy query.
It also resets.CRITERION_PARSE_CLASSES, which will pick up
the SQL capable criterion classes below.ClassSQTEntityIdTest(SQTCriterion, cs.tagset.TagSetCriterion, cs.deco.Promotable)A test onentity.id.Functionverbose(msg, *a)Emit message if in verbose mode.Release LogRelease 20240201.1:
Release with the "sqltags" script.Release 20240201:SQLTagSet.to_polyvalue: treat sets like lists.SQLTags.default_factory: honour new skip_refresh parameter, apply any presupplied tags.Pull the cmd_* methods from BaseSQLTagsCommand into new SQLTagsCommandsMixin for reuse.Release 20230612:SQLTagBasedTest.sql_parameters: fix general tag name.SQLTagSet: new jsonable class method to produce a JSON serialisable object - converts sets and Sequences to flat lists, etc.Release 20230217:
SQLTagsORM.search: previous changes seem to have dropped SQTCriterion support.Release 20230212.1:
Mark SQLTags as promotable.Release 20230212:@promote support for SQLTags, promoting a filesystem path to a .sqlite db.Simpler SQLTagsORM.search comparison implementation.SQLTagSet: inherit format attributes from superclasses (TagSet).New BaseSQLTagsCommand.cmd_shell method.New BaseSQLTagsCommand.cmd_orm method with "define_schema" subcommand to update the db schema.SQLTagsORM.init: drop case_sensitive, no longer supported?SQLTagsORM.init: always call define_schema, it seems there are scenarios where this does some necessary sqlalchemy prep.Release 20221228:
SQLTagsCommand: update implementation of BaseCommand.run_context to use super().run_context().Release 20220806:Bugfix for SQLTagsORM.search(mode='entity').SQLTags.find: new _without_tags=False parameter to allow fast searches omitting the entity tags.Release 20220606:New SQLTagsORM.Entities.add_new_tags method, use it in SQLTags.default_factory for bulk insert.SQTCriterion: new .from_equality(tag_name,tag_value) factory to make an equality criterion.SQLTags.find: accept criteria as positional parameters instead of a single iterable, accept new keyword parameters as equality criteria.SQLTags.getitem: accept a slice to index the .unixtime tag.SQLTagsORM: also turn on echo mode if "ECHO" in $SQLTAGS_MODES.Release 20220311:
Assorted updates.Release 20211212:Rename edit_many to edit_tagsets for clarity.Small bugfixes.Release 20210913:SQLTagsCommand: rename cmd_ns to cmd_list,cmd_ls.SQLTagsCommand.cmd_export: accept "-F export_format" for csv or fstags export, accept no criteria to mean all tagsets.Encoding schema for nonJSONable types.Rename the TagSets abstract base class to BaseTagSets.BaseSQLTagsCommand.cmd_edit: implement rename.Many other internal small changes.Release 20210420:New PolyValueMixin pulled out of Tags for common support of the (float_value,string_value,structured_value).SQLTagsORM: new TagSubValues relation containing broken out values for values which are sequences, to support efficient lookup if sequence values such as log entry categories.New BaseSQLTagsCommand.parse_categories static method to parse FOO,BAH into ['foo','bah'].sqltags find: change default format to "{datetime} {headline}".Assorted small changes.Release 20210404:SQLTags.getitem: when autocreating an entity, do it in a new session so that the entity is commited to the database before any further use.SQLTagsCommand: new cmd_dbshell to drop you into the database.Release 20210321:
Drop logic now merged with cs.sqlalchemy_utils, use the new default session stuff.Release 20210306.1:
Docstring updates.Release 20210306:
Initial release.