Package: dy-sql
This project consists of a set of python decorators that eases integration with SQL databases.
These decorators may trigger queries, inserts, updates, and deletes.The decorators are a way to help us map our data in python to SQL queries and vice versa.
When we select, insert, update, or delete the queries, we pass the data we want
to insert along with a well defined query.This is designed to be done with minimal setup and coding. You need to specify
the database connection parameters and annotate any SQL queries/updates you have with the
decorator that fits your needs.Installationpip install dy-sqlComponent Breakdownset_default_connection_parameters- this function needs to be used to set the database parameters on
initialization so that when a decorator function is called, it can setup a connection pool to a correct databaseis_set_current_database_supported- this function may be used to determine if the*_current_databasemethods
may be used or notset_current_database- (only supported on Python 3.7+) this function may be used to set the database name for the
current async context (not thread), this is especially useful for multitenant applicationsreset_current_database- (only supported on Python 3.7+) helper method to reset the current database afterset_current_databasehas been used in an async contextset_database_init_hook- sets a method to call whenever a new database is initializedQueryData- a class that may be returned or yielded fromsql*decorated methods which
contains query informationDbMapResult- base class that can be used when selecting data that helps to map the results of a
query to an object in pythonDbMapResultModel- pydantic version ofDbMapResultthat allows easy mapping to pydantic models@sqlquery- decorator for select queries that can return a SQL result in aDbMapResult@sqlupdate- decorator for any queries that can change data in the database, this can take a set of
values and yield multiple operations back for insertions or updates inside of a transaction@sqlexists- decorator for a simplified select query that will return true if a record exists and false otherwiseXDbTestManager- test manager classes that may be used for testing purposesDatabase PreparationIn order to initialize a connection pool for thesql*decorators, the database needs to first be set up
using theset_default_connection_parametersmethod.fromdysqlimportset_database_parametersdefset_database_from_config():maria_db_config={...}set_database_parameters(maria_db_host,maria_db_user,maria_db_password,maria_db_database,port=maria_db_port,charset=maria_db_charset)Note: the keyword arguments are not required and have standard default values,
the port for example defaults to 3306Database Init HookAt times, it is necessary to perform post-initialization tasks on the database engine after it has been created.
Theset_database_init_hookmethod may be used in this case. As an example, to instrument the engine usingopentelemetry-instrumentation-sqlalchemy, the following code may be used:fromtypingimportOptional# Used for type-hints onlyimportsqlalchemyfromopentelemetry.instrumentation.sqlalchemyimportSQLAlchemyInstrumentorfromdysqlimportset_database_init_hookdef_instrument_engine(database_name:Optional[str],engine:sqlalchemy.engine.Engine)->None:# The database name is unused in this case_=database_nameSQLAlchemyInstrumentor().instrument(engine=engine)set_database_init_hook(_instrument_engine)MultitenancyIn some applications, it may be useful to set a database other than the default database in order to support
database-per-tenant configurations. This may be done using theset_current_databaseandreset_current_databasemethods.fromdysqlimportreset_current_database,set_current_databasedefuse_database_for_query():set_database_parameters(...'db1',)set_current_database('db2')try:# Queries db2 and not db1query_database()finally:reset_current_database()WarningThese methods are only supported in Python 3.7+ due to their use of thecontextvarsmodule. Theis_set_current_database_supportedmethod is provided to help tell if these methods may be used.DecoratorsDecorators are an easy way for us to tell a function to be a ‘query’ and return
a result without having to have a big chunk of boiler plate code. Once the
database has been prepared, calling asql*decorated function will initialize
the database, parse the value returned in your function, make a corresponding
parameterized query and return the results.The basic structure is to decorate a method that returns information about the query.
There are multiple options for returning a query, below is a summary of some of the possibilities:return aQueryDataobject that possibly containsquery_paramsand/ortemplate_params(not available for allsql*decorators) yield one or moreQueryDataobjects,
each containingquery_paramsand/ortemplate_paramsDbMapResultThis class is used in the default mapper (see below) for anysqlquerydecorated method. This class may also be
overridden as shown below. The default class wraps and returns the results of a query for easy access to the data
from the query. For example, if you use the querySELECT id, name FROM table, it would return a list ofDbMapResultobjects where each contains theidandnamefields. You could then easily loop through
and access the properties as shown in the following example:@sqlquery()defget_items_from_sql_query():returnQueryData("SELECT id, name FROM table")defget_and_process_items():foriteminget_items_from_sql_query():# we are able to access properties on the objectprint('{name}goes with{id}'.format(item.name,item.id))We can inherit fromDbMapResultand override the way our data maps into the
object. This is primarily helpful in cases where we end up with multiple rows
such as a query for a 1-to-many relationship.classExampleMap(DbMapResult):defmap_result(self,result):# we know we are mapping multiple rows to a single resultifself.idisNone:# in our case we know the id is the same so we only set it the first timeself.id=result['id']# initialize our arrayself.item_names=[]# we know that every result for a given id has a unique item_nameself.item_names.append(result['item_name'])@sqlquery(mapping=ExampleMap)defget_table_items()returnQueryData("""
        SELECT id, name, item_name FROM table
            JOIN table_item ON table.id = table_item.table_id
            JOIN item ON item.id = table_item.item_id
    """)defprint_item_names()fortable_iteminget_table_items():foritem_nameintable_item.item_names:print(f'table name{table_item.name}has item{item_name}')DbMapResultModel (pydantic)If pydantic models are desired to be used, there is a record mapper available. Note that pydantic must be installed,
which is available as an extra package:pip install dy-sql[pydantic]This model attempts to make mapping records easier, but there are shortcomings of it in more complex cases.
Most fields will “just work” as defined by the type annotations.fromdysql.pydantic_mappersimportDbMapResultModelclassPydanticDbModel(DbMapResultModel):id:intfield_str:strfield_int:intfield_bool:boolMapping a record onto this class will automatically convert types as defined by the type annotations. Nomap_recordmethod needs to be defined since the pydantic model has everything necessary to map database fields.Lists, sets, dicts, csv strings, and json strings (when using the RecordCombiningMapper) require additional configuration on the model class.fromdysql.pydantic_mappersimportDbMapResultModelclassComplexDbModel(DbMapResultModel):# if any data has been aggregated or saved into a string as a comma delimited list, this will convert to a list# NOTE this only does simple splitting and is not fully rfc4180 compatible_csv_list_fields:Set[str]={'list_from_string'}# List fields (type does not matter)_list_fields:Set[str]={'list1'}# Set fields (type does not matter)_set_fields:Set[str]={'set1'}# Dictionary key fields as DB field name => model field name_dict_key_fields:Dict[str,str]={'key1':'dict1','key2':'dict2'}# Dictionary value fields as model field name => DB field name (this is reversed from _dict_key_fields!)_dict_value_mappings:Dict[str,str]={'dict1':'val1','dict2':'val2'}# JSON string fields. Type can be any dictionary type but for larger json objects its safe to stay with `dict`_json_fields:Set[str]={'json1','json2'}id:int=Nonelist_from_string:List[str]list1:List[str]set1:Set[str]=set()dict1:Dict[str,Any]={}dict2:Dict[str,int]={}json1:dictjson2:dictNotecsv strings can be useful in queries where you want to group by an id and thengroup_concatsome fieldjson strings are a handy way to extract json blobs into a python dictionary for ease of use without manually processing
each field everytime you need something.In this case, the_prefixed properties tell the model which fields should be treated differently when combining
multiple rows into a single object. For an example of how this works with database rows, see thetest_pydantic_mappers.pyfile in the source repository.Note that validationdoesoccur the very first timemap_recordis called, but not on subsequent runs. Therefore
if you desire better validation for list, set, or dict fields, this must most likely be done outside of dysql/pydantic.
Additionally, lists, sets, and dicts will ignore null values from the database. Therefore you must provide default
values for these fields when used or else validation will fail.Added annotations when using DbMapResultModelWhen using theDbMapResultModelmapper, there are some additional annotations that may be used to help with
mapping. These annotations are not required, but may be helpful in some cases.FromCSVToList - this annotation will convert a comma separated string into a list. This is useful when you have
a column containing a csv or a query that usesgroup_concatto combine multiple rows into a single row. This
annotation may be used on any field that is a list. For example:fromdysql.pydantic_mappersimportDbMapResultModel,FromCSVToListclassCsvModel(DbMapResultModel):id:intname:str# This annotation will convert the string into a list of intslist_from_string_int:FromCSVToList[List[int]]# This annotation will convert the string into a list of stringslist_from_string:FromCSVToList[List[str]]# This annotation will convert the string into a list of ints or None if the string is null or emptylist_from_string_int_nullable:FromCSVToList[List[int]|None]# This annotation will convert the string into a list of strings or None if the string is null or emptylist_from_string_nullable:FromCSVToList[List[str]|None]# if using python <= 3.9, you can use typing.Union instead of the pipe operator# list_from_string_nullable: FromCSVToList[Union[List[str],None]]@sqlqueryThis is for making SQLselectcalls. An optional mapper may be specified to
change the behavior of what is returned from a decorated method. The default
mapper can combine multiple records into a single result if there is anidfield present in each record. Mappers available:RecordCombiningMapper(default) - Returns a list of results where multiple records that can be combined with the
same unique identifer. An optionalrecord_mappervalue may be passed to the constructor to change
how records are mapped to result. By default therecord_mapperused isDbMapResult. The base identifier
is the columnidbut an array of columns can be used to create a unique key lookup for combining records.NoteThe_key_columnsfield of theDbMapResultModelis an array containing only theidbut can
be overriden in derived classes. For example, setting_key_columns = [ 'a', 'b' ]in your derived class
would make it so you class would use the values of columnsaandbin order to uniquely identify
records when being combined.SingleRowMapper- returns an object for the first record from the database (even if multiple records are
returned). An optionalrecord_mappervalue may be passed to the constructor to change how this first record is
mapped to the result.SingleColumnMapper- Returns a list of scalars with the first column from every record, even if multiple columns
are returned from the database.SingleRowAndColumnMapper- Returns a single scalar value even if multiple records and columns are returned
from the database.CountMapper- alias forSingleRowAndColumnMapperto make it clear that it may be used forcountqueries.KeyValueMapper- returns a dictionary mapping 1 column to the keys and 1 column to the values.
By default the key is mapped to the first column and value is mapped to the second column. You can override the key_column
and value_columns by specifying the name of the columns you want for each. You can also pass in a has_multiple_values
which defaults to False. Doing so will allow you to get a dictionary of lists based on the keys and values you specify.Custom mappers may be made by extending theBaseMapperclass and implementing themap_recordsmethod.basic query with conditions hardcoded into query and default mapperdefget_items():items=select_items_for_joe()# ... work on items@sqlquery()defselect_items_for_joe()returnQueryData("SELECT * FROM table WHERE name='joe'")basic query with params passed as a dictdefget_items():items=select_items_for_name('joe')# ... work on items, which contains all records matching the name@sqlquery()defselect_items_for_name(name)returnQueryData("SELECT * FROM table WHERE name=:name",query_params={'name':name})query that only returns a single result from the first rowdefget_joe_id():result=get_item_for_name('joe')returnresult.get('id')# Either an instance or class may be used as the mapper parameter@sqlquery(mapper=SingleRowMapper())defget_item_for_name(name)returnQueryData("SELECT id, name FROM table WHERE name=:name",query_params={'name':name})alternative to the above query that returns the id directlydefget_joe_id():returnget_id_for_name('joe')@sqlquery(mapper=SingleRowAndColumnMapper)defget_id_for_name(name)returnQueryData("SELECT id FROM table WHERE name=:name",query_params={'name':name})query that returns a list of scalar values containing the list of distinct names availabledefget_unique_names():returnget_names_from_items()@sqlquery(mapper=SingleColumnMapper)defget_names_from_items()returnQueryData("SELECT DISTINCT(name) FROM table")basic count query that only returns the scalar value returned for the countdefget_count_for_joe():returnget_count_for_name('joe')@sqlquery(mapper=CountMapper)defget_count_for_name(name):returnQueryData("SELECT COUNT(*) FROM table WHERE name=:name",query_params={'name':name})basic query returning dictionary@sqlquery(mapper=KeyValueMapper())defget_status_by_name():returnQueryData("SELECT name, status FROM table")query returning a dictionary where we are specifying the keys. Note that the columns are returning in a different order@sqlquery(mapper=KeyValueMapper(key_column='name',value_column='status'))defget_status_by_name():returnQueryData("SELECT status, name FROM table")query returning a dictionary where there are multiple results under each key. Note that here we are essentially grouping under status@sqlquery(mapper=KeyValueMapper(key_column='status',value_column='name',has_multiple_values=True))defget_status_by_name():returnQueryData("SELECT status, name FROM table")@sqlupdateHandles any SQL that is not a select. This is primarily, but not limited to,insert,update, anddelete.@sqlupdate()definsert_items(item_dict):returnQueryData("INSERT INTO",template_params={'in__item_id':item_id_list})You can yield multiple QueryData objects. This is done in a transaction and it can be helpful for data integrity or just
a nice clean way to run a set of updates.@sqlupdate()definsert_items(item_dict):insert_values_1,insert_params_1=TemplateGenerator.values('table1values',_get_values_for_1_from_items(item_dict))insert_values_2,insert_params_2=TemplateGenerator.values('table2values',_get_values_for_2_from_items(item_dict))yieldQueryData(f'INSERT INTO table_1{insert_values_1}',query_params=insert_values_params_1)yieldQueryData(f'INSERT INTO table_2{insert_values_2}',query_params=insert_values_params_2)if needed you can assign a callback to be ran after a query or set of queries completes successfully@sqlupdate(on_success=_handle_insert_success)definsert_items_with_callback(item_dict):insert_values_1,insert_params_1=TemplateGenerator.values('table1values',_get_values_for_1_from_items(item_dict))insert_values_2,insert_params_2=TemplateGenerator.values('table2values',_get_values_for_2_from_items(item_dict))yieldQueryData(f'INSERT INTO table_1{insert_values_1}',query_params=insert_values_params_1)yieldQueryData(f'INSERT INTO table_2{insert_values_2}',query_params=insert_values_params_2)def_handle_insert_success(item_dict):#  callback logic here happens after the transaction is complete@sqlexistsThis wraps a SQL query to determine if a row exists or not. If at least one row is returned from the query, it will
return True, otherwise False. The query you give here can return anything you want but as good practice,
try to always select as little as possible. For example, below we are just returning 1 because the value itself
isn’t used, we just need to know there are records available.@sqlexists()defitem_exists(item_id)returnQueryData("SELECT 1 FROM table WHERE id=:id",query_params={'id':item_id})Ultimately, the above query becomesSELECT EXISTS (SELECT 1 FROM table WHEREid=:id).
You’ll notice the inner select value isn’t actually used in the return.Decorator templatesTemplates and generators for these templates are also provided to simplify SQL query strings.intemplate - this template will allow you to pass a list as a single parameter and have theINcondition build out for you. This allows you to more dynamically include values in your queries.@sqlquery()defselect_items(item_id_list):returnQueryData("SELECT * FROM table WHERE{in__item_id}",template_params={'in__item_id':item_id_list})you can also use the TemlpateGenerate.in_column method to get back a tuple of query and params@sqlquery()defselect_items(item_id_list):in_query,in_params=TemplateGenerators.in_column('key',item_id_list)# NOTE: the query string is using an f-string and passing into query_params instead of template_paramsreturnQueryData(f"SELECT * FROM table WHERE{in_query}",query_params=in_params)in and not in multi column- this template works the same as the in and not in template but it will allow you to
pass a list of tuples to an in clause allowing you to match against multiple columns.NOTE: this is only available through the TemplateGenerators using query_params and not through the the template_params method@sqlquery()defselect_multi(tuple_list):in_query,in_params=TemplateGenerators.in_multi_column('(key1, key2)',tuple_list)returnQueryData(f"SELECT * FROM table WHERE{in_query}",query_params=in_params)@sqlquery()defselect_multi(tuple_list):in_query,in_params=TemplateGenerators.not_in_multi_column('(key1, key2)',tuple_list)returnQueryData(f"SELECT * FROM table WHERE{in_query}",query_params=in_params)not_intemplate -  this template will allow you to pass a list as a single parameter and have theNOT INcondition build out for you. This allows you more dynamically exclude values in your queries.@sqlquery()defselect_items(item_id_list)returnQueryData("SELECT * FROM table WHERE{not_in__item_id}",template_params={'not_in__item_id':item_id_list})you can also use the TemplateGenerators.not_in_column method to get back a tuple of query and params@sqlquery()defselect_items(item_id_list):not_in_query,not_in_params=TemplateGenerators.not_in_column('key',item_id_list)# NOTE: the query string is using an f-string and passing into query_params instead of template_paramsreturnQueryData(f"SELECT * FROM table WHERE{not_in_query}",query_params=not_in_params)valuestemplate - when inserting and you have multiple records to insert, this allows you to pass
multiple records for insert in a single INSERT statement.@sqlquery()definsert_items(items):returnQueryData("INSERT_INTO table(column_a, column_b){values__items}",template_params={'values__items':item_id_list})You can write queries that combinetemplate_paramsandquery_paramsas well..@sqlquery()defselect_items(item_id_list,name):returnQueryData("SELECT * FROM table WHERE{in__item_id}and name=:name",template_params={'in__item_id':item_id_list},query_params={'name':name})Testing with ManagersDuring testing, it may be useful to hook up a real database to the tests. However, this can be difficult to maintain
schema and isolate databases during testing. Database test managers exist for this reason. Usage is very simple with
pytest.@pytest.fixture(scope='module',autouse=True)defsetup_db(self):# Pass in the database name and any optional paramswithMariaDbTestManager(f'testdb_{self.__class__.__name__.lower()}'):yieldThe Maria database test manager is shown used above, but future implementations may be added for other SQL backends.