Package: pymssql-utils
pymssql-utilspymssql-utilsis a small library that wrapspymssqlto make your life easier.
It provides you with a higher-level API as well as various utility functions
for generating dynamic queries.This module features:Higher-level API that reduces the amount of boilerplate required.Baked-in sensible defaults and usage patterns.Provides optional execution batching, similar topyodbc'sfast_executemany.Provides consistent parsing between SQL Types and native Python types over different platforms and drivers.Makes it easy to serialize your data withorjson.Provides you with simple and clear options for error handling.Extra utility functions for building dynamic SQL queries.Fixing various edge case bugs that arise when usingpymssql.Fully type hinted.This module's enforced opinions (check these work for you):Each execution opens and closes a connection usingpymssql's
context management.Automatically converts certain data types for ease of use, e.g.Decimal->float,UUID->str.When you shouldn't use this module:If you need fine-grained control over your cursors.Please raise any suggestions or issues via GitHub.StatusThis library is tested and stable. There will not be any breaking changes to the
public API without a major version release.  There is also scope for expanding the
library if new features are requested.ChangesSee the repository'sGitHub releasesor theCHANGELOG.md.UsageInstallationThis library can be installed via pip:pip install --upgrade pymssql-utils.
This library requiresPython >= 3.7.If you want to serialize your results to JSON you can install the optional dependencyORJSONby runningpip install --upgrade pymssql-utils[json].If you want to cast your results to DataFrame you can install the optional dependencyPandasby runningpip install --upgrade pymssql-utils[pandas].QuickstartFor querying the database this library provides two high-level methods:query: executes a SQL operation that fetches the result, and DOES NOT commit the transaction (by default).execute: executes a SQL operation that does not fetch the result (by default), and DOES commit the transaction.This separation ofpymssql'sexecuteis to make your code more explicit and readable.Here is an example for running a simple query and accessing the returned data:>>>importpymssqlutilsassql>>>result=sql.query("SELECT SYSDATETIMEOFFSET() as now",server="...")>>>result.data[{'now':datetime.datetime(2021,1,21,23,31,11,272299,tzinfo=datetime.timezone.utc)}]And running a simple execution:>>>importpymssqlutilsassql>>>result=sql.execute("INSERT INTO mytable VALUES (1, 'test')",server="MySQLServer")Specifying ConnectionThere are two ways of specifying the connection parameters to the SQL Server:Passing the required parameters
(see pymssql docs)
toqueryorexecutelike in the quickstart example above.
All extra**kwargspassed to these methods are passed on to thepymssql.connection().Specify the connection parameters in the environment like the example below, this is the recommended way
(however, any parameters given explicitly to a method will take priority).importosimportpymssqlutilsassqlos.environ["MSSQL_SERVER"]="sqlserver.mycompany.com"os.environ["MSSQL_USER"]="my_login"os.environ["MSSQL_PASSWORD"]="my_password123"result=sql.execute("INSERT INTO mytable VALUES (%s,%s)",(1,"test"))There is a helper method to set this in code, seeset_connection_detailsbelow.Executing SQLQueryThequerymethod executes a SQL Operation which does not commit the transaction & returns the result.query(operation:str,parameters:SQLParameters=None,raise_errors:bool=True,**kwargs,)->DatabaseResult:Parameters:operation (str): the SQL operation to execute.parameters (SQLParameters): parameters to substitute into the operation,
these can be a single value, tuple or dictionary.raise_errors (bool): whether to raise exceptions or to return the error information with the result.Any extra kwargs are passed topymssql'sconnectmethod.Returns aDatabaseResultclass, see documentation below.ExecuteTheexecutemethod executes a SQL Operation which commits the transaction
& optionally returns the result (not by default).execute(operations:Union[str,List[str]],parameters:Union[SQLParameters,List[SQLParameters]]=None,batch_size:int=None,fetch:bool=False,raise_errors:bool=True,**kwargs,)->DatabaseResult:Parameters:operations (Union[str, List[str]]): the SQL Operation/s to execute.parameters (Union[SQLParameters, List[SQLParameters]]): parameters to substitute into the operation/s,
these can be a single value, tuple or dictionary OR this can be a list of any of the previous.batch_size (int): if specified concatenates the operations together according to the batch_size,
this can vastly increase performance if executing many statements.
Raises an error if set to True and both operations and parameters are singular.fetch (bool): if True returns the result from the LAST execution, by default false.raise_errors (bool): whether to raise exceptions or to return the error information with the result.Any extra kwargs are passed topymssql'sconnectmethod.Returns aDatabaseResultclass, see documentation below.There are two ways of using this function:Passing in a single operation (str) to operations:If parameters is singular, this callspymssql.execute()and executes a single operationIf parameters is plural, this callspymssql.execute_many()and executes one execution per parameter setPassing in multiple operations (List[str]) to operations:If parameters is None, this callspymssql.execute_many()and executes one execution per operationIf parameters is the same length as operations, this callspymssql.execute()multiple times
and executes one execution per operation.Optionallybatch_sizecan be specified to use string concatenation to batch the operations, this can
provide significant performance gains if executing 100+ small operations. This is similar tofast_executemanyfound in thepyodbcpackage. A value of 500-1000 is a good default.DatabaseResult ClassOne big difference between this library andpymssqlis that hereexecuteandqueryreturn an instance of theDatabaseResultclass.This class holds the returned data, if there is any, and provides
various useful attributes and methods to work with the result.Attributes & Propertiesok: True if the execution did not error, else False. Only useful if usingraise_errors = False,
see below section on Error Handling.error: Populated by the error raised during execution (if applicable). Only useful if usingraise_errors = False.fetch: True if results from the execution were fetched (e.g. if usingquery), else False.commit: True if the execution was committed (i.e. if usingexecute), else False.columns: A list of the column names in the dataset returned from the execution (if applicable)data: The dataset returned from the execution (if applicable), this is a list of dictionaries.raw_data: The dataset returned from the execution (if applicable), this is a list of tuples.set_count: Returns the count of result sets that the execution returned, as an integer.Methodsto_dataframe: (requires Pandas to be installed), returns the dataset as a DataFrame object.
All args and kwargs are parsed to the DataFrame constructor.to_json: returns the dataset as a json serialized string using theorjsonlibrary, make sure this
optional dependency is installed by runningpip install --upgrade pymssql-utils[json].
Note that this will fail if your data containsbytestype values. By default, this method returns a string, but
passas_bytes = Trueto return a byte string. Specifywith_columns = Trueto include the column names
(rows as dictionaries instead of tuples).write_error_to_logger: writes the error information to the library's logger, optionally pass anameparameter
to allow you to easier indentify the query in the logging output.raise_error: raises apymssqlutils.DatabaseErrorfrom the underlyingpymssqlerror,
optionally pass anameparameter to allow you to easier indentify the query in the error output.next_set: changes the class to return the data and metadata (columns etc) of the next result set. Returns True
if there was a next set to move to, otherwise returns False and doesn't do anything.previous_set: changes the class to return the data and metadata (columns etc) of the previous result set. Returns True
if there was a previous set to move to, otherwise returns False and doesn't do anything.Error handlingBothquery&executetakeraise_errorsas a parameter, which is by defaultTrue. This means that by defaultpymssql-utilswill allowpymssqlto raise errors as normal.Passingraise_errorsasFalsewill pass any errors onto theDatabaseResultclass, which allows you
to handle errors gracefully using theDatabaseResultclass (see above), e.g.:importpymssqlutilsasdbresult=db.query("Bad Operation",raise_errors=False)ifnotresult.ok:# result.ok will be False due to error# write the error to logging outputresult.write_error_to_logger('An optional query identifier to aid logging')# the error is stored under the error attributeerror=result.error# can always re-raise the errorresult.raise_error('Query Identifier')This can be useful in situations where you do not want the error to propogate, e.g. if querying the database
as part of an API response.Utility Functionsset_connection_detailsTheset_connection_detailsmethod is a helper function which will set the value of
the relevant environment variable for the connection kwargs given.Warning: this function has program wide side effects and will overwrite any
previously set connection details in the environment; therefore its usage is only recommended
in single script projects/notebooks. All the connections details will also be visible in your code.The preferred method in production scenarios is to set the environment variables directly.defset_connection_details(server:str=None,database:str=None,user:str=None,password:str=None)->None:Parameters:server (str): the network address of the SQL server to connect to, sets 'MSSQL_SERVER' in the environment.database (str): the default database to use on the SQL server, sets 'MSSQL_DATABASE' in the environment.user (str): the user to authenticate against the SQL server with, sets 'MSSQL_USER' in the environmentpassword (str): the password to authenticate against the SQL server with, sets 'MSSQL_PASSWORD' in the environmentsubstitute_parametersThesubstitute_parametersmethod does the same parameter substitution asqueryandexecute, but returns the
substituted operation instead of executing it. This allows you to see the actual operation being run
against the database and is useful for debugging and logging.substitute_parameters(operation:str,parameters:SQLParameters)->str:Parameters:operation (str): The SQL operation requiring substitution.parameters (SQLParameters): The parameters to substitute in.Returns the parameter substituted SQL operation as a string.Example:>>>substitute_parameters("SELECT%sCol1,%sCol2",("Hello",1.23))"SELECT N'Hello' Col1, 1.23 Col2"to_sql_listTheto_sql_listmethod converts a Python iterable to a string form of the SQL equivalent list. This is useful
when creating dynamic SQL operations using the 'IN' operator.to_sql_list(listlike:Iterable[SQLParameter])->str:Parameters:listlike (Iterable[SQLParameter]): The iterable of SQLParameter to transformReturns the SQL equivalent list as a stringExamples:>>>to_sql_list([1,'hello',datetime.now()])"(1, N'hello', N'2021-03-22T10:56:27.981173')">>>my_ids=[1,10,21]>>>f"SELECT * FROM MyTable WHERE Id IN{to_sql_list(my_ids)}"'SELECT * FROM MyTable WHERE Id IN (1, 10, 21)'model_to_valuesThemodel_to_valuesmethod converts a Python mapping (e.g. dictionary of Pydantic model) to the SQL equivalent
values string. This is useful when creating dynamic SQL operations using the 'INSERT' statement.model_to_values(model:Any,prepend:List[Tuple[str,str]]=None,append:List[Tuple[str,str]]=None,)->str:Parameters:model (Any): A mapping to transform, i.e. a dictionary or an object that has thedictmethod implemented,
with string keys and SQLParameter values.prepend (List[Tuple[str, str]]): prepend a variable number of columns to the beginning of the values statement.append (List[Tuple[str, str]]): append a variable number of columns to the end of the values statement.Returns a string of the form:([attr1], [attr2], ...) VALUES (val1, val2, ...).Warning: prepended and appended columns are not parameter substituted,
this can leave your code open to SQL injection attacks.Example:>>>my_data={'value':1.56,'insertDate':datetime.now()}>>>model_to_values(my_data,prepend=[('ForeignId','@Id')])"([ForeignId], [value], [insertDate]) VALUES (@Id, 1.56, N'2021-03-22T13:58:33.758740')">>>f"INSERT IN MyTable{model_to_values(my_data,prepend=[('foreignId','@Id')])}""INSERT IN MyTable ([foreignId], [value], [insertDate]) VALUES (@Id, 1.56, N'2021-03-22T13:58:33.758740')"NotesType Parsingpymssql-utilsparses SQL types to their native python types regardless of the environment.
This ensures consistent behaviour across various systems, see the table below for a comparison.WindowsUbuntuSQL DataTypepymssql-utilspymssqlpymssql-utilspymssqlDatedatedatedatestrBinarybytesbytesbytesbytesTime1timetimetimestrTime2timetimetimestrTime3timetimetimestrTime4timetimetimestrTime5timetimetimestrTime6timetimetimestrTime7timetimetimestrSmall DateTimedatetimedatetimedatetimedatetimeDatetimedatetimedatetimedatetimedatetimeDatetime2datetimedatetimedatetimestrDatetimeOffset0datetimebytesdatetimestrDatetimeOffset1datetimebytesdatetimestrDatetimeOffset2datetimebytesdatetimestrUniqueIdentifierstrbytesstr???TestingInstall pytest to run non-integration tests viapytest .,
these tests mock the cursor results allowing the library to test locally.To test against an MSSQL instance installpytest-dotenv.
Then create a.envfile with"TEST_ON_DATABASE"set as a truthy value, as well as any
connection environemt variables for the MSSQL server.
These tests will then be run (not-skipped), e.g.pytest . --envfile .test.envWhypymssqlwhen Microsoft officially recommendspyodbc(opinion)?There are other minor reasons someone might preferpymssql, e.g.:pymssqlsupports for MSSQL specific data types such asDatetimeoffset.pymssql'sparameter subsitution is done client-side improving debug/logging ability.pymssql'sdrivers are easily installed, meaning that your code 'just works' in
more environments without extra steps,e.g. this!