Package: ncl-sqlsnippets
NCL SQLSnippetsThis is a python module adding a standard set of functions to interact with the sandpitNCL SQLSnippetsHistory[1.0] - 10/07/2023[1.1] - 10/08/2023[1.2] - 06/12/2023[1.3] - 12/01/2024InstallationStandard UseFunctionsget_connection_stringconnect_to_sqlconnectupload_to_sqllist_all_tablestable_existsbackup_tablerestore_tablelist_all_columnscolumns_existexecute_queryexecute_sfwgenerate_sessionexecute_query_sessioncommit_changesLicenseHistory[1.0] - 10/07/2023Initial set of functionsInstructions for installing as a module[1.1] - 10/08/2023Added connect function[1.2] - 06/12/2023Converted to project (https://pypi.org/project/ncl-sqlsnippets/1.2.0/)[1.3] - 12/01/2024Added functionality for manual commits through new functionsgenerate_session, execute_query_session, commit_changes functions addedInstallationAs of v1.2.0 the sql snippets module can be install through pippip install ncl_sqlsnippetsStandard UseIn your other python and notebook files you can add the sqlsnippets module like any other standard module.importncl_sqlsnippetsassnipsLikewise functions from sqlsnippets can be called like functions from any standard module#Build connection stringconn_str=snips.get_connection_string(SQL_ADDRESS,SQL_DATABASE)#Connect to databaseengine=snips.connect_to_sql(conn_str=conn_str)#Upload datasnips.upload_to_sql(data,engine,SQL_TABLE,SQL_SCHEMA,replace=False)Functionsget_connection_stringget_connection_string(server_addres,database,server_type="mssql",driver="SQL+Server")Returns a connection string for the sandpit.Parameters:server_address:strThe server name of the database. It can be found in the Connect to Server pop-up when openning MS SQL Server Management Studio.database:strName of the database.server_type:strType of SQL server. Does not need to be set unless we ever migrate from MS SQL.driver:strDriver for the SQL server. Does not need to be set unless we ever migrate from MS SQL.Returns:strConnection string to establish the connection in the connect_to_sql function.connect_to_sqlconnect_to_sql(conn_str)Takes a connection string and returns a connection to the database in the form of an engine.Parameters:conn_str:strConnection string.Returns:engineConnection to the database. Store this in a variable and use it as a parameter in the other sqlsnippets functions.connectconnect (server_address, database)Wrapper for the main connection functions. Returns the engine connection object.Parameters:server_address:strThe server name of the database. It can be found in the Connect to Server pop-up when openning MS SQL Server Management Studio.database:strName of the database.upload_to_sqlupload_to_sql(data,engine,table,schema,replace,chunks=100,dtypes={})Takes data and uploads it to a target table in the database. If the table does not exist it will be created.Parameters:data:DataFrameData to upload in the form of a pandas data frame.engine:engineEngine object from the connect_to_sql function.table:strName of the table in the database.schema:strName of the schema in the database (this should either be "dbo" or your BLUE account prefix in most cases).replace:BooleanIf set to True, this function will replace any existing data in the table. If set to False it will append the data instead.Warning: This will drop the original table when set to Truechunks:intThis specifies the block size for data being uploaded. Generally bigger blocks means faster uploads. The function will fail if the block size exceeds the maximum limit for MS SQL (2100). As a rule of thumb, this figure should not exceed 2100/(number of columns).dtypes:dictOptional parameter to specify the types of the data columns, useful for specifying when a column should be intrepretted by the server as a date value. You can choose to only specify the type for a subset of the columns.dtypes={"date":types.Date}A list of dtype values recognised by the function are listed here:https://docs.sqlalchemy.org/en/20/core/type_basics.htmlReturns:Nonelist_all_tableslist_all_tables(engine)Lists all tables in a database. This will use the database specified in the connection string used.Parameters:engine:engineEngine object from the connect_to_sql function.Returns:DataFrameReturns a database with the following columns: SchemaName, TableName.table_existstable_exists(engine,table,schema)Checks if a given table exists in the database. The schema for the table needs to be provided as well.Parameters:engine:engineEngine object from the connect_to_sql function.table:strName of the table in the database.schema:strName of the schema in the database (this should either be "dbo" or your BLUE account prefix in most cases).Returns:BooleanReturns True if [schema].[table] exists in the database.backup_tablebackup_table(engine,table,schema)Creates a backup of the specified table ([database].[schema].[table]). The backup table will have the same name as the original but with "_backup" as a suffix.Parameters:engine:engineEngine object from the connect_to_sql function.table:strName of the table in the database.schema:strName of the schema in the database (this should either be "dbo" or your BLUE account prefix in most cases).Returns:None\restore_tablerestore_table(engine,table,schema)Replaces the table specified ([database].[schema].[table]) and replaces it with the backup table. Any table with the same name as the specified table but with "_backup" as a suffix to the table name will be considered to be the backup table.Parameters:engine:engineEngine object from the connect_to_sql function.table:strName of the table in the database.schema:strName of the schema in the database (this should either be "dbo" or your BLUE account prefix in most cases).Returns:None\list_all_columnslist_all_columns(engine,table,schema)Lists all columns for a given table. This will use the database specified in the connection string used.Parameters:engine:engineEngine object from the connect_to_sql function.table:strName of the table in the database.schema:strName of the schema in the database (this should either be "dbo" or your BLUE account prefix in most cases).Returns:SeriesReturns a list of columns in the table in the form of a pandas Series.columns_existcolumns_exist(engine,table,schema,columns)Checks if given columns exists in the specified table. The schema for the table needs to be provided as well.Parameters:engine:engineEngine object from the connect_to_sql function.table:strName of the table in the database.schema:strName of the schema in the database (this should either be "dbo" or your BLUE account prefix in most cases).Returns:ArrayReturns a list of all missing columns.Warning: If all columns exist in the table then this returns an empty list which can be logically treated as False.execute_queryexecute_query(engine,query)Executes a generic query. For queries where you expect rows to be returned use execute_sfw().Warning: This function commits changes to the database at the end of every call.Parameters:engine:engineEngine object from the connect_to_sql function.query:strThe query to execute as a stringReturns:None\execute_sfwexecute_sfw(engine,query)Executes a Select From Where query. The function is for queries where you expect a response to process, for other queries use execute_query().Parameters:engine:engineEngine object from the connect_to_sql function.query:strThe query to execute as a stringReturns:DataFrameThe query result is returned as a pandas data frame.generate_sessionsession, transaction = generate_session(engine)Generates a session object. Non-sfw queries can be executed with it without auto-committing the result.Parameters:engine:engineEngine object from the connect_to_sql function.Returns:Session,TransactionThe generated session along with a transaction object are returnedexecute_query_sessionexecute_query_session(query, session)Executes a non-sfw query. This will not commit changes.Parameters:query:strSQL query command as stringsession:sessionSession object from the generate_session function.commit_changescommit_changes(transaction)Commits changes made during a session.Parameters:transaction:transactionTransaction object from the generate_session function.LicenseThis repository is dual licensed under theOpen Government v3& MIT. All code can outputs are subject to Crown Copyright.