Package: ahjo
Ahjo FrameworkDescriptionAhjo is a database project framework and a deployment script. It is made to unify database project deployment and development practices and to give basic tooling for such projects.Ahjo provides a base scripts for database deployment with simple commands ("actions"), and the possibility to define custom actions for project's special needs. The scripts are designed to reduce accidental operations to production environments. The actions and their parts are logged by Ahjo.Database tooling is currently based on sqlalchemy/alembic and SQL scripts. Support for other backends than Microsoft SQL Server is currently limited.DependenciesCommonalembicpyparsingSQL AlchemyPyYAMLlarkPlatform-specificmssqlpyodbcazureazure-identityInstall GuideInstall Guide 1 - PyPIInstall Ahjo (without platform-specific dependencies) fromPython Package Indexwith the following command:pip install ahjoIn order to use Ahjo with the database engine of your choice, install platform-specific dependencies using available tags. For example, if you use Ahjo with Microsoft SQL Server, use tagmssqlto install required dependencies. See full list of available tags below.pip install ahjo[mssql]Install Guide 2 - Clone and installClone AhjoGet your clone command from the repository.Install with pipUse-eflag to install package in develop mode.cd .\ahjo
pip install [-e] .[mssql,azure]Available platform tagsmssql- Microsoft SQL Serverazure- Microsoft Azure SQL DatabaseInstall Guide 3 - MSI installation packageBuilding an MSI installation packageThis assumes you have cloned the source code repository and have it open in a shell.Create a new, empty build venv and install build requirements into it.Notice:at the time of writing this (10/2023),the latest stable cx_freeze version(6.15.10)does not support Python 3.12yet. As soon as new cx_freeze version with Python 3.12 support is released, it should be taken in to use.py -3.11 -m venv venv_msi_build
.\venv_msi_build\Scripts\Activate.ps1 
pip install -r .\msi_build_requirements.txtNotice:the last command installs ahjo with the most common options. You may need to update msi_build_requirements.txt to suit your needs.Notice:if you make changes to the source code, you must install ahjo into build venv again to have those changes included in the next build. Editable pip install doesn't work here, so don't use it.With the build venv active, build the MSI package with the following command.python .\msi_build.py bdist_msiFind the built MSI installation package under (automatically created)distdirectory.Installing the MSI packageMSI installation package installs everything that is needed to execute ahjo shell commands including the required parts of the Python runtime setup. In other words, the target environment doesn't need to have Python installed and there is no need to create separate venvs for ahjo.Run the msi installer with the default settings it offersMake sure to start a new shell instance (e.g. Windows PowerShell) after the installationAfter a successful installation the following CLI commands are available in the shell:ahjoahjo-init-projectahjo-multi-project-buildahjo-upgradeahjo-scanahjo-install-git-hookahjo-configIf a new shell instance can't find the executables, ensure that installation path is included in the PATH enviroment variableProject InitializationCreate a new project by running the following command:ahjo-init-projectThis will start the project initialization command and you'll be asked to enter a name for your new project:This is Ahjo project initialization command.
Enter project name:Enter a name for your project and hit enter. Confirm, if project name and locations are correct.Enter project name: project_1
You are about to initialize a new project project_1 to location C:\projects
Are you sure you want to proceed?
[Y/N] (N): Y
confirmed
[2019-06-04 08:46:23] Ahjo - Creating new project project_1
Project C:\projects\project_1 created.Usage ExampleBefore running actions:Install Ahjo (see "Install Guide")Initialize project using ahjo-init-project (see "Project Initialization")Have your development database server running (SQL Server for the example)Fill database connection information to the config-fileTo create a database without objects, run the following command in the project root:ahjo init config_development.jsoncAfter the command, there should be a empty database at the server, as configured in config_development.jsonc. This step is usually run only once, and is not required if the database already exists in the server.After tables are defined using alembic (see alembic's documentation for creating new version scripts), the tables can be deployed using:ahjo deploy config_development.jsoncThis command also runs all the SQL scripts that are defined in directories database/functions, database/procedures and database/views.Conventionally scripts under database/data include row inserts for dimension tables, and database/testdata for mock data insertion scripts. To populate the dimension tables run:ahjo data config_development.jsoncTo run test SQL on top of mock data:ahjo testdata config_development.jsonc
ahjo test config_development.jsoncTo run all the previous commands at once, a single (multi-)action "complete-build" can be used:ahjo complete-build config_development.jsoncTo deploy your project to production you need a new config-file. In production environment actions like "downgrade" can be quite hazard. To exclude such actions set "allowed_actions" to a list:"allowed_actions": ["deploy", "data"]Now running "downgrade" is not possible using production configuration.ahjo downgrade config_production.jsonc
[2019-10-01 12:58:12] Starting to execute "downgrade"
Action downgrade is not permitted, allowed actions: deploy, data
------To add your own actions (f.e. for more complex testing), modify ahjo_actions.py.Script and argumentsahjo <action> <config_filename><config_filename>is not mandatory if the config path is defined in environment variableAHJO_CONFIG_PATH.
By default, confirmation is asked for actions that affect the database. Confirmation can be skipped with-nior--non-interactiveargument.
Depending on the configuration, the database credentials can be stored into files or be asked when needed, once per application run. The later option is recommended for production environments. The credential handling is shared with alembic with customenv.pyfile.Pre-defined actions include:init-configInitializes local configuration file. This file is intended to hold configurations that should not be versioned.initCreates the database. Database is created with modulecreate_db.py. Required configurations aretarget_database_name,target_server_hostnameandsql_port. For optional configurations, see config file cheat sheet below.structureCreates database structure, that is schemas, tables and constraints. Primary method runs all SQL files under directories./database/schemas,./database/tablesand./database/constraints. Alternate method runs SQL script./database/create_db_structure.sql. If structure can't be created with primary method (one of the listed directories does not exists etc.), alternate method is attempted.deployRuns alembic migrations, creates views, procedures and functions. First, runsalembic upgrade head. Second, creates functions, views and procedures by executing all SQL files under directories./database/functions,./database/viewsand./database/procedures. Third, attempts to update documented extended properties to database. Finally, updates current GIT version (git describe) to GIT version table.deploy-filesRunsalembic upgrade head, creates database objects by executing all SQL files listed in --files argument and updates current GIT version (git describe) to GIT version table.
Example usage:ahjo deploy-files .\config_development.jsonc --files ./database/procedures/dbo.procedure.sql ./database/functions/dbo.function.sql.dataRuns data insertion scripts. Runs all SQL files under./database/data.testdataInserts data for testing into database. Runs all SQL files under./database/data/testdata.complete-buildRuns actions init, deploy, data, testdata and test in order.dropDrops all views, procedures, functions, clr-procedures and assemblies that are listed in directory./database. Drops are executed with TRY-CATCH.drop-filesDrops database objects from locations that are listed in --files argument. Database objects can be views, procedures, functions or assemblies. Object type is read from --object_type argument. Acceptable --object_type parameters: view, procedure, function, assembly.
Example usage:ahjo drop-files .\config_development.jsonc --files ./database/procedures/dbo.procedure_1.sql ./database/procedures/dbo.procedure_2.sql --object_type procedure.downgradeReverts the database back to basic structure.
First, drops all views, procedures, functions, clr-procedures and assemblies that are listed in directory./database. Drops are executed with TRY-CATCH. Second, runsalembic downgrade base.versionPrints the alembic- and git-version currently in the database.
Alembic version is read fromalembic_version_table. GIT version is read fromgit_table.update-file-obj-propWrite objects and their extended properties (only SQL Server) to JSON files under./docsdirectory.
Documented schemas are listed inmetadata_allowed_schemas.update-db-obj-propUpdate documented extended properties (only SQL Server) from JSON files under./docsdirectory to database.
Updated schemas are listed inmetadata_allowed_schemas.testRuns tests and returns the results.
Runs all SQL scripts under./database/tests.scanScans files in the working directory or git staging area and searches for matches with search rules. Search results are printed to the console and logged to a file.ArgumentShorthandDescriptionRequiredDefault Value--search-rules-srList of rules used to search for matches. Currently it is only possible to search for Finnish Personal Identity Numbers (hetu) from files.Nohetu--filesList of file paths to scan. File paths are regular expressions.No^database/--stage-stScan files in git staging area instead of working directory.NoExamplesScan all files under database directory and git staging area:ahjo scan --files "^database/" --stageScan all files under project root:ahjo scan --files "^.*"Scan only employee.sql file under database/data directory:ahjo scan --files "^database/data/employee.sql$"Scan all .sql files under database/data and database/procedures directories:ahjo scan --files "^database/(data|procedures)/.*\.sql"Scan all files starting with dm. in database/data directory:ahjo scan --files "^database/data/dm\..*"Ignoring scan resultsTo filter out false positives, scan results can be ignored by adding them to theahjo_scan_ignore.yamlfile (in the project root directory).
The file is created automatically when the scan command is run for the first time.
The file should be in the following format:files:-file_path:database/data/example_1.sqlmatches:-match_pattern_1-match_pattern_2-file_path:database/data/example_2.sqlmatches:-match_pattern_3ListYou can view all available actions and their descriptions with commandahjo list.ahjo list
-------------------------------
List of available actions
-------------------------------
'assembly': (MSSQL) Drop and deploy CLR-procedures and assemblies.
'complete-build': (MSSQL) Run 'init', 'deploy', 'data', 'testdata' and 'test' actions.
.
.
.Config FileAhjo requires config file to be JSON, JSONC (JSON with comments) or YAML format. Ahjo configs are located inBACKENDsection of file.
Below is an example of config file (in both JSONC and YAML format) and a cheat sheet for config file parameters.JSONC config file{
    "BACKEND": {
        "allowed_actions": "ALL",
        //Git repository and git version table information
        "url_of_remote_git_repository": "https:\\\\github.com\\user\\projectx\\",
        "git_table": "git_version",
        "git_table_schema": "dbo",
        //Database connection information
        "sql_port": 1433,
        "sql_driver": "ODBC Driver 17 for SQL Server",
        "target_database_name": "PROJECTX",
        "target_server_hostname": "localhost",
        // Database file location
        "database_data_path": "/var/opt/mssql/data/PROJECTX.mdf",
        "database_log_path": "/var/opt/mssql/data/PROJECTX.ldf",
        //Alembic
        //the table that alembic creates automatically for migration version handling
        "alembic_version_table": "alembic_version",
        "alembic_version_table_schema": "dbo",
    }
}YAML config fileBACKEND:## List of allowed Ahjo actions. If all actions are allowed use "ALL" option.allowed_actions:ALL## Git repository url and git version table information.url_of_remote_git_repository:''git_table:git_versiongit_table_schema:dbo## Database connection information.sql_port:14330sql_driver:ODBC Driver 17 for SQL Serversql_dialect:mssql+pyodbctarget_server_hostname:localhosttarget_database_name:DB_NAMEConfig file cheat sheetParameterRequiredDescriptionTypeDefault Valuealembic_version_tableNoName of Alembic version table. Table holds the current revision number.str"alembic_version"alembic_version_table_schemaNoSchema of Alembic version table.str"dbo"allowed_actionsYesList of actions Ahjo is allowed to execute. If all actions are allowed, use"ALL".str or list of strskipped_actionsNoList of actions that are skipped.list of str[]azure_authenticationNoAuthentication type to Azure AD. Possible values are"ActiveDirectoryPassword","ActiveDirectoryInteractive","ActiveDirectoryIntegrated"and"AzureIdentity".strazure_identity_settingsNoA dictionary containing parameters for azure-identity library (used only if azure_authentication is"AzureIdentity"). Dictionary holds a key:"token_url"(str). Note: currently ahjo supports only AzureCliCredential authentication method.dictdatabase_collationNoCollation of database. If the defined collation is different from the database collation, a warning is logged.str"Latin1_General_CS_AS"database_compatibility_levelNoCompatibility level of database.intDepends on server. SQL Server 2017 default is140.database_data_pathNoPath of database data file.strdatabase_file_growthNoThe size (MB) of how much database data file will grow when it runs out of space.int500database_init_sizeNoInitial size (MB) of database data file.int100database_log_pathNoPath of database log file.strdatabase_max_sizeNoMaximum size (MB) of database data file.int10000git_tableNoName of git hash table. Table holds current branch, commit hash and URL of remote repository.str"git_version"git_table_schemaNoSchema of git hash table.str"dbo"metadata_allowed_schemasNoList of schemas that extended properties will be written to JSON files and updated to database. If list left empty, nothing is documented or updated.list of strpassword_fileNoPath of file where password will be stored. If no path given, credentials are asked everytime any database altering action is run.strsql_dialectNoDialect used by SQL Alchemy.str"mssql+pyodbc"sql_driverNoName of ODBC driver.strsql_portYesPort number of target database server.inttarget_database_nameYesName of target database.strtarget_server_hostnameYesHost name of target database server.strurl_of_remote_git_repositoryNoURL of project's remote repository.strusername_fileNoPath of file where username will be stored. If no path given, credentials are asked everytime any database altering action is run.strdb_permissionsNoList of dictionaries containing file locations & scripting variables for setting up database permissions from sql file(s). Dictionary holds keys: "source" (str) and "variables" (dict).listofdictdb_permission_invoke_methodNoInvoke method for setting up database permissions. Available options:"sqlcmd"or"sqlalchemy"(default).str"sqlalchemy"upgrade_actions_fileNoConfiguration file for upgrade actions.str"./upgrade_actions.jsonc"catalog_collation_type_descNoCatalog collation setting of database. If the defined setting is different from the database setting, a warning is logged. Applies only to Azure SQL Databasestr"DATABASE_DEFAULT"display_db_infoNoPrint database collation information to console before running actions.booleantruecontext_connectable_typeNoType of SQLAlchmey object returned by Context.get_connectable(). Possible values are"engine"and"connection".str"engine"transaction_modeNoTransaction management style for ahjo actions. Applied only if context_connectable_type is"connection". Possible values are"begin_once"and"commit_as_you_go". If"begin_once", a transaction is started before running actions and committed after all actions are run. If"commit_as_you_go", a transaction is started before running actions but not committed automatically.str"begin_once"git_version_info_pathNoPath to git version info file. Retrieve git commit information from this file if git repository is not available. JSON file format:{"repository": "<url>", "commit": "<commit hash>", "branch": "<branch name>"}strwindows_event_logNoLog Ahjo events to Windows Event Log.booleanfalseahjo_action_filesNoDefines the location and name of project-specific Ahjo actions files.list of dictsqlalchemy.urlNoSQLAlchemy database URL. If defined, overrides the values ofdialect,sql_port,sql_driver,target_server_hostnameandtarget_database_name.strsqlalchemy.*NoItems under sqlalchemy.* are passed as parameters to SQLAlchemy'screate_enginefunction. For examplesqlalchemy.pool_size: 10is passed as pool_size=10 tocreate_enginefunction.dictIfdialectismssql+pyodbc:"sqlalchemy.use_insertmanyvalues": false,"sqlalchemy.use_setinputsizes": falsesqla_url_query_mapNoA dictionary containingSQLAlchemy URL query connection parameters.dictIf ODBC Driver 18:{"Encrypt" : "yes", "LongAsMax": "Yes"}. If ODBC Driver 17 or older:{"Encrypt" : "no"}. Else:{}Config conversionConfig file can be converted from JSON/JSONC to YAML or vice versa withahjo-configcommand:ahjo-config --convert-to <target_format> --config <config_file_path> --output <output_file_path>Using Alembic with AhjoAlembic upgrade HEAD is used in deploy action, but for many use cases other alembic commands are needed. For these needs Ahjo comes with aenv.pyfile that enables running Alembic commands without running Ahjo.The env.py modifications provide logging integration to Ahjo, default naming schema and possibility to run alembic according to project configuration. The engines are created according to configuration, and there is no need for storing plain-text connection strings in the project.Usage example:alembic -x main_config=config_development.jsonc downgrade -1Theenv.pyis created in initialize-project command.Authentication with azure-identityInstructions for enabling azure identity authentication in ahjo:InstallAzure CLI&azure-identitySet the config variableazure_authenticationtoAzureIdentitySign in interactively through your browser with theaz logincommand.
If the login is successful, ahjo will use Azure credentials for creating an engine that connects to an Azure SQL database.Running actions from multiple projectsTo run all selected actions from different projects at once, a single command "ahjo-multi-project-build" can be used:ahjo-multi-project-build path/to/config_file.jsoncUse-cor--confirmflag to enable confirmation messages for ahjo actions.Below is an example of JSONC config file. With the following definition, multi-project-build command executes complete-build actions of three ahjo projects:{"projects_path":"path/to/projects_folder","connection_info":{"sql_port":14330,"sql_driver":"ODBC Driver 17 for SQL Server","target_server_hostname":"localhost"},"projects":{"ahjo_project_1_name":{"config":"path/to/projects_folder/ahjo_project_1_name/config_development.jsonc","actions":["complete-build"]},"ahjo_project_2_name":{"config":"path/to/projects_folder/ahjo_project_2_name/config_development.jsonc","actions":["complete-build"]},"ahjo_project_3_name":{"config":"path/to/projects_folder/ahjo_project_3_name/config_development.jsonc","actions":["complete-build"]}}}Settings underconnection_infocontains database server definitions in the same format as in ahjo project config file (excludingtarget_database_nameparameter, which is not used here).Currently in this version ahjo projects should be located under the folder specified inprojects_pathsetting.Ahjo project names are listed underprojectssection in run order. In this example, the actions of projectahjo_project_1_nameare executed first and the actions of projectahjo_project_3_nameare executed last.The following settings are defined under the name of the ahjo project(s):config- File path to the project-specific config fileactions- List of project actions to be executedAhjo project upgradeDatabase updates can be run withahjo-upgradecommand. The command detects automatically the latest installed git version and runs the required database version updates (in order).
The upgrade actions are defined in a JSONC file and its location is defined inupgrade_actions_filesetting in project config file.
The ahjo actions required for version upgrades are defined with key-value pairs, where key is the git version tag and value is a list of actions.
The list of actions can contain strings of action names or lists of action names and action parameters.
If the action is defined with parameters, the action name is the first item in the list and the parameters are defined as a dictionary in the second item in the list.
The dictionary contains the parameters of the action as key-value pairs, where key is the parameter name and value is the parameter value.Below is an example of upgrade actions file:{"v3.0.0":["deploy","data"],"v3.1.0":["deploy",["deploy-files",{"files":["./database/procedures/dbo.procedure_1.sql","./database/procedures/dbo.procedure_2.sql"]}]],"v3.1.1":["deploy"]}To upgrade specific version, use-vor--versionflag:ahjo-upgrade -v v3.1.0Scan pre-commit hookAhjo scan command can be used as a pre-commit hook to prevent committing files that contain sensitive information (e.g. personal identity numbers) to the repository.
This can be accomplished by utilizing a Git pre-commit hook script that automatically executes ahjo-scan command on each commit and prevents the commit if the scan finds matches with the defined search rules.
Currently the scan pre-commit hook searches for Finnish Personal Identity Numbers (hetu) from staged files under the project root directory. Later on, the search rules can be extended to cover other use cases as well.
To use the hook, you need to have theahjo-scan.exeaccessible as a shell command. (e.g. the tool is installed from an MSI package, see Installation Guide 3).Setting up the hookTo install the hook, run the following command in the project root directory:ahjo-install-git-hookThe script creates a file namedpre-committo Git hooks directory. By default, Git hooks are located in the.git/hooksdirectory of the repository. This can be changed by setting thecore.hooksPathconfiguration variable to the desired path. SeeGit documentationfor more information.LoggingAhjo's logging is very inclusive. Everything Ahjo prints to console, is also written into log file ahjo.log. Logging can be done to Windows Event Log by settingwindows_event_logtotruein config file. This feature can be utilized for Azure Monitor activities, for example.CustomizationSetting up custom action filesEvery default Ahjo action and multiaction can be overwritten in project's Ahjo actions file. By default, the file is located inahjo_actions.py, but the location can be changed in config file with keyahjo_action_files. It is possible to define multiple Ahjo actions files. This can be useful for example when you want to use different actions for different environments or separate actions that are compatible with MSI-installed ahjo from actions that are compatible with pip-installed ahjo. The action files are loaded in order, so if you define multiple actions for the same action name, the last loaded action file will be used. If a multi-action uses actions from different files, remember to load all the subactions before the multi-action. Here is an example ofahjo_action_filesconfiguration:"ahjo_action_files": [
    {
        "source_file": "ahjo_prod_actions.py", // The location is relative to project root directory.
        "name": "Example project Ahjo actions (prod)" // Name is used in logging.
    },
    {
        "source_file": "ahjo_dev_actions.py", 
        "name": "Example project Ahjo actions (dev)"
    }
]Overwriting default Ahjo actionsIn example below,'init'and'complete-build'actions are overwritten.importahjo.operationsasopfromahjo.actionimportaction,create_multiaction@action(affects_database=True)defstructure(context):"""New structure action."""frommodelsimportBaseBase.metadata.create_all(context.get_engine())# overwrite Ahjo complete-buildcreate_multiaction("complete-build",["init","structure","deploy","data","test"])LicenseCopyright 2019 - 2023 ALM Partners OyLicensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.