Package: database-factory
database-factoryDatabase factory is used to manage/create database connection with execute queries using the connection.
The concept of having single source to connect various databases and perform database operations.User need not to worry on the crafting the connection string and to identify the methods for the database operations.
Database factory supports DML / DDL executions and have support of Pandas DataFrame to create or replace existing tables.Database factory is wrapper on sqlalchemy for crafting the connection and supports below databases:*Sqlite3
*PostgreSQl
*BigQuery
*Snowflake
*MariaDB
*MySQLDatabase factory can be enhanced for all the sqlalchemy supported database.Getting Startedpipinstalldatabase-factoryUsing database-factoryfromdatabase_factory.managerimportDatabaseManagerimporttempfiletemp_dir=tempfile.gettempdir()db=DatabaseManager(engine_type="sqlite",database="test_db",sqlite_db_path=temp_dir)db.create_session()db.execute_sql(sql="create table test (id int PRIMARY KEY)")db.execute_sql(sql="insert into test values (1)")db.execute_sql(sql="insert into test values (2)")rows=db.execute_sql(sql="select * from test")ifrows:print(rows)df=db.get_df(sql="select * from test")print(df)db.execute_df(panda_df=df,table_name=copy_test,exist_action="replace")# db.execute_df(panda_df=df, table_name=copy_test, exist_action="replace", chunk_size=100)db.execute_sql(sql="insert into copy_test values (3)")rows_copy=db.execute_sql(sql="select * from copy_test")ifrows_copy:print(rows_copy)AppendixSupported database type:*   sqlite `default`
*   postgres
*   mysql
*   mariadb
*   snowflakeConnection parameters for sqlite:*engine_type:sqlite*database:<nameofdatabase>*sqlite_db_path:<pathwheredatabasewillbecreated>Connection parameters for postgres:*engine_type:postgres*database:<nameofdatabase>*username:<postgresuser>*password:<userpassword>*host:<hostofpostgresservice>*port:<portofpostgresservice>Connection parameters for mysql:*engine_type:mysql*database:<nameofdatabase>*username:<mysqluser>*password:<userpassword>*host:<hostofmysqlservice>*port:<portofmysqlservic\>Connection parameters for mariadb:*engine_type:mariadb*database:<nameofdatabase>*username:<mariadbuser>*password:<userpassword>*host:<hostofmariadbservice>*port:<portofmariadbservice>Connection parameters for snowflake:*engine_type:snowflake*database:<nameofdatabase>*username:<snowflakeuser>*password:<userpassword>*schema:<schemaname>*snowflake_role:<snowflakerole>*snowflake_warehouse:<snowflakewarehouse>*snowflake_account:<snowflakeaccount>Connection parameters for bigquery:*engine_type:bigquery*database:<nameofdatabase>Getting connection properties from AWS / GCP Secret Manager Service:Note:GCP:On Cloud Server:Set server to execute the all cloud api servicesAttach following permissionsProject ViewerSecret Manager Secret AccessorOn Premises:Attach following permissions to user service account and download service account file for authentication:Project ViewerSecret Manager Secret AccessorSet environment variable "GOOGLE_APPLICATION_CREDENTIALS" pointing to service account file.AWS:On Cloud Server:Set execution profile with "secretsmanager:GetSecretValue" policyOn Premises:AWS should be configuredUser should have permissions of "secretsmanager:GetSecretValue" policy.*engine_type:bigquery*database:<nameofdatabase>*secret_id:<SecretnameofAWS/GCPSecretManagerService>*secrete_manager_cloud:<awsorgcpaspercloud>*aws_region:<awsregion:default=>us-east-1>Development SetupUsing virtualenvpython3-mvenvvenvsourceenv/bin/activate
pipinstall.ContributingFork repo-https://github.com/shrivastava-v-ankit/database-factory.gitCreate your feature branch -git checkout -b feature/nameInstall Python packagessqlalchemy==1.4.47pandas==1.5.3GitPythoncoverage==7.2.3exceptiongroup==1.1.1iniconfig==2.0.0pluggy==1.0.0pytest==7.3.0pytest-cov==4.0.0tomli==2.0.1Run Python test (pytest)pytest -v --cov --cov-report html --cov-report xml --junitxml=test-results/database_factory_test/results.xmlAdd Python test (pytest) and covrage report for new/changed feature.Commit your changes -git commit -am "Added name"Push to the branch -git push origin feature/nameCreate a new pull request