Package: dbsavior
DataBaseBackupRestoreOnK8sProject IntroductionIf you have data that you want to import or export from a database that you deployed in a k8s cluster, you need to
install a database client, create a dump file, then upload the dump file somewhere else.This project will help you to avoid all that. The only thing you need to do is to provide the name of the database and
where you want to upload/download the backup file.FeaturesCreate database backup fileUpload backup file to remote storageDownload backup file from remote storageRestore/Populate databaseProvide k8s cron job to create backup with configurable time intervalIf you use auto backup/restore feature in the cronjob/job, the restore job will always find the latest backup of the
target database to do restore.List available database that can be backed up.List available backup that can be used to restoreUsageThere are different ways to use this project.Use this project as a python packageUse this project as a python Command Line InterfaceUse this project as a k8s cronjob/jobUse this project as a python package# install this project as a pypi packagepipinstalldbsaviorimportosfromdbsavior.db.PostgresDbManagerimportPostgresDbManagerfromdbsavior.storage.S3StorageEngineimportS3StorageEnginefromdbsavior.storage.LocalStorageEngineimportLocalStorageEnginefromdbsavior.DbBackupRestoreBotimportDbBackupRestoreBotdefmain():# params to be configured in the job or cron jobuser_name:str="user-pengfei"user_password:str="changeMe"host_name:str="postgresql-124499"port:str="5432"db_name:str="test"backup_storage_path:str="s3://pengfei/tmp/sql_backup"# create an instance of postgresDbManagerp_manager=PostgresDbManager(user_name,user_password,host_name=host_name,port=port)# temp local path if you use remote storage# get s3 credsendpoint="https://"+os.getenv("AWS_S3_ENDPOINT")access_key=os.getenv("AWS_ACCESS_KEY_ID")secret_key=os.getenv("AWS_SECRET_ACCESS_KEY")session_token=os.getenv("AWS_SESSION_TOKEN")# build s3 clients3=S3StorageEngine(endpoint,access_key,secret_key,session_token)# create an instance of DbBackupBotbackup_restore_bot=DbBackupRestoreBot(s3,p_manager)# do an auto backupbackup_restore_bot.make_auto_backup(db_name,backup_storage_path)# restore the database by using the latest backupbackup_restore_bot.restore_db_with_latest_backup(db_name,backup_storage_path)if__name__=="__main__":main()Use this project as a python Command Line InterfaceNote, if you use remote storage such as s3 (or minio), you need to set up your s3 creds correctly in your env variables.gitclonehttps://github.com/pengfei99/K8sCronJobPostgresBackup.gitcdK8sCronJobPostgresBackup# show the command line options and commentspythondbsavior/main.py-h# list existing backuppythondbsavior/main.py--db_typepostgres--storage_types3--actionlist_backups--backup_dirs3://path/to/sql_backup# list existing databasepythondbsavior/main.py--db_typepostgres--storage_types3--actionlist_dbs--db_loginuser-pengfei--db_pwdchangeMe--db_host127.0.0.1--db_port5432# auto backuppythondbsavior/main.py--db_typepostgres--storage_types3--actionauto_backup--db_loginuser-pengfei--db_pwdchangeMe--db_host127.0.0.1--backup_dirs3://path/to/sql_backup--target_dbtest# auto restorepythondbsavior/main.py--db_typepostgres--storage_types3--actionauto_restore--db_loginuser-pengfei--db_pwdchangeMe--db_host127.0.0.1--backup_dirs3://path/to/sql_backup--target_dbtest# populate a database with a sql dumppythondbsavior/main.py--db_typepostgres--storage_types3--actionpopulate--db_loginuser-pengfei--db_pwdchangeMe--db_host127.0.0.1--backup_files3://path/to/sql_backup/2022-01-12_test_pg_bck.sql--target_dbtestUse this project as a k8s cronjob/jobFor the job and cronjob is set up for s3 and postgresql. You need to add your s3 and postgresql server credentials
into the yaml file to run the job and cronjob correctly.gitclonehttps://github.com/pengfei99/K8sCronJobPostgresBackup.gitcdK8sCronJobPostgresBackup/k8s# this job calls auto backup on a database oncekubectlapply-fjob_backup.yaml# this cronjob calls auto backup on a database based on the cron. The default cron starts at mid night every daykubectlapply-fcronjob_backup.yaml# this job calls auto restore of a database, if it finds many available backups, it will apply the latest onekubecltapply-fjob_restore.yaml# this job populate a database with a specific backup file.kubectlapply-fjob_populate.yamlFuture worksFor now, this project only implements the postgres server for theDbManagerInterface. For theStorageEngineInterface, we only implement the s3 and local filesystem(tested for linux). If you want to back up a
database in a mysql server and store the backup on HDFS, you only need to implement theDbManagerInterfaceandStorageEngineInterface. The rest of the project can be reused automatically.Other docs that may be useful for youIf you are not familiar with postgresql backup and restore procedure. Please visit
thisdoc.