Package: yoyo-indexima
yoyo-indeximaVersions followingSemantic VersioningOverviewIndeximamigration schema based onyoyoandpyhive.The little storyIn the land of database migration tool, i have tried:flywayliquidbase with hive extentionBoth either did not support hive (as flyway), or indexima did not fully compliant with hive (wich cause probleme with liquidbase)So I try to found a module not too complex in order to migrate our indexima schema in a safe way.In this early release, I just trying to do the job.SetupRequirementsPython 3.7+InstallationInstall this library directly into an activated virtual environment:$ pip install yoyo-indeximaor add it to yourPoetryproject:$ poetry add yoyo-indeximaUsageHive connectionbackend ui must start withindexima://If you have trouble to obtain an hive connection, please readhttp://dwgeek.com/guide-connecting-hiveserver2-using-python-pyhive.html/Note:
If you using python in docker, you should install :apt-get update -qq
apt-get install -qqy gcc libsasl2-dev libsasl2-2 libsasl2-modules-gssapi-mitMigrationYou could see a complete sample under 'example' folder.using python clientyoyo_indexima
usage: yoyo_indexima [-h] [-s SOURCE] -u URI {show,apply}example:yoyo_indexima  apply  -s $(pwd)/example/migrations/ -u "indexima://admin:super_password@localhost:10000/default"Commands:show                Show migrationsapply               Apply migrationsreapply             Reapply migrationsrollback            Rollback migrationsmark                Mark migrations as applied, without running themunmark              Unmark applied migrations, without rolling them backbreak-lock          Break migration locksHelp for apply:>yoyo_indeximaapply-h
usage:yoyo_indeximaapply[-h][-sSOURCE]-uURI[-f][-a][-rREVISION][-d]optionalarguments:-h,--helpshowthishelpmessageandexit-sSOURCE,--sourceSOURCEsourcepathofmigrationscript(default./migrations)-uURI,--uriURIbackenduri-f,--forceForceapply/rollbackofstepsevenifpreviousstepshavefailed-a,--allSelectallmigrations,regardlessofwhethertheyhavebeenpreviouslyapplied-rREVISION,--revisionREVISIONApply/rollbackmigrationwithidREVISION-d,--dry-runDryrun:nomodificationwillbeappliedwithin python codeIf your migrations script are under directorymigrationfolderimportosfromyoyo_indeximaimportget_backend,read_migrationsif__name__=="__main__":# obtain IndeximaBackendbackend=get_backend('indexima://admin:super_password@localhost:10000/default?auth=CUSTOM')# Read migrations foldermigrations=read_migrations(os.path.join(os.getcwd(),'migrations/**/*'))print(f'migrations:{migrations}')ifmigrations:# apply migrationwithbackend.lock():backend.apply_migrations(backend.to_apply(migrations))Management tableThis tool create in yourdefaultschema:a log table: 'yoyo_log'a lock_table: 'yoyo_lock'a version table: 'yoyo_version'a migration table: 'yoyo_migration'Migration script template"""{message}"""fromyoyoimportstep__depends__={{{depends}}}steps=[step("create ...","drop ...")]Configure hive connectionIn python script, onIndeximaBackendinstance, you could use:set_hive_configuration: A dictionary of Hive settings (functionally same as thesetcommand)set_hive_thrift_transport: an instance of TSaslClientTransportAs see inhttps://github.com/dropbox/PyHive/issues/162, you could do things like that:importsaslfromthrift_saslimportTSaslClientTransportfromthrift.transport.TSocketimportTSocketdefcreate_hive_plain_transport(host,port,username,password,timeout=60):socket=TSocket(host,port)socket.setTimeout(timeout*1000)sasl_auth='PLAIN'defsasl_factory():sasl_client=sasl.Client()sasl_client.setAttr('host',host)sasl_client.setAttr('username',username)sasl_client.setAttr('password',password)sasl_client.init()returnsasl_clientreturnTSaslClientTransport(sasl_factory,sasl_auth,socket)backend=get_backend('indexima://admin:super_password@localhost:10000/default?auth=CUSTOM')backend.set_hive_thrift_transport(create_hive_plain_transport(...))Extends IndeximaBackendIf you extendsIndeximaBackendyou could register your classes, in the function:get_backend(uri: str, backend=IndeximaBackend) -> DatabaseBackend:TODO: add a client parameter to specify full class name in cli.LicenseThe MIT License (MIT)ContributingSeeContributingNext stepproduction usage in order to see how this tool made the jobmore documentation in codepurpose few change in 'yoyo' like set all SQL command on Backend class...