Package: hashmap-data-migrator
Hashmap Data MigratorTable of ContentsAboutUsing hdmState ManagementPipeline YAMLConnection Profile YAMLLoggingUser DocumentationSourcesNetezzaSourceNetezzaExternalTableSourceFSSourceFSChunkSourceAzureBlobSourceDummySourceSinksFSSinkAzureBlobSinkSnowflakeAzureCopySinkDummySinkStateManagementAzureSQLServerStateManagerSQLServerStateManagerMySQLStateManagerSqLiteStateManagerStateManager- Base classData Access Object (DAO)NetezzaJDBCNetezzaODBCSnowflakeAzureCopyMySQLSqLiteStateSQLServerAzureSQLServerCatalogNetezzaCrawlerNetezzaToSnowflakeMapperSnowflakeDDLWriterOrchestratorDeclaredOrchestratorBatchOrchestratorAutoOrchestratorUtilsProject ConfigurationRepository CloningMiscellaneousAboutHashmap Data Migrator, or hdm, is a collection of composable data transport modules designed to incrementally move data from sources systems to cloud data systems.Data is moved through pairs of sources and sinks. Ideally, this movement is meant to use a dataset as an intermediary. This allows one to create pipelines that follow modern practices while at the same time solving many issues that may arise if a pipeline is kept in memory in its entirety. While there is an additional IO overhead in many cases, this also allows workloads to be distributed arbitrarily and for portions of the pipelines to be ran on disparate systems - sharing only a state management data store (when full lineage is desired). In fact, in many situations, especially those of interest, data necessarily must be stored in files, broken into smaller chunks and transported across the wire to a cloud provider from an on-premises system.Using Hashmap Data MigratorTo use the Hashmap Data Migrator (hdm) you must firstIdentify all locations where you would like for your solution to be deployed. It is a distributed application and can be partially deployed in many locations.If it does not already exist in the deployment environment, create a hidden directory in the 'user' root called .hashmap_data_migrator.Within the directory created in step 2 above, you must create a [connection profile YAML](Connection Profile YAML). This will hold the necessary connection information to connect Netezza, BigQuery and other data sources. Out of the box, at this time, there is no key management solution integrated. This is on the feature roadmap.Install hashmap_data_migrator and all of its dependencies. This is a pypi package and can be installed aspipinstallhashmap-data-migratorHave a database available to use for state management.Pipelines are defined declaratively in YAML files. These YAML files identifyThe orchestrator (internal hdm concept) used to orchestrate the execution of your pipeline. The options are:declared_orchestrator - for manual or fully specified executionbatch_orchestrator - for when orchestration is defined in a fully specified batchauto_batch_orchestrator - for when the execution is across all tables in specified combinations of databases and schemasIt is formatted in the YAML as such:orchestrator:name:Manual Orchestrationtype:declared_orchestratorconf:nullNext, and this should be consistent across all of the pipelines, the State Manager is specified. This is the glue the couples the otherwise independent portions of the pipeline together.It is formatted in the YAML as such:state_manager:name:state_managertype:SQLiteStateManagerconf:connection:state_managerNext is the portion of the YAML that specifies the different steps in the data movement. These will be specified in two separate sections:declared_data_links - these are fully specified portions of a pipeline. In this each pair of source & sink is called a stage. See the example below that is targeted at offloading data from Netezza and storing it on a filesystem.declared_data_links:stages:-source:name:Netezza Source Connectiontype:NetezzaSourceconf:...sink:name:File System Sink Connectiontype:FSSinkconf:...template_data_links - these are partially defined source/sink pairs. Instead of being called stages they are called templates. A template has an additional field called a batch_definition. A batch definition will define how the template source is used - source is ALWAYS the template. See an example below for creating a pipeline that is pulling multiple tables at once. A similar example would be found for auto batch orchestration.template_data_links:templates:-batch_definition:-source_name:netezza_sourcefield:table_namevalues:-database.schema.table_1-database.schema.table_1threads:5source:name:netezza_sourcetype:NetezzaSourceconf:...table_name:<<template>>sink:name:fs_chunk_stgtype:FSSinkconf:...NOTE:Any asset (source or sink) specified must either exist or be creatable through the connector. Any and all credentials must exist in the hdm_profiles.yml as well.NOTE:The pipeline can be split into separate files and executed distributively.Now, before we move on to more detailed documentation there remains one last function - cataloging operations. Before you run your code, when you are migrating data from one database to another you mustCatalog the existing assetsMap the assets in the source system to the target systemNow that the environment is specified, pipeline defined, and so on, all that remains is to run the code. The code is executed from bash (or at the terminal) throughpython -m hashmap_data_migrator {manifest} -l {log settings} -e {env}

or

hashmap_data_migrator {manifest} -l {log settings} -e {env}The parameters are:manifest - path of manifest to runlog_settings - log settings path , default value ="log_settings.yml"env - environment to take connection information , default value ="prod"State ManagementThe management of the state of data movement. Useful for handling failures, storing history (audit trails) and much more. State of all data transport is stored in a database - which database is up to you - there is an extensible API with many out of the box implementations. This state management data is used to manage the control flow of a pipeline from end-to-end across a distributed deployments.As of now MySQL, SQLLite, SQL Server and Azure SQL Server database tables can be used for state management.The state management table will track the following values:state_id: unique identifierrun_id: unique identifier for a runjob_id: unique identifier for a stage (source and sink pair).correlation_id_in: Correlation id linking to a preceding pipelinecorrelation_id_out: Correlation id on persistenceaction: Action performed - sourcing pre-pull | sourcing post-pull| sinking pre-pull | sinking post-pullstatus: Status of the transformation - success | failure | in_progresssource_name: Name of sourcesource_type: Type of sourcesink_name: Name of sinksink_type: Type of sinksource_entity: Asset being transportedsource_filter: Any filtering appliedsink_entity: Asset being dumpedsink_filter: Any filtering appliedfirst_record_pulled: First record pulled in the run.Relevant to database only.last_record_pulled: Last record pulled in the run.Relevant to database only.git_sha: Correlates code execution to the latesourcing_start_time: When sourcing startedsourcing_end_time: When sourcing endedsinking_start_time: When sinking startedsinking_end_time: When sourcing endedupdated_on: When this entry was last updatedrow_count: Number of distinct rows extractedcreated_on: When this entry was createdmanifest_name: Name of the pipeline YAML filePipeline YAMLWhat the user should be focused on until a front-end gets built. This is merely a configuration file,
given parameters by the user it will flow these data points to the relevant classes and move the data from source to sink.
Example YAML can be found under manifests folder.version:1state_manager_type:data_links:type:buildermode:manualstages:-source:name:source_name_1type:NetezzaSourceconf:sink:name:sink_name_1type:FSSinkconf:-source:name:source_name_2type:FSSourceconf:sink:name:sink_name_2type:s3Sinkconf:-source:name:source_name_3type:s3Sourceconf:sink:name:sink_name_3type:SnowflakeCopySinkconf:Connection Profile YAMLThis files stores the connection information to the source, stage , sink, database for state management.
Its stored in local FS and its path is set in environment variable "HOME".
The below yml file format is based on netezza to snowflake data transport using azure blob staging:dev:netezza_jdbc:*Note:Add this section if using JDBC driverhost:<host>port:<port>database:<database_name>user:<user_name>password:<password>driver:name:<driver_name>path:<driver_path>netezza_odbc:*Note:Add this section if using ODBC driverhost:<host>port:<port>database:<database_name>user:<user_name>password:<password>driver:<driver_name>snowflake_admin_schema:authenticator:snowflakeaccount:<account>role:<role>warehouse:<warehouse_name>database:<database_name>schema:<schema_name>user:<user_name>password:<password>azure:url:<blob_url>azure_account_url:<blob_url starting with azure://...>sas:<sas_key>container_name:<blob_container_name>state_manager:host:<host>port:<port>database:<database_name>user:<user_name>password:<password>driver:ODBC Driver 17 for SQL Server <*Note:only for azure sql server>LoggingThe application logging is configurable in log_settings.yml. The log files are created at the root.version:1formatters:simple:format:'%(asctime)s-%(name)s-%(levelname)s-%(message)s'json:format:'%(asctime)s%(name)s%(levelname)s%(message)s'class:pythonjsonlogger.jsonlogger.JsonFormatterhandlers:console:class:logging.StreamHandlerformatter:simplelevel:INFOstream:ext://sys.stdoutfile:class:logging.handlers.RotatingFileHandlerformatter:jsonlevel:<logging level>filename:<log file name>maxBytes:<maximum bytes per log file>backupCount:<backup log file count>loggers:hdm:level:<logging level>handlers:[file,console]propagate:Truehdm.core.source.netezza_source:level:<logging level>handlers:[file,console]propagate:Truehdm.core.source.fs_source:level:<logging level>handlers:[file,console]propagate:<logging level>hdm.core.source.snowflake_external_stage_source:level:<logging level>handlers:[file,console]propagate:TrueUser DocumentationSourcesNetezzaSourceNetezza storage source.base classRDBMSSourceconfigurationenv - section name in hdm profile yml file for connection informationtable_name - table namewatermarkcolumn: watermark column nameoffset: offset value to compare tochecksumfunction: checksum function. supported checksum methods are: hash, hash4, hash8column: checksum column name- source:
        name: netezza_source
        type: NetezzaSource
        conf:
          env: netezza_jdbc
          table_name: ADMIN.TEST1
          watermark:
              column: T1
              offset: 2
          checksum:
              function:
              column:consume APIinput:env: section name in hdm profile yml file for connection information | required
table_name: table name | required
watermark: watermark information
checksum: checksum information | default randomoutput:data_frame: pandas.DataFrame
source_type: 'database'
record_count: pandas.DataFrame shape
table_name: table nameNetezzaExternalTableSourceNetezza External Table storage source.base classSourceconfigurationenv - section name in hdm profile yml file for connection informationtable_name - table namedirectory - staging directorywatermarkcolumn: watermark column nameoffset: offset value to compare tochecksumfunction: checksum function. supported checksum methods are: hash, hash4, hash8column: checksum column name- source:
        name: netezza_source
        type: NetezzaExternalTableSource
        conf:
          env: netezza_jdbc
          table_name: ADMIN.TEST1
          watermark:
              column: T1
              offset: 2
          checksum:
              function:
              column:
          directory: $HDM_DATA_STAGINGconsume APIinput:env: section name in hdm profile yml file for connection information | required
table_name: table name | required
directory: staging directory | required
watermark: watermark information
checksum: checksum information | default randomoutput:data_frame: pandas.DataFrame | required
source_type: 'database'
record_count: pandas.DataFrame shape | requiredFSSourceFile system storage source.base classSourceconfigurationdirectory - staging directory- source:
        name: fs_stg
        type: FSSource
        conf:
          directory: $HDM_DATA_STAGINGconsume APIinput:directory: staging directory | requiredoutput:data_frame: pandas.DataFrame | required
file_name: file name
record_count: pandas.DataFrame shape | required
table_name: extracted table name from blob file pathFSChunkSourceFile system chunking storage source.base classSourceconfigurationdirectory - staging directorychunk - file chunk size- source:
        name: fs_chunk_stg
        type: FSChunkSource
        conf:
          directory: $HDM_DATA_STAGING
          chunk: 200consume APIinput:directory: staging directory | required
chunk: file chunk sizeoutput:data_frame: pandas.DataFrame | required
file_name: file name
parent_file_name: file name of the large parent file | required
record_count: pandas.DataFrame shape | required
table_name: extracted table name from file path
source_filter: source filter to query state management record for updates when the source_entity is same (the parent file name)| requiredAzureBlobSourceBLOB data storage source.base classSourceconfigurationenv - section name in hdm profile yml file for connection informationcontainer - blob container name- source:
        name: azure_source
        type: AzureBlobSource
        conf:
          env: azure
          container: dataconsume APIinput:env: section name in hdm profile yml file for connection information | required
container: container name | required
file_format: file format | default csvoutput:data_frame: pandas.DataFrame | required
file_name: file name
record_count: pandas.DataFrame shape | required
table_name: extracted table name from blob file pathDummySourceDummy storage source. Use when a source is not needed.base classSourceconfigurationdummy: None- source:
        name: source_name
        type: DummySource
        conf:
          dummy: Noneconsume APIinput:dummy: placeholder | requiredoutput:NoneSinksFSSinkFile system storage sinkbase class:Sinkconfigurationdirectory - staging directorysink:
        name: sink_name
        type: FSSink
        conf:
          directory: $HDM_DATA_STAGINGconsume APIinput:directory: staging directory | requiredoutput:record_count: pandas.DataFrame shape | requiredAzureBlobSinkAzure blob storage sinkbase classSinkconfigurationenv - section name in hdm profile yml file for connection informationcontainer - staging directorysink:
        name: azure_sink
        type: AzureBlobSink
        conf:
          env: azure
          container: dataconsume APIinput:env: section name in hdm profile yml file for connection information | required
container: container name | requiredoutput:record_count: pandas.DataFrame shape | requiredSnowflakeAzureCopySinkSnowflake Azure storage stage sinkbase classSnowflakeCopySinkconfigurationenv - section name in hdm profile yml file for connection informationstage_name - staging directoryfile_format - file formatstage_directory - azure blob container namesink:
        name: sflk_copy_into_sink
        type: SnowflakeAzureCopySink
        conf:
          stage_name: TMP_KNERRIR
          file_format: csvconsume APIinput:env: section name in hdm profile yml file for connection information | required
stage_name: snowflake storage stage name| required
file_format: file format | required
stage_directory: container name | requiredoutput:record_count: pandas.DataFrame shape | requiredDummySinkDummy sink. Use when a sink is not needed.base classSinkconfigurationdummy: Nonesink:
        name: sink_name
        type: DummySink
        conf:
          dummy: Noneconsume APIinput:dummy: placeholder | requiredoutput:NoneState ManagementAzureSQLServerStateManagerAzure SQL Server State Managementbase classStateManagerconfigurationconnection - state_managerstate_manager:
  name: state_manager
  type: AzureSQLServerStateManager
  conf:
    connection: state_managerconsume APIinput:connection: state_manager  | required
dao: 'azuresqlserver'  | preset value in code | required
format_date: False   | preset value in code | requiredSQLServerStateManagerSQL Server State Managementbase classStateManagerconfigurationstate_manager:
  name: state_manager
  type: SQLServerStateManager
  conf:
    connection: state_managerconsume APIinput:connection: state_manager  | required
dao: 'azuresqlserver'  | preset value in code | required
format_date: False   | preset value in code | requiredMySQLStateManagerMYSQL State Managementbase classStateManagerconfigurationconnection - state_managerstate_manager:
  name: state_manager
  type: MySQLStateManager
  conf:
    connection: state_managerconsume APIinput:connection: state_manager  | required
dao: 'mysql'  | preset value in code | required
format_date: True   | preset value in code | requiredSqLiteStateManagerSQLLite State Managementbase classStateManagerconfigurationconnection - state_managerstate_manager:
  name: state_manager
  type: SqLiteStateManager
  conf:
    connection: state_managerconsume APIinput:connection: state_manager  | required
dao: 'sqlite'  | preset value in code | required
format_date: True   | preset value in code | requiredStateManagerState Management base classMethods:name: insert_state
      insert new state
params: source_entity, source_filter,action,state_id,
       status, correlation_id_in, correlation_id_out,
       sink_entity, sink_filter,sourcing_start_time, 
       sourcing_end_time, sinking_start_time, sinking_end_time,
       record_count, first_record_pulled, last_record_pulled

return value: dictionary of state_id,job_id,correlation_id_in,
               correlation_id_out,source_entity,source_filter,
               sourcing_start_time,sourcing_end_time,sinking_start_time,
               sinking_end_time, first_record_pulled,last_record_pulled,
               record_count,run_id,manifest_namename: update_state
      update a state
params:source_entity, source_filter,action,state_id,
           status, correlation_id_in, correlation_id_out,
           sink_entity, sink_filter,sourcing_start_time, 
           sourcing_end_time, sinking_start_time, sinking_end_time,
           record_count, first_record_pulled, last_record_pulled
return value :dictionary of state_id,job_id,correlation_id_in,
                   correlation_id_out,source_entity,source_filter,
                   sourcing_start_time,sourcing_end_time,sinking_start_time,
                   sinking_end_time, first_record_pulled,last_record_pulled,
                   record_count,run_id,manifest_namename: get_current_state
      get current state
params: job_id | required, entity, entity_filter
return value : dictionary of state_id,job_id,correlation_id_in,
                   correlation_id_out,source_entity,source_filter,
                   sourcing_start_time,sourcing_end_time,sinking_start_time,
                   sinking_end_time, first_record_pulled,last_record_pulled,
                   record_count,run_id,manifest_namename: get_last_record
      gets last_record_pulled value
params: entity
return value : last_record_pulledname: get_processing_history
      get processing history
params: none
return value : list of sink_entity for a source_nameData Access ObjectNetezzaJDBCconnect to netezza using JDBC driverbase classnetezzainput:connection:  section name in hdm profile yml file for connection information | requiredoutput:connection - connection to netezzaNetezzaODBCconnect to netezza using ODBC driverbase class:netezzainput:connection:  section name in hdm profile yml file for connection information | requiredoutput:connection - connection to netezzaAzureBlobconnect to azure storage accountbase classObjectStoreDAOinput:connection:  section name in hdm profile yml file for connection information | requiredoutput:connection - connection to netezzaSnowflakeAzureCopyconnect to snowflake
create or replace snowflake azure storage stagebase classSnowflakeCopyinput:connection:  section name in hdm profile yml file for connection information | required
stage_directory: name of azure blob container used for staging files.
stage_name: name of snowflake azure storage stageoutput:connection - connection to snowflakeMySQLconnect to MySQL dbbase classDBDAOinput:connection:  section name in hdm profile yml file for connection information | requiredoutput:connection - connection to MySQL
engine - connection engineSQLliteconnect to SQLlite dbbase classDBDAOinput:connection:  section name in hdm profile yml file for connection information | requiredoutput:connection - connection to SQLlite
engine - connection engineSQLServerconnect to SQL Server dbbase classDBDAOinput:connection:  section name in hdm profile yml file for connection information | required
engine - connection engineoutput:connection - connection to SQL Server
engine - connection engineAzureSQLServerconnect to Azure SQL Server dbbase classDBDAOinput:connection:  section name in hdm profile yml file for connection information | requiredoutput:connection - connection to Azure SQL Server
engine - connection enginecatalogGet databases, schemas and tables from Netezza DW and create the same in Snowflake DW.
Run the below:python -m CloneNetezzaDDLNetezzaCrawlerGet database, schema and table information in Netezza DW.input:connection_name:  section name in hdm profile yml file for connection information | requiredoutput:databases: list of databases | required
schemas: list of schemas| required
tables: list of tables | requiredNetezzaToSnowflakeMapperExecute database, schema and table ddlsinput:databases: list of databases | required
schemas: list of schemas| required
tables: list of tables | requiredoutput:database_sql: database ddl | required
schema_sql: schema ddl | required
table_sql: table ddlSnowflakeDDLWriterExecute database, schema and table ddlsinput:env: section name in hdm profile yml file for connection information | required
database_sql: database ddl | required
schema_sql: schema ddl | required
table_sql: table ddloutput:noneOrchestrationDeclaredOrchestratorThis is an orchestrator which build DataLinks and will run them as defined - they must be fully defined.base classorchestratorconfigurationorchestrator:
  name: Manual Orchestration
  type: DeclaredOrchestrator
  conf: nullconsume APIinput:noneoutput:data_links: list of data links  | requiredBatchOrchestratorThis is an orchestrator which build DataLinks and will run them as defined or templated.base classorchestratorconfigurationorchestrator:
  name: Batch Orchestration
  type: BatchOrchestrator
  conf: nullconsume APIinput:noneoutput:data_links: list of data links  | requiredAutoOrchestratorThis is an orchestrator which build DataLinks and will run them as defined or templated.
The template will have information about schema and database of the source system.Note: This is work in progress ...base classorchestratorconfigurationorchestrator:
  name: Auto Batch Orchestration
  type: AutoOrchestrator
  conf: nullconsume APIinput:noneoutput:data_links: list of data links  | requiredUtilsProject ConfigurationThe following can be configured in utils/project_config.pyprofile_path - folder and name of profile YML filefile_prefix - prefix to be used for staged filesstate_manager_table_name - state management table namearchive_folder - archive folder name (used in FSChunkSource)connection_max_attempts - maximum number of try  (used in DAOs)connection_timeout - connection timeout value  (used in DAOs)query_limit - numbers rows returned by a queryRepository CloningPlease refer toclone readmeMiscellaneousThe database, schema, table name are part of the folder structure where files are dumped - in any file staging location.The sink_name of a stage is part of the folder structure where files are dumped - in any file staging location.Any file created locally is stored in {HDM_DATA_STAGING}{sink_name}{table_name}\Any file created in cloud staging is stored in folder structure {sink_name}{table_name}\The source_name and sink_nameMUST MATCH between stages (combination of source and sink)for the migrator to be able to pick up files for processing.
E.g. Below you can see that sink_name for stage "FS to AzureBlob" isazure_sink. so, the source_name for stage "cloud storage create staging and run copy"
is alsoazure_sink# FS to AzureBlob
    - source:
        name: fs_chunk_stg
        type: FSSource
        conf:
          directory: $HDM_DATA_STAGING
      sink:
        name: *azure_sink*
        type: AzureBlobSink
        conf:
          env: azure
          container: data

#cloud storage create staging and run copy
    - source:
        name: *azure_sink*
        type: AzureBlobSource
        conf:
          env: azure
          container: data
      sink:
        name: sflk_copy_into_sink
        type: SnowflakeAzureCopySink
        conf:
          env: snowflake_knerrir_schema
          stage_name: TMP_KNERRIR
          file_format: csv
          stage_directory: data