Project: springxd
Issue XD-2309: Incremental data import with jdbchdfs job
Enhance the current jdbchdfs job in spring-xd to have an incremental load / delta load feature similar to sqoop. See sqoop documentation [here|http://sqoop.apache.org/docs/1.4.5/SqoopUserGuide.html#_incremental_imports]. The job will need to maintain some state between executions in order to decide the start point for the next data load. The jdbchdfs job definition could take the following 2 new options. h5. checkColumn optional Specifies the column to be examined when determining which rows to import. (the column should not be of type CHAR/NCHAR/VARCHAR/VARNCHAR/ LONGVARCHAR/LONGNVARCHAR). Column should be numeric or timestamp. h5. lastValue optional If specified this will override any data saved from previous job runs. If not specified will take the saved max-value from the last job run. If no last job run data is available then it will not be an incremental load and all the data which satisfies the query will be used. Sqoop provides 2 modes of operation for incremental load, 'append' and 'lastModified'. For jdbchdfs the job will always append as it is writing to a hdfs file. Example: To import data from the database table some_table which has a last update column called lastUpdated, you could use. {code} xd:> job create myjob --definition "jdbchdfs --sql='select col1,col2,col3 from some_table' --checkColumn=lastUpdated" --deploy {code} The batch job should also be capable of being partitioned to run in parallel across multiple containers
Story points: 5