Project: springxd
Issue XD-3001: Unable to call Sqoop Job commands other than --create from within Spring-Xd Job
Running a (SQOOP Job --create) from within (Spring-XD JOB create) statement runs successfully.Unable to run the (Sqoop Job --exec) from within (Spring-XD Job) CREATE DEFINITION: job create sqoop_lookup_d_job_v1 --definition "sqoop --command=job --args='--create lookupd_job --meta-connect \"jdbc:postgresql://xx.xxx.xx.xx:5433/bdl?user=xxxxx&password=xxxxxx\" -- import --connect jdbc:oracle:thin:@xxxxxx.com:xxxx/GDWP --username xxxxx --password xxxxx --query \"SELECT ROW_WID, LKUP_ID, TRIM(TRANSLATE(LKUP_CD, CHR(10)||CHR(13), ''-'')) AS LKUP_CD, TRIM(TRANSLATE(LKUP_DESC, CHR(10)||CHR(13), ''-'')) AS LKUP_DESC, TRIM(TRANSLATE(LKUP_TYPE, CHR(10)||CHR(13), ''-'')) AS LKUP_TYPE, TRIM(TRANSLATE(LKUP_VAL, CHR(10)||CHR(13), ''-'')) AS LKUP_VAL, SRC_STRT_DT, SRC_END_DT, W_INSERT_DT, W_UPDATE_DT FROM MASTERDATA.W_LOOKUP_D WHERE 1=1 AND $CONDITIONS\" --incremental append --check-column ROW_WID --last-value 10150 --target-dir /user/monasj1/bdl/data/sqoop/jobtest --num-mappers 1 --fields-terminated-by \"|\" --append'" Validated job definition details stored in PostgreSQL repository Trying to run EXECUTE In Same fashion. EXECUTION DEFINITION: job create sqoop_lookup_d_exec_v1 --definition "sqoop --command=job --args='--exec lookupd_job --meta-connect \"jdbc:postgresql://xx.xxx.xx.xx:5433/bdl?user=xxxxx&password=xxxxx\"'" Stack Trace: batch.stepType	org.springframework.batch.core.step.tasklet.TaskletStep batch.taskletType	org.springframework.xd.sqoop.SqoopTasklet sqoop.command	job --exec lookupd_job --meta-connect "jdbc:postgresql://xx.xxx.xx.xx:5433/bdl?user=xxxxxx&password=xxxxxxx" -- exec sqoop.errors	No job operation specified Try --help for usage instructions. Exception in thread "main" java.lang.RuntimeException: Sqoop failed - return code 1 at org.springframework.xd.sqoop.SqoopRunner.main(SqoopRunner.java:81) sqoop.log	16:01:45,668 INFO main sqoop.SqoopRunner - Sqoop command: job 16:01:45,668 INFO main sqoop.SqoopRunner - Using args: [--exec, lookupd_job, --meta-connect, "jdbc:postgresql://xx.xxx.xx.xx:5433/bdl?user=xxxxxxx&password=xxxxx"] 16:01:45,668 INFO main sqoop.SqoopRunner - Mapreduce home: /opt/pivotal/spring-xd-1.1.1.RELEASE/xd/lib/phd21 16:01:45,679 INFO main sqoop.SqoopRunner - Setting configuration property: fs.defaultFS=hdfs://xxxxxxxxxxxxx.com:8020 16:01:45,759 INFO main sqoop.SqoopRunner - Setting configuration property: yarn.resourcemanager.address=xxxxxxxxxxxx.com:8032 16:01:45,759 INFO main sqoop.SqoopRunner - Setting configuration property: yarn.application.classpath=$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,$USS_HOME/*,$USS_CONF 16:01:45,759 INFO main sqoop.SqoopRunner - Setting configuration property: mapreduce.framework.name=yarn 16:01:45,817 WARN main tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 16:01:45,863 INFO main sqoop.Sqoop - Running Sqoop version: 1.4.5 I have tested --list, --exec,--delete they are all erroring same way. Looking at SqoopRunner not sure what makes --create any different from other saved-job functionality. Please let me know how i can call or execute save job definition's through spring-xd Sqoop tasklet
Story points: 5