Project: springxd
Issue XD-3025: SpringXD sqoop module is hanging
The SpringXD Sqoop module is in execution status until it times out, it is hanging. The container logs show: 2015-05-04 15:15:45,365 1.2.0.M1 INFO DeploymentsPathChildrenCache-0 container.DeploymentListener - Deploying job 'sqoop_lookup' 2015-05-04 15:15:45,536 1.2.0.M1 INFO DeploymentsPathChildrenCache-0 container.DeploymentListener - Deploying module [ModuleDescriptor@239b037a moduleName = 'sqoop', moduleLabel = 'sqoop', group = 'sqoop_lookup', sourceChannelName = [null], sinkChannelName = [null], index = 0, type = job, parameters = map['args' -> '--connect=jdbc:oracle:thin:@************:****/******* —username=******** --password-file=/user/zeybeb/workspace/secure-files/gdw.password --table=MASTERDATA.W_LOOKUP_D --target-dir=/user/zeybeb/workspace/ent/masterdata_src/lookup_d -m 1', 'command' -> 'import'], children = list[[empty]]] 2015-05-04 15:16:17,061 1.2.0.M1 INFO inbound.job:sqoop_lookup-redis:queue-inbound-channel-adapter1 sqoop.SqoopTasklet - Sqoop system.out: /tmp/Sqoop-948322291323951735.out The /tmp/Sqoop-948322291323951735.out file content is: 15:16:17,612 INFO main sqoop.SqoopRunner - Sqoop command: import 15:16:17,613 INFO main sqoop.SqoopRunner - Using args: [--connect=jdbc:oracle:thin:@************:****/*******, —username=*********, --password-file=/user/zeybeb/workspace/secure-files/gdw.password, --table=MASTERDATA.W_LOOKUP_D, --target-dir=/user/zeybeb/workspace/ent/masterdata_src/lookup_d, -m, 1] 15:16:17,613 INFO main sqoop.SqoopRunner - Mapreduce home: /opt/pivotal/spring-xd-1.2.0.M1/xd/lib/phd21 15:16:17,631 INFO main sqoop.SqoopRunner - Setting configuration property: fs.defaultFS=hdfs://ilabphd07.isus.emc.com:8020 15:16:17,753 INFO main sqoop.SqoopRunner - Setting configuration property: yarn.resourcemanager.address=ilabphd08.isus.emc.com:8050 15:16:17,753 INFO main sqoop.SqoopRunner - Setting configuration property: yarn.application.classpath=/usr/hdp/2.2.0.0-2041/etc/hadoop/conf.empty,/usr/hdp/2.2.0.0-2041/hadoop/*,/usr/hdp/2.2.0.0-2041/hadoop/lib/*,/usr/hdp/2.2.0.0-2041/hadoop-hdfs/*,/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/*,/usr/hdp/2.2.0.0-2041/hadoop-yarn/*,/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/*,/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/*,/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/*,/usr/hdp/2.2.0.0-2041/sqoop/*,/usr/hdp/2.2.0.0-2041/sqoop/lib/*,/usr/hdp/2.2.0.0-2041/flume/*,/usr/hdp/2.2.0.0-2041/flume/lib/*,/usr/hdp/2.2.0.0-2041/storm/*,/usr/hdp/2.2.0.0-2041/storm/lib/* 15:16:17,754 INFO main sqoop.SqoopRunner - Setting configuration property: mapreduce.framework.name=yarn 15:16:17,837 WARN main tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 15:16:17,907 INFO main sqoop.Sqoop - Running Sqoop version: 1.4.5 15:16:18,282 WARN main util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15:16:19,552 WARN main sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 15:16:19,657 INFO main oracle.OraOopManagerFactory - Data Connector for Oracle and Hadoop is disabled. 15:16:19,673 INFO main manager.SqlManager - Using default fetchSize of 1000 15:16:19,673 INFO main tool.CodeGenTool - Beginning code generation 15:16:20,639 INFO main manager.OracleManager - Time zone has been set to GMT 15:16:20,853 INFO main manager.SqlManager - Executing SQL statement: SELECT t.* FROM MASTERDATA.W_LOOKUP_D t WHERE 1=0 15:16:21,018 INFO main orm.CompilationManager - HADOOP_MAPRED_HOME is /opt/pivotal/spring-xd-1.2.0.M1/xd/lib/phd21 15:16:23,171 INFO main orm.CompilationManager - Writing jar file: /tmp/sqoop-spring-xd/compile/4e11123a52fa36d6677efdb47bcdc43b/MASTERDATA.W_LOOKUP_D.jar 15:16:23,191 INFO main manager.OracleManager - Time zone has been set to GMT 15:16:24,109 INFO main manager.OracleManager - Time zone has been set to GMT 15:16:24,825 INFO main mapreduce.ImportJobBase - Beginning import of MASTERDATA.W_LOOKUP_D 15:16:24,848 INFO main manager.OracleManager - Time zone has been set to GMT 15:16:24,876 WARN main mapreduce.JobBase - SQOOP_HOME is unset. May not be able to find all job dependencies. 15:16:25,083 INFO main client.RMProxy - Connecting to ResourceManager at ilabphd08.isus.emc.com/10.15.232.191:8050 15:16:25,977 INFO main db.DBInputFormat - Using read commited transaction isolation 15:16:26,117 INFO main mapreduce.JobSubmitter - number of splits:1 15:16:26,361 INFO main mapreduce.JobSubmitter - Submitting tokens for job: job_1429280992648_0019 15:16:26,717 INFO main impl.YarnClientImpl - Submitted application application_1429280992648_0019 to ResourceManager at ilabphd08.isus.emc.com/10.15.232.191:8050 15:16:26,782 INFO main mapreduce.Job - The url to track the job: http://http://ilabphd08.isus.emc.com:8088/proxy/application_1429280992648_0019/ 15:16:26,783 INFO main mapreduce.Job - Running job: job_1429280992648_0019 The logs on the Hadoop side are: Showing 4096 bytes. Click here for full log mumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:04,026 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:05,033 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:06,042 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:07,049 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:08,056 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:09,062 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:10,069 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:41,093 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:42,100 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:43,107 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:44,113 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:45,120 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:46,129 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:47,136 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:48,143 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:49,150 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2015-05-04 15:36:50,156 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) Even though the maxRetries is set to 10 the process is going into several sets of retrials.
Story points: 5