Project: springxd
Issue XD-3132: Sqoop job doesn't run when deployed on YARN on Ambari deployed HDP
Got this error when submitting Sqoop job: {code} 2015-06-01 19:09:42,932 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:main(75)) - Sqoop command: import 2015-06-01 19:09:42,939 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:main(76)) - Using args: [--table, XD_JOB_REGISTRY, --target-dir, /xd/sqoop2, -m=1] 2015-06-01 19:09:42,939 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:main(77)) - Mapreduce home: /usr/hdp/2.2.4.2-2/hadoop-mapreduce 2015-06-01 19:09:42,977 WARN [main] conf.Configuration (Configuration.java:(646)) - DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively 2015-06-01 19:09:42,984 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: fs.defaultFS=hdfs://hawaii:8020 2015-06-01 19:09:43,743 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: yarn.resourcemanager.hostname=hawaii 2015-06-01 19:09:43,758 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: yarn.resourcemanager.address=hawaii:8050 2015-06-01 19:09:43,758 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: yarn.resourcemanager.scheduler.address=hawaii 2015-06-01 19:09:43,758 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: mapreduce.framework.name=yarn 2015-06-01 19:09:43,758 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: mapreduce.jobhistory.address=hawaii 2015-06-01 19:09:43,760 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:createConfiguration(158)) - Setting configuration property: yarn.resourcemanager.scheduler.address=hawaii:8030 2015-06-01 19:09:43,760 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:createConfiguration(158)) - Setting configuration property: mapreduce.application.classpath=$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.2.4.2-2/hadoop/lib/hadoop-lzo-0.6.0.2.2.4.2-2.jar:/etc/hadoop/conf/secure 2015-06-01 19:09:43,760 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:createConfiguration(158)) - Setting configuration property: mapreduce.application.framework.path=/hdp/apps/2.2.4.2-2/mapreduce/mapreduce.tar.gz#mr-framework 2015-06-01 19:09:44,141 INFO [main] sqoop.Sqoop (Sqoop.java:(92)) - Running Sqoop version: 1.4.5 2015-06-01 19:09:44,691 INFO [main] manager.SqlManager (SqlManager.java:initOptionDefaults(98)) - Using default fetchSize of 1000 2015-06-01 19:09:44,691 INFO [main] tool.CodeGenTool (CodeGenTool.java:generateORM(92)) - Beginning code generation 2015-06-01 19:09:45,057 INFO [main] manager.SqlManager (SqlManager.java:execute(749)) - Executing SQL statement: SELECT t.* FROM XD_JOB_REGISTRY AS t WHERE 1=0 2015-06-01 19:09:45,074 INFO [main] manager.SqlManager (SqlManager.java:execute(749)) - Executing SQL statement: SELECT t.* FROM XD_JOB_REGISTRY AS t WHERE 1=0 2015-06-01 19:09:45,148 INFO [main] orm.CompilationManager (CompilationManager.java:findHadoopJars(94)) - HADOOP_MAPRED_HOME is /usr/hdp/2.2.4.2-2/hadoop-mapreduce 2015-06-01 19:09:45,230 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(184)) - It seems as though you are running sqoop with a JRE. 2015-06-01 19:09:45,230 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(185)) - Sqoop requires a JDK that can compile Java code. 2015-06-01 19:09:45,230 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(186)) - Please install a JDK and set $JAVA_HOME to use it. 2015-06-01 19:09:45,232 ERROR [main] tool.ImportTool (ImportTool.java:run(609)) - Encountered IOException running import job: java.io.IOException: Could not start Java compiler. at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:187) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.springframework.xd.sqoop.SqoopRunner.main(SqoopRunner.java:87) {code}
Story points: 3