Project: springxd
Issue XD-3611: SqoopTasklet doesn't use keytab configuration from hadoop.properties file
Hey Guys, I have been using the sqoopTasklet for a while but I found an unexpected problem. Basically I'm not able to configure kerberos authentication from hadoop.properties file as follow:

import --connect otheroptions

hadoop.properties file: fs.defaultFS=hdfs://hdfshost:8020 yarn.resourcemanager.hostname=host001 yarn.resourcemanager.address=host001:8032 yarn.resourcemanager.scheduler.address=host001:8030 mapreduce.jobhistory.address=host003:10020 yarn.application.classpath=$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/* mapreduce.framework.name=yarn spring.hadoop.security.authMethod=kerberos spring.hadoop.security.userPrincipal=user1@COMPANY.COM spring.hadoop.security.userKeytab=/home/user1/user1.keytab spring.hadoop.security.namenodePrincipal=hdfs/_HOST@COMPANY.COM spring.hadoop.security.rmManagerPrincipal=yarn/_HOST@COMPANY.COM spring.hadoop.security.jobHistoryPrincipal=mapred/_HOST@COMPANY.COM or fs.defaultFS=hdfs://hdfshost:8020 yarn.resourcemanager.hostname=host001 yarn.resourcemanager.address=host001:8032 yarn.resourcemanager.scheduler.address=host001:8030 mapreduce.jobhistory.address=host003:10020 yarn.application.classpath=$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/* mapreduce.framework.name=yarn security.authMethod=kerberos security.userPrincipal=user1@COMPANY.COM security.userKeytab=/home/user1/user1.keytab security.namenodePrincipal=hdfs/_HOST@COMPANY.COM security.rmManagerPrincipal=yarn/_HOST@COMPANY.COM security.jobHistoryPrincipal=mapred/_HOST@COMPANY.COM Running the job I'm getting the following error: Encountered IOException running import job: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS So it means the sqooptasklet isn't setting the kerberos authentication, this basically because in the SqoopTasklet class is adding some prefix to the configurations (SPRING_HADOOP_CONFIG_PREFIX) https://github.com/spring-projects/spring-xd/blob/master/extensions/spring-xd-extension-sqoop/src/main/java/org/springframework/xd/sqoop/SqoopTasklet.java Really doesn't make sense for me add those prefix and remove it later in the next call in the SqoopRunner class. https://github.com/spring-projects/spring-xd/blob/master/extensions/spring-xd-extension-sqoop/src/main/java/org/springframework/xd/sqoop/SqoopRunner.java If I inject the security.* configurations directly to the list arguments it works. I'm sure you guys have a good reason to add the prefix but I don't see why. Unfortunately is annoying when you are developing in a local VM where you can test the simple authentication and after move the job to dev/prod environments with kerberos auth, the above because you must change your sqooptasklet configuration injecting the new parameters. If the SqoopTasklet allows inject those parameters directly from the hadoop.properties file you don't need change tasklet configurations to run your jobs with different authentication methods. Thanks in advance. Héctor
Story points: 1