Bug: SparkThriftServer can not start when HMS use DBTokenStore
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/21 11:36
Affects Version/s: 3.1.1

CMD:
./sbin/start-thriftserver.sh --master=local --conf spark.sql.hive.metastore.version=3.1.2 --conf spark.sql.hive.metastore.jars=path --conf spark.sql.hive.metastore.jars.path="file:///tmp/metastorejars/*.jar"


ERROR LOG:
21/06/24 16:43:09 ERROR HiveThriftServer2: Error starting HiveThriftServer2
org.apache.hive.service.ServiceException: Failed to Start HiveServer2
        at org.apache.hive.service.CompositeService.start(CompositeService.java:80)
        at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:105)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.start(HiveThriftServer2.scala:154)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.startWithContext(HiveThriftServer2.scala:64)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:104)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hive.service.ServiceException: Error initializing ThriftBinaryCLIService
        at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.initializeServer(ThriftBinaryCLIService.java:113)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.start(ThriftCLIService.java:177)
        at org.apache.hive.service.CompositeService.start(CompositeService.java:70)
        ... 17 more
Caused by: org.apache.thrift.transport.TTransportException: Failed to start token manager
        at org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:167)
        at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.initializeServer(ThriftBinaryCLIService.java:66)
        ... 19 more
Caused by: java.io.IOException: Failed to initialize master key
        at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213)
        at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:97)
        at org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:162)
        ... 20 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:211)
        ... 22 more
Caused by: java.lang.ClassCastException: com.sun.proxy.$Proxy45 cannot be cast to java.lang.Class
        at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:154)
        at org.apache.hadoop.hive.thrift.DBTokenStore.addMasterKey(DBTokenStore.java:43)
        at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:193)
        at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:364)
        ... 27 more