Bug: Fix ProcessEnvironment ClassCastException in Shell.java
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/24 07:09
Affects Version/s: 3.4.0

We tried to upgrade Hadoop version from 3.6.6 to 3.4.0 in Apache Hive HIVE-28191. But found exception:
{code:java}
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.ProcessEnvironment$Variable
    at java.lang.ProcessEnvironment$StringEnvironment.toEnvironmentBlock(ProcessEnvironment.java:273) ~[?:1.8.0_221]
    at java.lang.ProcessEnvironment.toEnvironmentBlock(ProcessEnvironment.java:298) ~[?:1.8.0_221]
    at java.lang.ProcessImpl.start(ProcessImpl.java:86) ~[?:1.8.0_221]
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_221]
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:998) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hadoop.util.Shell.run(Shell.java:959) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1282) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:1377) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:1359) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1535) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO(RawLocalFileSystem.java:1000) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:991) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:952) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hadoop.hive.ql.exec.Utilities.ensurePathIsWritable(Utilities.java:4954) ~[classes/:?]
    at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:843) ~[classes/:?]
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:784) ~[classes/:?]
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:708) ~[classes/:?]
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:669) ~[classes/:?]
    at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:182) ~[classes/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_221]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_221]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_221]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_221]
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[classes/:?]
    at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[classes/:?]
    at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[classes/:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_221]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) ~[hadoop-common-3.4.0.jar:?]
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[classes/:?]
    at com.sun.proxy.$Proxy58.open(Unknown Source) ~[?:?]
    at org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485) ~[classes/:?]
    ... 40 more  {code}
 

After some debugging, i found the  failed  hive tests {{TestRemoteHiveMetastoreWithHttpJwt#testValidJWT}} and {{TestHttpJwtAuthentication#testAuthorizedUser}} are related Hadoop3.4.0 change [HADOOP-17009: Embrace Immutability of Java Collections hadoop#1974|https://github.com/apache/hadoop/pull/1974]
[https://github.com/apache/hadoop/pull/1974/files#diff-372a0d25bcccd88b409a8149949628abd7d3472a1798bebe813e7617b0ef73c7L918-L920]