Bug: Getting NullPointerException when the unauthorised user tries to perform the key operation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Aug/24 13:42

While validating the tomcat 9.x in apache Ranger when user doesn't have appropriate permission in Ranger policies we faced the NPE for key operation using hadoop cmd.

*Problem :*

_Functionally -_ We are facing the NPE while performing key operations from hadoop cmd with the user not having permission in policy on cluster with tomcat v9.x. However with curl to Ranger KSM Server is working as expected.

_Technically -_ Getting response message as null on client side in hadoop-common at [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L565]

*E.G.*

_with Ranger KMS tomcat v9.x_
{code:java}
 hadoop key list
The list subcommand displays the keynames contained within
a particular provider as configured in core-site.xml or
specified with the -provider argument. -metadata displays
the metadata. If -strict is supplied, fail immediately if
the provider requires a password and none is given.
Exception in thread "main" java.lang.NullPointerException
	at org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
	at org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
	at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
	at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
_on_ _Ranger KMS_ _tomcat v8.5.x_
{code:java}
hadoop key list
The list subcommand displays the keynames contained within
a particular provider as configured in core-site.xml or
specified with the -provider argument. -metadata displays
the metadata. If -strict is supplied, fail immediately ifthe provider requires a password and none is given.
Executing command failed with the following exception: AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS'{code}
*Debug logs on Ranger KMS Server side*

1.) Added logs in [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java] in method _createResponse()_ and _toResponse()_ where we are generating response to send it to client i.e. _hadoop-common_
Logs are exactly same on both the tomcat scenario. Refer below the added logs, detailed logs will be available in ranger kms log file on cluster. 
{code:java}
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== Entered into toResponse =========
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== exception =========org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS'
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== exception.getClass() =========class org.apache.hadoop.security.authorize.AuthorizationException
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== AuthorizationException =========
2024-07-25 11:35:51,452 WARN  org.apache.hadoop.crypto.key.kms.server.KMS: [https-jsse-nio-9494-exec-2]: User systest@ROOT.COMOPS.SITE (auth:KERBEROS) request GET https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names caused exception.
org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS'
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ===== Entered into createResponse ======
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== status ======= Forbidden
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ======= ex ======= org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS'
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ======= ex.getStackTrace() ======= [Ljava.lang.StackTraceElement;@3e75ae9d
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ======= ex.getMessage() ======= User:systest not allowed to do 'GET_KEYS'
2024-07-25 11:35:51,452 INFO  org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ======= ex.toString() ======= org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS'  {code}
2.) Also added logs in [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]

Adding logs in code base
{code:java}
public void setStatus(int sc, String sm) {
        LOG.info("========= setStatus with message============ ");
      statusCode = sc;
      msg = sm;
        LOG.info("========= sc ============ " +sc);
        LOG.info("========= msg ============ " +msg);

            if(sc == 403) {
                LOG.info("===== its 403 ====");
                super.setStatus(sc, sm);
            } else{
                super.setStatus(sc, sm);
            }
    } {code}
LOGS:
{code:java}
2024-07-25 11:35:51,460 INFO  org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter: [https-jsse-nio-9494-exec-2]: ========= setStatus with message============
2024-07-25 11:35:51,460 INFO  org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter: [https-jsse-nio-9494-exec-2]: ========= sc ============ 403
2024-07-25 11:35:51,460 INFO  org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter: [https-jsse-nio-9494-exec-2]: ========= msg ============ Forbidden
2024-07-25 11:35:51,460 INFO  org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter: [https-jsse-nio-9494-exec-2]: ===== its 403 ==== {code}
This explains that the KMS server is sending the code and message appropriately.

*Debug logs on Hadoop Common Client side*
1.) Added logs in [HttpExceptionUtils.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java] to make sure whether appropriate response is received.

Logs will be available in ranger kms log file.
{code:java}
2024-07-25 11:35:51,453 INFO  org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ====== Entered into createJerseyExceptionResponse ====
2024-07-25 11:35:51,453 INFO  org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== ex ========   org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS'
2024-07-25 11:35:51,454 INFO  org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== ex.getMessage ========   User:systest not allowed to do 'GET_KEYS'
2024-07-25 11:35:51,454 INFO  org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== status ========   Forbidden
2024-07-25 11:35:51,454 INFO  org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== status.getStatusCode ========   403
2024-07-25 11:35:51,454 INFO  org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== status.getReasonPhrase ========   Forbidden
2024-07-25 11:35:51,454 INFO  org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]:  =======  response  ======== com.sun.jersey.core.spi.factory.ResponseImpl@5bd8a59b  {code}
2.) Added logs exactly before NPE occurs  i.e. [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L564]
Adding logs in code base


LOG.info(" =========== conn ======== " + conn);
Map<String, List<String>> map = conn.getHeaderFields();
LOG.info("======= map ======== " + map);for (Map.Entry<String, List<String>> entry : map.entrySet()) {
  LOG.info("=============== " + "Key : " + entry.getKey() +          " ,Value : " + entry.getValue());
}
LOG.info(" =========== conn.getResponseMessage ======== " + conn.getResponseMessage());
LOG.info(" =========== conn.getResponseCode ======== " + conn.getResponseCode());if ((conn.getResponseCode() == HttpURLConnection.HTTP_FORBIDDEN
    && (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED) ||
        conn.getResponseMessage().contains(INVALID_SIGNATURE)))
    || conn.getResponseCode() == HttpURLConnection.HTTP_UNAUTHORIZED) { 

LOGS: This logs gets printed on terminal where we execute hadoop cmd .

_with Ranger KMS tomcat v9.x_


{code:java}
hadoop key list
24/07/25 11:38:15 INFO kms.KMSClientProvider: ======== Entered into call ========
24/07/25 11:38:15 INFO kms.KMSClientProvider:  =========== conn ======== sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
24/07/25 11:38:15 INFO kms.KMSClientProvider: ======= map ======== {Keep-Alive=[timeout=60], null=[HTTP/1.1 403], Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload], Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220], Date=[Thu, 25 Jul 2024 11:38:15 GMT], Content-Type=[application/json]}
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Keep-Alive ,Value : [timeout=60]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : null ,Value : [HTTP/1.1 403]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains; preload]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Server ,Value : [Apache Ranger]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Connection ,Value : [keep-alive]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Content-Length ,Value : [220]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Date ,Value : [Thu, 25 Jul 2024 11:38:15 GMT]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Content-Type ,Value : [application/json]
24/07/25 11:38:15 INFO kms.KMSClientProvider:  =========== conn.getResponseMessage ======== null
24/07/25 11:38:15 INFO kms.KMSClientProvider:  =========== conn.getResponseCode ======== 403
list [-provider <provider>] [-strict] [-metadata] [-help]:


The list subcommand displays the keynames contained within
a particular provider as configured in core-site.xml or
specified with the -provider argument. -metadata displays
the metadata. If -strict is supplied, fail immediately if
the provider requires a password and none is given.
Exception in thread "main" java.lang.NullPointerException
	at org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
	at org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
	at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
	at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}

_with Ranger KMS tomcat v8.5.x_
hadoop key list
24/07/25 11:02:25 INFO kms.KMSClientProvider: ======== Entered into call ========
24/07/25 11:02:25 INFO kms.KMSClientProvider:  =========== conn ======== sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ds-tomcat-test1.root.comops.site:9494/kms/v1/keys/names24/07/25 11:02:25 INFO kms.KMSClientProvider: ======= map ======== {Keep-Alive=[timeout=60], null=[HTTP/1.1 403 Forbidden], Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload], Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220], Date=[Thu, 25 Jul 2024 11:02:25 GMT], Content-Type=[application/json]}
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Keep-Alive ,Value : [timeout=60]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : null ,Value : [HTTP/1.1 403 Forbidden]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains; preload]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Server ,Value : [Apache Ranger]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Connection ,Value : [keep-alive]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Content-Length ,Value : [220]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Date ,Value : [Thu, 25 Jul 2024 11:02:25 GMT]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Content-Type ,Value : [application/json]
24/07/25 11:02:25 INFO kms.KMSClientProvider:  =========== conn.getResponseMessage ======== Forbidden
24/07/25 11:02:25 INFO kms.KMSClientProvider:  =========== conn.getResponseCode ======== 403
Cannot list keys for KeyProvider: org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@209da20d
list [-provider <provider>] [-strict] [-metadata] [-help]:


The list subcommand displays the keynames contained within
a particular provider as configured in core-site.xml or
specified with the -provider argument. -metadata displays
the metadata. If -strict is supplied, fail immediately ifthe provider requires a password and none is given.
Executing command failed with the following exception: AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS' 

Please notice 
_with tomcat v9.x : *Key : null ,Value : [HTTP/1.1 403]*_ 
_with. tomcat v8.5.x : *Key : null ,Value : [HTTP/1.1 403 Forbidden]*_

**

Message "Forbidden" is not present with tomcat v9.x.

It seems that tomcat v9.x is not setting the message and hadoop-common is trying to get where we are facing  NPE.



Also checked for _*org.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER*_ but its not available in tomcat 9.x
Ref:
Tomcat Doc for 8.5.x [https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/coyote/Constants.html#USE_CUSTOM_STATUS_MSG_IN_HEADER]

Tomcat Doc for 9.x [https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/coyote/Constants.html]

Thanks