Bug: InvalidProtocolBufferException caused by JDK 11 < 11.0.18 AES-CTR cipher state corruption with AVX-512 bug
Status: Open | Priority: Major | Resolution: 
Created: 12/Apr/23 19:43

This serves as a PSA for a JDK bug. Not really a bug in Hadoop / HDFS. Symptom/Workaround/Solution detailed below.

[~relek] identified [JDK-8292158|https://bugs.openjdk.org/browse/JDK-8292158] (backported to JDK 11 in [JDK-8295297|https://bugs.openjdk.org/browse/JDK-8295297]) causes HDFS clients to fail with InvalidProtocolBufferException due to corrupted protobuf message in Hadoop RPC request when all of the below conditions are met:

1. The host is capable of AVX-512 instruction sets
2. AVX-512 is enabled in JVM. This should be enabled by default on AVX-512 capable hosts, equivalent to specifying JVM arg {{-XX:UseAVX=3}}
3. Hadoop native library (e.g. libhadoop.so) is not available. So the HDFS client falls back using Hotspot JVM's {{aesctr_encrypt}} implementation for AES/CTR/NoPadding.
4. Client uses JDK 11. And OpenJDK version < 11.0.18

As a result, the client could print messages like these:

{code:title=Symptoms on the HDFS client}
2023-02-21 15:21:44,380 WARN org.apache.hadoop.hdfs.DFSClient: Connection failure: Failed to connect to <HOST/IP:PORT> for file /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2023_02_21-15_21_25.b6788e89894a61b5 for block BP-1836197545-10.125.248.11-1672668423261:blk_1073935111_194857:com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.
com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.

2023-02-21 15:21:44,378 WARN org.apache.hadoop.hdfs.DFSClient: Connection failure: Failed to connect to <HOST/IP:PORT> for file /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2023_02_21-15_21_25.b6788e89894a61b5 for block BP-1836197545-<IP>-1672668423261:blk_1073935111_194857:com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.

2023-02-21 15:06:55,530 WARN org.apache.hadoop.hdfs.DFSClient: Connection failure: Failed to connect to <HOST/IP:PORT> for file /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2023_02_21-15_06_55.b4a633a8bde014aa for block BP-1836197545-<IP>-1672668423261:blk_1073935025_194771:com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.
com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.
{code}

The error message might mislead devs/users into thinking this is a Hadoop Common or HDFS bug (while it is a JDK bug in this case).


{color:red}Solutions:{color}
1. As a workaround, append {{-XX:UseAVX=2}} to client JVM args; or
2. Upgrade to OpenJDK >= 11.0.18.


I might post a repro test case for this, or find a way in the code to prompt the user that this could be the potential issue (need to upgrade JDK 11) when it occurs.