av9ash/GitBugs huggingface dataset:
Bug: s3a: TestS3AAWSCredentialsProvider and TestS3AInputStreamRetry really slow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jun/24 18:24
Affects Version/s: 3.4.0

Not noticed this before, but the unit tests TestS3AAWSCredentialsProvider and TestS3AInputStreamRetry are so slow they will be hurting over all test performance times: no integration tests will start until these are all complete.


{code}

mvn test -T 1C -Dparallel-tests

...
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.877 s - in org.apache.hadoop.fs.s3a.TestS3AInputStreamRetry
...
[INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.038 s - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
{code}

The PR cuts total execution time of a 10 thread test run from 3 minutes to 2:30
Bug: Hadoop release contains a 530MB bundle-2.23.19.jar
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 17/Jun/24 11:24

The size of Hadoop binary release (v3.4.0) is 1.7 GB.
{code:java}
hadoop-3.4.0$du -h -d 1
$du -h -d 1 .
2.0M	./bin
260K	./libexec
 72K	./include
212K	./sbin
184K	./etc
232K	./licenses-binary
316M	./lib
1.4G	./share
1.7G	.
{code}
A large component is bundle-2.23.19.jar, which is [AWS Java SDK :: Bundle|https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.23.19]
{code:java}
hadoop-3.4.0$ls -lh share/hadoop/tools/lib/bundle-2.23.19.jar  
-rw-r--r--@ 1 szetszwo  staff   530M Mar  4 15:41 share/hadoop/tools/lib/bundle-2.23.19.jar
{code}
We should revisit if such a large jar is really needed to be included in the release.
Bug: WrappedIO BulkDelete API to raise IOEs as UncheckedIOExceptions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jun/24 10:29
Affects Version/s: 3.4.1


It's easier to invoke methods through reflection through parquet/iceberg DynMethods if the invoked method raises unchecked exceptions, because it doesn't then rewrape the raised exception in a generic RuntimeException

Catching the IOEs and wrapping as UncheckedIOEs makes it much easier to unwrap IOEs after the invocation
Bug: [ABFS]Prevent ABFS initialization for non-hierarchical-namespace account if Customer-provided-key configs given.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/24 10:00
Affects Version/s: 3.4.0

Store doesn't flow in the namespace information to the client. 

In https://github.com/apache/hadoop/pull/6221, getIsNamespaceEnabled is added in client methods which checks if namespace information is there or not, and if not there, it will make getAcl call and set the field. Once the field is set, it would be used in future getIsNamespaceEnabled method calls for a given AbfsClient.

Since, CPK both global and encryptionContext are only for hns account, the fix that is proposed is that we would fail fs init if its non-hns account and cpk config is given.
Bug: Reduce the number of headObject when opening a file with the s3 file system
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 08/Jun/24 12:20
Affects Version/s: 3.3.6

In the implementation of the S3 filesystem, of the hadoop aws package, if you use it with spark, every time you open a file for anything you will have to send two Head Objects, since to open the file, you will first look to see if this file exists, executing a HeadObject, and then when opening it, the implementation, both of sdk1 and sdk2, forces you to make a head object again. This is not the fault of the implementation of this class (S3AFileSystem), but of the abstract FileSystem class of the Hadoop core, since it does not allow the FileStatus to be passed but only allows the use of Path.

If the FileSystem implementation is changed, it could be used to not have to request that HeadObject again.
Bug: SecretManager should not hardcode HMAC algorithm
Status: Open | Priority: Major | Resolution: 
Created: 07/Jun/24 14:34

{code}
//SecretManager.java
private static final String DEFAULT_HMAC_ALGORITHM = "HmacSHA1";
{code}
See https://github.com/apache/hadoop/blob/10df59e4210206508da648d5676f1c7d423b0353/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java#L114
Bug: Change loglevel to ERROR/WARNING so that it would easy to identify the problem without ignoring
Status: Open | Priority: Major | Resolution: 
Created: 24/May/24 10:58

On the new Host with Java version 11, the DN was not able to communicate with the NN. We enabled DEBUG logging for the DN and the below message was logged under DEBUG level.

DEBUG org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException as:hdfs/av3l704p.bigdata.it.internal@PRODUCTION.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcept
ion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Receive timed out)]

Without a DEBUG level logging, this was shown up as a WARNING as below

WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for hdfs/av3l704p.bigdata.it.internal@PRODUCTION.LOCAL to avl2785p.bigdata.it.internal/172.24.178.32:8022
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Receive timed out)]

A considerable amount of time was spent troubleshooting this issue as this exception was moved to a DEBUG level which was difficult to track in the logs.

Can we have such critical WARNINGs shown up at the WARN/ERROR level so that it's not missed when we enable DEBUG level logging for datanodes?
Bug: ZooKeeper based state stores use different ZK address configs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/24 10:02

Currently, the Zookeeper-based state stores of RM, YARN Federation, and HDFS Federation use the same ZK address config `{{{}hadoop.zk.address`{}}}. But in our production environment, we hope that different services can use different ZKs to avoid mutual influence.

This jira adds separate ZK address configs for each service.
Bug: Upgrade Kafka Clients due to CVEs
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/May/24 11:32

Upgrade Kafka Clients due to CVEs

CVE-2023-25194:- Affected versions of this package are vulnerable to Deserialization of Untrusted Data when there are gadgets in the {{{}classpath{}}}. The server will connect to the attacker's LDAP server and deserialize the LDAP response, which the attacker can use to execute java deserialization gadget chains on the Kafka connect server.
CVSS Score:- 8.8(High)
[https://nvd.nist.gov/vuln/detail/CVE-2023-25194] 

CVE-2021-38153

CVE-2018-17196

Insufficient Entropy

[https://security.snyk.io/package/maven/org.apache.kafka:kafka-clients] 

Upgrade Kafka-Clients to 3.4.0 or higher.
Bug: TestS3ACachingBlockManager fails intermittently in Yetus
Status: Open | Priority: Major | Resolution: 
Created: 15/May/24 16:54
Affects Version/s: 3.4.0

{code:java}
[ERROR] org.apache.hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager.testCachingOfGet -- Time elapsed: 60.45 s <<< ERROR!
java.lang.IllegalStateException: waitForCaching: expected: 1, actual: 0, read errors: 0, caching errors: 1
	at org.apache.hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager.waitForCaching(TestS3ACachingBlockManager.java:465)
	at org.apache.hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager.testCachingOfGetHelper(TestS3ACachingBlockManager.java:435)
	at org.apache.hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager.testCachingOfGet(TestS3ACachingBlockManager.java:398)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR] org.apache.hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager.testCachingFailureOfGet
[ERROR]   Run 1: TestS3ACachingBlockManager.testCachingFailureOfGet:405->testCachingOfGetHelper:435->waitForCaching:465 IllegalState waitForCaching: expected: 1, actual: 0, read errors: 0, caching errors: 1
[ERROR]   Run 2: TestS3ACachingBlockManager.testCachingFailureOfGet:405->testCachingOfGetHelper:435->waitForCaching:465 IllegalState waitForCaching: expected: 1, actual: 0, read errors: 0, caching errors: 1
[ERROR]   Run 3: TestS3ACachingBlockManager.testCachingFailureOfGet:405->testCachingOfGetHelper:435->waitForCaching:465 IllegalState waitForCaching: expected: 1, actual: 0, read errors: 0, caching errors: 1 {code}
Discovered in [https://github.com/apache/hadoop/pull/6646#issuecomment-2111558054]
Bug: Fixes compilation issues on Mac
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/24 10:30

When I build hadoop-common native in Mac OS, I found this error:
{code:java}
/xxxxx/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:114:50: error: function-like macro '__GLIBC_PREREQ' is not defined
#if defined(__sun) || defined(__GLIBC_PREREQ) && __GLIBC_PREREQ(2, 32) {code}
The reason is that Mac OS does not support glibc. And C conditional compilation requires validation of all expressions.
Bug: hadoop-auth should not depend on kerb-simplekdc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/24 10:24
Affects Version/s: 3.4.0

HADOOP-16179 attempted to remove dependency on {{kerb-simplekdc}} from {{hadoop-common}}.  However, {{hadoop-auth}} still has a compile-scope dependency on the same, and {{hadoop-common}} proper depends on {{hadoop-auth}}.  So {{kerb-simplekdc}} is still a transitive dependency of {{hadoop-common}}.

{code}
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ hadoop-common ---
[INFO] org.apache.hadoop:hadoop-common:jar:3.5.0-SNAPSHOT
...
[INFO] +- org.apache.hadoop:hadoop-auth:jar:3.5.0-SNAPSHOT:compile
...
[INFO] |  \- org.apache.kerby:kerb-simplekdc:jar:2.0.3:compile
{code}
Bug: Add LzoCodec implementation based on aircompressor
Status: Open | Priority: Major | Resolution: 
Created: 02/May/24 19:50

I remember due to license issue, Hadoop doesn't contain built-in LzoCodec. Users can choose to build and install Lzo codec like hadoop-lzo manually. Some implement LzoCodec based on other open source implementations like aircompressor. But it is somehow inconvenience to maintain it separately.

I'm wondering if we can add LzoCodec implementation based on aircompressor into Hadoop as default LzoCodec.
Bug: S3A: Support ByteBufferPositionedReadable through vector IO
Status: Open | Priority: Major | Resolution: 
Created: 25/Apr/24 17:55
Affects Version/s: 3.4.0

Make it easy for any stream with vector io to support {{ByteBufferPositionedReadable}}

Specifically, {{ByteBufferPositionedReadable.readFully()}}

is exactly a single range read so is easy to read.

the simpler read() call which can return less isn't part of the vector API.
Proposed: invoke the readFully() but convert an EOFException to -1
Bug: Software Architecture Document
Status: Open | Priority: Major | Resolution: 
Created: 06/Apr/24 21:25

We (GitHub @lkhorasandzhian & @vacherkasskiy) have prepared features for documentation. This attached Software Architecture Document is very useful for new contributors and developers to get acquainted with enormous system in a short time. Currently it's only in Russian, but if you're interested in such files we can translate it in English.
There are no changes in code, only adding new documentation files.
Bug: [ABFS] Filesystem contract tests to use methodPath for robust parallel test runs
Status: Open | Priority: Major | Resolution: 
Created: 23/Apr/24 14:32
Affects Version/s: 3.4.0

hadoop-azure supports parallel test runs, but unlike hadoop-aws, the azure ones are parallelised across methods in the same test suites.

this can fail badly where contract tests have hard coded filenames and assume that they can use this across all test cases. Shows up when you are testing on a store with reduced IO capacity triggering retries and making some test cases slower

Fix: hadoop-common contract tests to use methodPath() names
Bug: ABFS: Implement ThreadLocal for ObjectMapper in AzureHttpOperation via config option with static shared instance as an alternative.
Status: Open | Priority: Major | Resolution: 
Created: 16/Apr/24 17:10
Affects Version/s: 3.4.0

While doing internal tests on Hive TPCDS queries we have seen many instances of ObjectMapper have been created in an Application Master thus sharing a thread local object mapper instances will improve the performance.  

 

CC [~stevel@apache.org]  [~harshit.gupta]
Bug: CSE-KMS S3A: Support for InstructionFile to store ECEK meta info
Status: Open | Priority: Major | Resolution: 
Created: 03/Apr/24 11:12

{*}Task{*}: Support for InstructionFile to store ECEK meta info 

*Current implementation/Context:*  

Hadoop-aws supports CSE-KMS. During CSE, key encryption info needs to be kept somewhere. AWS SDK supports two ways:
 # *S3 Object's metadata* : Current integration in haddop-aws only supports this approach.
 ## But S3 metadata has limitation of 2 KB size.
 ## Also, metadata can not be updated independently. It would be complete object read/write operation even if we only need to change the metadata.  
 # *Instruction file approach:* It's a small file containing meta-info in the same bucket at the same location. This approach needs one extra trip to S3 Read/Write operation but could be useful if business needs frequent metadata changes.

*Use case:* to implement KMS RE-ENCRYPT, where only CEK(DEK) needs to be encrypted with new key material. Here instruction file approach could be useful.

Plus there could be many other use cases based on different business needs.

*My analysis:* I tried to enable this by setting *CryptoStorageMode.InstructionFile* in 

CryptoConfigurationV2 while building AmazonS3EncryptionClientV2Builder. 

Note: ObjectMetadata is the default value.

{*}Result{*}: Write operation worked but read failed due to missing instruction file.

*RCA:* On debugging, I found following:

On put request, say myfile.txt : 
 * First , S3AFileSystem writes the file to S3 like *myfile.txt_COPYING_*
 * Second, it writes the corresponding instruction file as  *myfile.txt_COPYING_.instruction*
 * Third, it calls rename.
 ** Rename here means copy the file bytes to *myfile.txt and*
 ** *delete the* *myfile.txt_COPYING*
 * Here problem occurs, 
 ** AmazonS3EncryptionClientV2 class, after deleting any file it looks for corresponding instruction file and if found it deletes that one also. As a result, it deletes *myfile.txt_COPYING_.instruction* as well.

Related  Code:

com.amazonaws.services.s3.AmazonS3EncryptionClientV2.deleteObject() // part of aws sdk bundle

*Possible solution:* S3AFileSystem (part of hadoop-aws) needs to be updated to first rename the instruction file , then the original file. This way deletion of instruction file can be avoided.

It also requires config changes to take Objemetadata/InstructionFile as config parameter.

Let's discuss if we have any better solution and can be incorporated.

Once we agree on one common solution, I can work on implementation part.
Bug: ITestS3ACommitterFactory failing
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/May/24 13:22
Affects Version/s: 3.4.0

we've had ITestS3ACommitterFactory failing for a while, where it looks like changed committer settings aren't being picked up.

{code}
ERROR] ITestS3ACommitterFactory.testEverything:115->testInvalidFileBinding:165 Expected a org.apache.hadoop.fs.s3a.commit.PathCommitException to be thrown, but got the result: : FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl
{code}

I've spent some time looking at it and it is happening because the test sets the fileystem ref for the local test fs, and not that of the filesystem created by the committer, which is where the option is picked up.

i've tried to parameterize it but things are still playing up and I'm not sure how hard to try to fix.
Bug: [ABFS, S3A] Add IORateLimiter api to hadoop common
Status: Open | Priority: Minor | Resolution: 
Created: 03/Apr/24 15:53
Affects Version/s: 3.4.0

Create a rate limiter API in hadoop common which code (initially, manifest committer, bulk delete).. can request iO capacity for a specific operation.

this can be exported by filesystems so support shared rate limiting across all threads

pulled from HADOOP-19093 PR
Bug: testUpdateDeepDirectoryStructureToRemote intermittent failures
Status: Open | Priority: Minor | Resolution: 
Created: 28/Jun/24 10:47

Test testUpdateDeepDirectoryStructureToRemote intermittently fails. Following is an instance in ABFS test runs:

 
{code:java}
[ERROR] testUpdateDeepDirectoryStructureToRemote(org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp) Time elapsed: 2.951 s <<< FAILURE!
java.lang.AssertionError: Files Copied value 2 above maximum 1
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.assertCounterInRange(AbstractContractDistCpTest.java:294)
at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.distCpUpdateDeepDirectoryStructure(AbstractContractDistCpTest.java:334)
at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.testUpdateDeepDirectoryStructureToRemote(AbstractContractDistCpTest.java:259) {code}
 

There is one JIRA in Apache-Ozone which was raised in S3 test run:
https://issues.apache.org/jira/browse/HDDS-10616

 
{code:java}
org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
testUpdateDeepDirectoryStructureToRemote(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp) Time elapsed: 2.375 s <<< FAILURE!
java.lang.AssertionError: Files Copied value 2 above maximum 1
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.assertCounterInRange(AbstractContractDistCpTest.java:294)
at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.distCpUpdateDeepDirectoryStructure(AbstractContractDistCpTest.java:334)
at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.testUpdateDeepDirectoryStructureToRemote(AbstractContractDistCpTest.java:259){code}
Bug: Bulk delete api doesn't take the path to delete as the base path
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jun/24 16:58
Affects Version/s: 3.4.1

If you use the path of the file you intend to delete as the base path, you get an error. This is because the validation requires the list to be of children, but the base path itself should be valid.
Bug: TestHarFileSystem and TestFilterFileSystem failing after bulk delete API added
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/May/24 15:49
Affects Version/s: 3.4.1

oh, we need to update a couple of tests so they know not to worry about the new interface/method. The details are in the javadocs of FileSystem.

Interesting these snuck through yetus, though they fail in PRs based atop #6726

{code}
[ERROR] Failures: 
[ERROR] org.apache.hadoop.fs.TestFilterFileSystem.testFilterFileSystem
[ERROR]   Run 1: TestFilterFileSystem.testFilterFileSystem:181 1 methods were not overridden correctly - see log
[ERROR]   Run 2: TestFilterFileSystem.testFilterFileSystem:181 1 methods were not overridden correctly - see log
[ERROR]   Run 3: TestFilterFileSystem.testFilterFileSystem:181 1 methods were not overridden correctly - see log
[INFO] 
[ERROR] org.apache.hadoop.fs.TestHarFileSystem.testInheritedMethodsImplemented
[ERROR]   Run 1: TestHarFileSystem.testInheritedMethodsImplemented:402 1 methods were not overridden correctly - see log
[ERROR]   Run 2: TestHarFileSystem.testInheritedMethodsImplemented:402 1 methods were not overridden correctly - see log
[ERROR]   Run 3: TestHarFileSystem.testInheritedMethodsImplemented:402 1 methods were not overridden correctly - see log

{code}
Bug: Skip ITestS3AEncryptionWithDefaultS3Settings.testEncryptionFileAttributes when bucket not encrypted with sse-kms
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/May/24 22:03
Affects Version/s: 3.4.1

[ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 4, Time elapsed: 12.80 s <<< FAILURE! -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[ERROR] org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings.testEncryptionFileAttributes -- Time elapsed: 5.065 s <<< FAILURE!
org.junit.ComparisonFailure: [Server side encryption algorithm must match] expected:<"[aws:kms]"> but was:<"[AES256]">
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at org.apache.hadoop.fs.s3a.EncryptionTestUtils.validateEncryptionFileAttributes(EncryptionTestUtils.java:138)
        at org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings.testEncryptionFileAttributes(ITestS3AEncryptionWithDefaultS3Settings.java:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
Bug: Change of Codec configuration does not work
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/May/24 08:50

In one of my projects, I need to dynamically adjust compression level for different files. 
However, I found that in most cases the new compression level does not take effect as expected, the old compression level continues to be used.

Here is the relevant code snippet:
ZStandardCodec zStandardCodec = new ZStandardCodec();
zStandardCodec.setConf(conf);
conf.set("io.compression.codec.zstd.level", "5"); // level may change dynamically
conf.set("io.compression.codec.zstd", zStandardCodec.getClass().getName());
writer = SequenceFile.createWriter(conf, SequenceFile.Writer.file(sequenceFilePath),
                                SequenceFile.Writer.keyClass(LongWritable.class),
                                SequenceFile.Writer.valueClass(BytesWritable.class),
                                SequenceFile.Writer.compression(CompressionType.BLOCK));

The reason is SequenceFile.Writer.init() method will call CodecPool.getCompressor(codec, null) to get a compressor. 
If the compressor is a reused instance, the conf is not applied because it is passed as null:
public static Compressor getCompressor(CompressionCodec codec, Configuration conf) {
Compressor compressor = borrow(compressorPool, codec.getCompressorType());
if (compressor == null)

{ compressor = codec.createCompressor(); LOG.info("Got brand-new compressor ["+codec.getDefaultExtension()+"]"); }

else {
compressor.reinit(conf);   //conf is null here
......

 

Please also refer to my unit test to reproduce the bug. 
To address this bug, I modified the code to ensure that the configuration is read back from the codec when a compressor is reused.