VertiPub/hive — pull request #3 by chutong-verticloud (2013-05-05 20:24:26+00:00)
Title: Overriding final params

The reason for the warnings presented in this bug is Hive client tries to override some Hadoop final parameters defined in mapred-default.xml. To dig it deeper, Hive configuration inherits Hadoop configuration and the current way Hive overrides Hadoop parameters is:

1) Create a default Hadoop configuration (this contains all default Hadoop parameters including the ones defined as final).
2) Overlay parameters it wants to override on configuration created in 1).
3) Overlay configuration generated in 2) over the default Hadoop parameters it inherits.

Since configuration generated in 2) contains all the default Hadoop parameters and when it comes to 3), the warning is thrown.

Solution to resolve this problem is when 1) happens, instead of create a default Hadoop configuration, an empty Hadoop configuration is created and 2) overlays Hive parameters on this empty configuration. This way, in 3), configuration in 2) will override any default Hadoop parameters it wants to overrides, however, no warning will be thrown as 2) does not contain default Hadoop parameters.

I took a look at Hive 0.70 and similar logic is used to deal with this problem.

I have tested this by different code path including:

1) keep everything as default
2) define overriding parameters in hive-site.xml
3) define overriding parameters in hive client shell

and all these cases work well.