Bug: Disable reverse DNS lookup for SASL Kerberos client connection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/21 09:19
Affects Version/s: 1.4.13

In some unusual network environment that forward DNS lookup is supported, but revers isn't,

we can configure the HBase cluster by deploying/etc/hosts which support reverse lookup for all nodes in the cluster or hbase.unsafe.regionserver.hostname.disable.master.reversedns=true which is introduced in HBASE-18226(See also HBASE-12954, HBASE-24667).

Our network environment is also unusual and doesn't provide a reverse lookup.
Thus, we configure the HBase cluster by deploying /etc/hosts.

Assume our DNS setup is

{code:java}
master1.example.com A 12.34.56.1
master2.example.com A 12.34.56.2
master3.example.com A 12.34.56.3

regionserver1.example.com A 12.34.56.4
regionserver2.example.com A 12.34.56.5
regionserver3.example.com A 12.34.56.6
{code}

We deploy the following /etc/hosts for the HBase cluster nodes by our deployment system.

{code:java}

12.34.56.1 master1.example.com master1
12.34.56.2 master2.example.com master2
12.34.56.3 master3.example.com master3

12.34.56.4 regionserver1.example.com regionserver1
12.34.56.5 regionserver2.example.com regionserver2
12.34.56.6 regionserver3.example.com regionserver3
{code}

(We don't use hbase.unsafe.regionserver.hostname.disable.master.reversedns=true for now)

So all nodes in the cluster have stable reverse lookup for the IPs in the cluster, and the HBase cluster deployed in this way is quite stable and we can expand the cluster easily without any modification on the client-side.

Now we need to introduce Kerberos SASL secured cluster for security reasons.

We tried to construct in the same way as is, i.e. deploy /etc/hosts for the HBase cluster nodes.
However, this won't work well because the HBase client does a reverse lookup to get principal for Kerberos.
(hbase.unsafe.regionserver.hostname.disable.master.reversedns=true won't work as well as)


Thus we need to deploy /etc/hosts to all application servers, which contains all nodes of the HBase cluster to be connected.

This is quite terrible for our cluster operation and application server setup.



We, the HBase cluster manager, need to take care of application server setup and deployment.
We must provide all master and region server lists.
It's much more complicated when applications access multiple HBase clusters...
We, the HBase cluster manager, cannot expand the cluster unless the latest /etc/hosts are deployed to all application servers.
If we expand the cluster before deployment, the application is unable to connect and got an error

Assume their own Kerberos principal is their FQDN i.e. master1.example.com for example, and the cluster is aware of their FQDN.
So all clients can connect cluster nodes using the FQDN for Kerberos principal.



Could we provide an advanced unsafe option to disable DNS reverse lookup for clients using Kerberos SASL like hbase.unsafe.regionserver.hostname.disable.master.reversedns and other config?

Let's say `hbase.unsafe.client.kerberos.hostname.disable.reversedns` and if this is true, client uses InetAddress.getHostname() for Kerberos principal instead of InetAddress.getCanonicalHostName().