cluster/dht: Fix dht_access treating directory like files

When the cluster topology changes due to add-brick, all sub
volumes of DHT will not contain the directories till a rebalance
is completed. Till the rebalance is run, if a caller bypasses
lookup and calls access due to saved/cached inode information
(like NFS server does) then, dht_access misreads the error
(ESTALE/ENOENT) from the new subvolumes and incorrectly tries
to handle the inode as a file. This results in the directories
in memory state in DHT to be corrupted and not heal even post
a rebalance.

This commit fixes the problem in dht_access thereby preventing
DHT from misrepresenting a directory as a file in the case
presented above.

Change-Id: Idcdaa3837db71c8fe0a40ec0084a6c3dbe27e772
BUG: 1140338
Signed-off-by: Shyam <srangana@redhat.com>
Reviewed-on: http://review.gluster.org/8462
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
(cherry picked from commit 6630fff4812f4e8617336b98d8e3ac35976e5990)
Signed-off-by: Nithya Balachandran <nbalacha@redhat.com>
Reviewed-on: http://review.gluster.org/8721
Reviewed-by: Niels de Vos <ndevos@redhat.com>