cluster/dht: Ignore subvols with error in min-free-disk/inodes

Currently when selecting a alternative subvolume when hashed
subvol has exceeded min-free-disk/inodes, we do not check if
layouts have errors (including decommissioning). This leads
to data being written to those subvolumes, and in case of
decommissioning, will lead to data loss.

Change-Id: Ie0c6cf4a29d7c53d8a6d8a8c1bd595cf58a0012a
BUG: 982919
Signed-off-by: shishir gowda <sgowda@redhat.com>
Reviewed-on: http://review.gluster.org/5299
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
test-ffop: the result of (O_TRUNC|O_RDONLY) is unspecified

On RHEL-6 an open(.., O_TRUNC|O_RDONLY) fails. test-ffop expects this to
succeed, but the behaviour is unspecified according to 'man 2 open':

       O_TRUNC
              If  the  file  already exists and is a regular file and the open
              mode allows writing (i.e., is O_RDWR or  O_WRONLY)  it  will  be
              truncated to length 0.  If the file is a FIFO or terminal device
              file, the O_TRUNC flag is  ignored.   Otherwise  the  effect  of
              O_TRUNC is unspecified.

Change-Id: I76b55419871fc758cbe82532f9b75476a0c14101
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/5268
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>