fix for HADOOP-3733

tested with:
- unit tests included in patch
- hadoop fs -ls ...
- distcp to/from s3:// and s3n://

relevant links:
- https://verticloud.atlassian.net/browse/AE-173
- https://issues.apache.org/jira/browse/HADOOP-3733