aws/aws-cli — pull request #468 by jamesls (2013-11-05 18:37:33+00:00)
Title: Add detailed error message on local file encoding errors

The general error was that `list_files` was raising an exception
that would propogate up to the main clidriver handler and stop
execution. While `list_files` is doing the right thing, there were
a few issues:

* There were no error message logged to the console (bug in log configuration).
* There were debug logs that indicated an error but the error message
was vague and complained about a UnicodeDecodeError. It also wasn't
clear why it happened or how to fix it.

As to why `list_files` is raising an exception:

The issue is with the call `os.listdir()`. What is *suppose*
to happen in this case is that python should return unicode objects
if called with a unicode object, and should return bytestrings if called
with a bytestring. So in python2:

os.listdir(u'.') -> [u'unicode']
os.listdir('.') -> ['not unicode']

In order to decode the bytes to unicode it uses
`sys.getfilesystemencoding()`. On Mac/Windows, these values are
hardcoded, but on unix, this is loaded via `nl_langinfo(CODESET)`,
which maps to the `LC_CTYPE` env var.

This is documented here http://docs.python.org/2/library/sys.html#sys.getfilesystemencoding

However, `os.listdir` will return the original bytestring
*if it fails to decode the bytestring* using `getfilesystemencoding()`.
The full picture is therefore:

os.listdir(u'.') -> [u'unicode']
os.listdir(u'.') -> ['possibly byte string if decoding fails']
os.listdir('.') -> ['byte string']

Once this happens we're calling `os.path.join(, )`,
which means we'll need to decode the bytestring to unicode. In this
situation we use `sys.getdefaultencoding()`, which is likely going to be
ascii, hence the error message about the ascii codec not being able
to decode a byte.

One alternative considered was to just skip the file if we can't decode
the object to unicode. However, if someone uses the `--delete` flag,
skipping the file will result in the file being deleted on the other
end, which not not desired behavior.

The error message when this happens looks like this:

```
2013-11-05 18:28:50,195 - awscli.customizations.s3.s3handler - ERROR - Exception caught during task execution:
There was an error trying to decode the the file "\xce\x24" in directory "/home/mydir/sync/bar/": 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

Please check your locale settings. The filename was decoded as: ANSI_X3.4-1968
On posix platforms, check the LC_CTYPE environment variable.

```
Closes #383.