saltstack/salt — pull request #12268 by jslatts (2014-04-24 17:03:10+00:00)
Title: utils/s3.py throws UnicodeDecodeError when attempting to download tarball from S3

While trying to retrieve a tarball from S3 and extract it locally using archive.tar, I ran into this error below.

Traceback (most recent call last):
File "/root/salt/salt/states/file.py", line 1284, in managed
makedirs
File "/root/salt/salt/modules/file.py", line 2798, in manage_file
sfn = __salt__['cp.cache_file'](source, saltenv)
File "/root/salt/salt/modules/cp.py", line 314, in cache_file
result = __context__['cp.fileclient'].cache_file(path, saltenv)
File "/root/salt/salt/fileclient.py", line 145, in cache_file
return self.get_url(path, '', True, saltenv)
File "/root/salt/salt/fileclient.py", line 542, in get_url
raise MinionError('Could not fetch from {0}'.format(url))
MinionError: Could not fetch from s3://my.fancy.bucket/test-build-123.tar.gz

I traced the issue to [line 192 in utils/s3.py](https://github.com/saltstack/salt/blob/develop/salt/utils/s3.py#L192):

# This can be used to save a binary object to disk
if local_file and method == 'GET':
log.debug('Saving to local file: {0}'.format(local_file))
with salt.utils.fopen(local_file, 'w') as out:
out.write(response)
return 'Saved to local file: {0}'.format(local_file)

Which throws this error:

'ascii' codec can't encode character u'\u2039' in position 1: ordinal not in range(128)

The response object is getting decoded by the requests library. According to the [Requests docs](http://docs.python-requests.org/en/latest/user/quickstart/#response-content), for a binary object (like what we would expect to get/retrieve from S3), the _content_ property will not attempt to decode the returned object. This PR fixes the issue for me, though I am not sure if there are any unintended side effects.

For the record, here is the state I am using to generate the issue:

generated_hsd[full_local_path] = {
'archive': [
'extracted',
{ 'name': app['deploy']['path'] },
{ 'user': 'myuser' },
{ 'group': 'mygroup' },
{ 'mode': 400 },
{ 'source': 's3://{0}/{1}'.format(bucket, build_archive) },
{ 'source_hash': 'md5={0}'.format(build_hash) },
{ 'tar_options': 'xf' },
{ 'archive_format': 'tar' },
{ 'if_missing': '{0}/{1}'.format(local_path, build) },
{ 'require': [
{ 'user': 'myuser' },
{ 'file': local_path }
]
}
]
}