ENH: More permissive S3 reading
State: closed

When calling `get_bucket`, boto will by default try to establish that the S3 bucket exists by listing all of the keys that exist in it. This behavior is controlled by the "validate" keyword, which defaults to True. If your access key doesn't have permission to read everything in a bucket (even if you do have permission to read the file you're trying to access), this generates an uninformative exception. 

This PR sets "validate=False". This means that boto will trust you that the bucket exists, and not try to check immediately. If the bucket actually doesn't exist, the `get_contents_as_string` call a couple of lines later will generate the exception "S3ResponseError: S3ResponseError: 404 Not Found".


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
I didn't know that validate behaved that way. @stephen-hoover do you know what version of boto you're using? The behavior may have changed in 2.25 http://boto.readthedocs.org/en/latest/ref/s3.html#boto.s3.connection.S3Connection.get_bucket

Either way this seems reasonable given the way we use it.

I didn't know that until yesterday myself. I'm using the most recent version of boto, 2.38.0.

Ok. I'll merge this later today assuming there aren't any objections.

if this is a newish feature in `boto`, then need some version checking

Oh, sorry, can you add a release not too. in `doc/source/whatsnew/v0.17.0.rst`, probably under bug-fixes. You can just refer to this PR for the issue number. Thanks.

release note as well

@jreback I don't think the keyword argument is new, just what boto does when `validate` is True.

edit: yeah the `validate` kwarg is there in boto 2.0, which is old.

I added a release note. Thanks for checking the old version of boto!

One of the test cases tried to read the file "s3://cant_get_it/tips.csv" and expected to get a permission denied error. With the change I made, this file is now accessible -- apparently the file itself is publicly readable, but the bucket contains non-public contents. 

For now I modified the tests to expect that the file should now be readable -- it's a good check that my change works. You might prefer to change the permissions on that "tips.csv" file so that it's really not publicly accessible, and revert the test to expecting a "permission denied" error. If so, I'll change the test case back.

Did we set up that `cant_get_it` s3 bucket? Seems strange that it would exist and have that file be publicly available.

Anyway, can you add an additional test on a bucket / key that we know doesn't exist? I'd like to make sure that it still raises the S3ResponseError.

The remaining test in `test_s3_fails` tries to access a nonsense address and checks that it gets the expected "S3ResponseError: 404 Not Found" error, so I think that's already covered. I restored the test checking for a 403 error, this time hitting only the bucket. That generates a 403 error. For the purpose of this test, I don't think it matters that the test isn't trying to access a real CSV file, and I don't know of a non-permissioned file in S3 that I could point at instead.

Ahh, yes I missed that other test. Ok, I'll merge later tonight. Thanks

I am curious who owns those S3 buckets. There was already a test hitting a live bucket, so it seems reasonable to keep the check on "cant_get_it/tips.csv". But it does cost a (small) amount of money for external access to a file in S3, so it's a surprising thing to do in a big open-source project like this one.

Thanks!