freeman-lab/thunder — pull request #67 by industrial-sloth (2014-12-04 20:04:14+00:00)
Title: add wildcard handling to BotoS3FileReader

Fixes #64.

Not easy to set up unit tests for this, short of mocking out the boto library, but here's a before-and-after comparison that I ran:

Before:
```python
In [1]: from thunder.rdds.fileio.readers import BotoS3FileReader
In [2]: reader = BotoS3FileReader()
In [3]: len(reader.list("s3n://thunder.datasets/test/stack-big/TM*"))
Out[3]: 5503
In [4]: len(reader.list("s3n://thunder.datasets/test/stack-big/*.stack"))
Out[4]: 5503
```
This s3 bucket actually has 5500 `TM*.stack` files, together with a .tif, a .log, and a .xml.

After:
```python
In [11]: fnames = reader.list("s3n://thunder.datasets/test/stack-big/TM00001*.stack")
In [12]: fnames[:10]
Out[12]: [u's3n:///thunder.datasets/test/stack-big/TM00001_CM0_CHN00.stack']
In [13]: len(reader.list("s3n://thunder.datasets/test/stack-big/"))
Out[13]: 5503
In [14]: len(reader.list("s3n://thunder.datasets/test/stack-big"))
Out[14]: 5503
In [15]: len(reader.list("s3n://thunder.datasets/test/stack-big*.tif"))
Out[15]: 1
In [16]: len(reader.list("s3n://thunder.datasets/test/stack-big/*.tif"))
Out[16]: 1
In [17]: len(reader.list("s3n://thunder.datasets/test/stack-big/*.stack"))
Out[17]: 5500
# ...
In [20]: len(reader.list("s3n://thunder.datasets/test/stack-big/TM*"))
Out[20]: 5500
In [21]: len(reader.list("s3n://thunder.datasets/test/stack-big/", filename="ch0.xml"))
Out[21]: 1
```