sunpy/sunpy — pull request #1002 by dpshelio (2014-05-23 07:25:57+00:00)
Title: WIP: Scrapper to generalise the downloads from http/ftp directories

This is a Work in Progress to create a single scrapper which could be used to download data from any site, avoiding therefore to create custom web scrappers for each data service.

For example, a way to use it within Callisto would be:
```python
pattern = "http://soleil.i4ds.ch/solarradio/data/2002-20yy_Callisto/%Y/%m/%d/{observatory}_%Y%m%d_%H%M%S_{number}.fit.gz"
scrap = Scrapper(pattern, observatory='BIR', number='01')
urls = scrap.filelist(start, end)
```
or for goes:
```python
pattern = "http://umbra.nascom.nasa.gov/goes/fits/%Y/go{sat_num:02d}%Y%m%d.fits"
scrap = Scrapper(pattern, sat_num = sat_num[0])
urls = scrap.filelist(start, end)
```
So far (and not in this PR) I've just tried to implement it in Callisto, however I haven't figured out yet how to make to take more than one `number`, as I cannot write in the current status a list of possible inputs (eg. `number = 01 || 03`)

Potentially we could then have a list of patterns and the general downloader in plan could choose the one to use.

Please, comments, criticisms, help is appreciated :)