astropy/astropy — pull request #615 by mdboom (2013-01-08 21:29:13+00:00)
Title: multiprocessing fixes for downloads

This addresses a few things related to multiprocessing and downloading from URLs.

1. This keeps the cache lock for a much shorter amount of time. It's only needed when the cache mapping or files are written to, not for the entire time that the file is downloaded over the network. This does have the downside that it makes it possible for two threads to download the same URL simultaneously and possibly overwrite each other's files at the end. In practice, I'm not sure this matters too much. If it does, it would be nice to lock on a file-by-file basis, but unfortunately, we don't know the name of the local file until we have its contents fully downloaded.

2. It creates the `.astropy` directory heirarchy in a more race-condition-safe way. If the directory doesn't exist, so we create it, but it turns out another process has already created it in the meantime, we can safely ignore the exception.

3. Only the main process and main thread will display progress bars. That's a cheap way to address the "competing progress bar" problem.

4. Adds a `download_files_in_parallel` method as a convenience for downloading a number of files at once. Downloading them in parallel may be faster if they exist on different servers. You get a nice combined progress bar with this approach, too.

Related to https://github.com/astropy/astroquery/pull/28