Homebrew/homebrew — pull request #33469 by dtinth (2014-10-22 11:06:26+00:00)
Title: [WIP] Download packages concurrently (contributions wanted)

The issue #2947 (Simultaneous downloads for multiple formula) has been closed because:

> Closing until someone provides a PR. For me this is nice to have and I have more pressing work to attend to.

This one is like that, but comes with a pull request with a working implementation.

Making Homebrew download dependencies' packages in advance turned out not to be that complicated, with only 28 additions and 3 deletions. It just creates a Thread for the download process and calling them ahead of time. When the actual installation take place, the FormulaInstaller waits for the download thread to finish if one has been started.

![screen shot 2014-10-22 at 17 51 14](https://cloud.githubusercontent.com/assets/193136/4734887/5f6092a8-59d9-11e4-849d-8051b3c517b6.png)

However, even though this PR works (it downloads __all__ direct dependencies __at once__), it is very far from complete. Standard output from Curl are smashing into each other all over the terminal. Maybe this thing should be optional. Maybe it should not download all dependencies at once, but one by one in the background, or at most 2, or 4.

But I probably don't have time to complete this, since I already get the behavior I wanted (concurrent and ahead-of-time downloads). It'd be great if the community can help finish it. To give some pointers, a simple way to limit concurrency to, say, 4 simultaneous downloads, is to create a thread pool. [meh/ruby-thread](https://github.com/meh/ruby-thread) provides it, but I'm not sure how to integrate it into Homebrew. It doesn't use gems, does it?

If anyone is interested in helping me complete it and get it merged into Homebrew, please comment so I can give you push access, so you can add commits to this PR.

----

You can find the commit message below:

> Adds a method `#prefetch` to Resource, which is delegated to by SoftwareSpec. This methods instantiates the download in the background. The actual `#fetch` method waits for this background thread to finish, if one has already been started. Otherwise, `#fetch` downloads the package synchronously.
>
> Now, a FormulaInstaller calls `#prefetch` on `active_spec` right away, so that the package can be downloaded while its dependencies are being brewed.
>
> Also, when a list of dependencies is obtained to be installed, we convert each dependency into formula beforehand, calling `#prefetch` on all of them, and proceed.
>
> This is a hack, and this makes the terminal output go crazy, due to several `curl` instances being run at the same time.