docker/docker — pull request #9714 by IanCal (2014-12-17 22:31:47+00:00)
Title: Run the remote image presence checks in parallel (speeds up pushes)

This PR makes all of the requests to check for the image already existing in the registry in parallel, then pushes only those that it needs to one by one. This can cut down times where you are pushing a small number of new layers. For an image with 15 layers this can cut down push times from ~20s to ~6s.

This currently doesn't provide a huge speedup for the docker hub, but will for anything using docker-registry. This is due to the hub responding with a 401 for every check (meaning a push will always be attempted). This does still cut down on the overall time however (~20s to ~17s).

Possible considerations:

* Is this too many requests at once?
* Do we want to also do the pushes themselves in parallel?
* Are there more requests we can bundle together / streamline?
* I'm pretty new to go, is this a poor way of doing this?

# Examples of output

Current master:

$ sudo time docker push tutum.co/iancal/touched
The push refers to a repository [tutum.co/iancal/touched] (len: 1)
Sending image list
Pushing repository tutum.co/iancal/touched (1 tags)
Image 511136ea3c5a already pushed, skipping
Image df7546f9f060 already pushed, skipping
Image e433a6c5b276 already pushed, skipping
Image e72ac664f4f0 already pushed, skipping
Image 4b7b4bdd15f5 already pushed, skipping
Image dfa042d4c557 already pushed, skipping
Image 5ca9e32c43a4 already pushed, skipping
Image 530a1e076a6f already pushed, skipping
Image 9b5ea9aa53b9 already pushed, skipping
Image 8179f0d15d9b already pushed, skipping
Image c591bbba5ed4 already pushed, skipping
Image a6ee666e4fff already pushed, skipping
Image 0f2344772f88 already pushed, skipping
Image 0881e8acb442 already pushed, skipping
Image 346a283c1060 already pushed, skipping
Image 3857b7bcd7e9 already pushed, skipping
Image feede2781b3d already pushed, skipping
Image 8f9f59ed1c63 already pushed, skipping
Image bd59c53042c8 already pushed, skipping
Pushing tag for rev [bd59c53042c8] on {https://tutum.co/v1/repositories/iancal/touched/tags/latest}
0.00user 0.01system 0:23.02elapsed 0%CPU (0avgtext+0avgdata 6356maxresident)k
0inputs+0outputs (0major+2025minor)pagefaults 0swaps

New build:

$ sudo time docker push tutum.co/iancal/touched
The push refers to a repository [tutum.co/iancal/touched] (len: 1)
Sending image list
Pushing repository tutum.co/iancal/touched (1 tags)
Image 0f2344772f88 already pushed, skipping
Image dfa042d4c557 already pushed, skipping
Image 4b7b4bdd15f5 already pushed, skipping
Image 346a283c1060 already pushed, skipping
Image c591bbba5ed4 already pushed, skipping
Image 5ca9e32c43a4 already pushed, skipping
Image e433a6c5b276 already pushed, skipping
Image df7546f9f060 already pushed, skipping
Image a6ee666e4fff already pushed, skipping
Image 530a1e076a6f already pushed, skipping
Image 9b5ea9aa53b9 already pushed, skipping
Image e72ac664f4f0 already pushed, skipping
Image 0881e8acb442 already pushed, skipping
Image 8179f0d15d9b already pushed, skipping
Image 511136ea3c5a already pushed, skipping
Pushing tag for rev [346a283c1060] on {https://tutum.co/v1/repositories/iancal/touched/tags/latest}
0.01user 0.00system 0:05.19elapsed 0%CPU (0avgtext+0avgdata 8488maxresident)k

Checking new layers can still be pushed:

$ sudo time docker push tutum.co/iancal/touched
The push refers to a repository [tutum.co/iancal/touched] (len: 1)
Sending image list
Pushing repository tutum.co/iancal/touched (1 tags)
Image a6ee666e4fff already pushed, skipping
Image 8179f0d15d9b already pushed, skipping
Image 530a1e076a6f already pushed, skipping
Image 346a283c1060 already pushed, skipping
Image e433a6c5b276 already pushed, skipping
Image dfa042d4c557 already pushed, skipping
Image 0881e8acb442 already pushed, skipping
Image 0f2344772f88 already pushed, skipping
Image df7546f9f060 already pushed, skipping
Image e72ac664f4f0 already pushed, skipping
Image 9b5ea9aa53b9 already pushed, skipping
Image 5ca9e32c43a4 already pushed, skipping
Image c591bbba5ed4 already pushed, skipping
Image 4b7b4bdd15f5 already pushed, skipping
Image 511136ea3c5a already pushed, skipping
Image 3857b7bcd7e9 not pushed, adding to queue
Image bd59c53042c8 not pushed, adding to queue
Image 8f9f59ed1c63 not pushed, adding to queue
Image feede2781b3d not pushed, adding to queue
3857b7bcd7e9: Image successfully pushed
feede2781b3d: Image successfully pushed
8f9f59ed1c63: Image successfully pushed
bd59c53042c8: Image successfully pushed
Pushing tag for rev [bd59c53042c8] on {https://tutum.co/v1/repositories/iancal/touched/tags/latest}
0.00user 0.00system 0:21.15elapsed 0%CPU (0avgtext+0avgdata 8448maxresident)k
0inputs+0outputs (0major+2043minor)pagefaults 0swaps