aws/aws-cli — pull request #385 by jamesls (2013-09-30 20:28:49+00:00)
Title: Tasks should share connection pools

Fixes #381.

The issue was that each task object was using it's own endpoint class, which maps to a requests Session, and ultimately a connection pool. The high level change is here is that we should only be creating a single `Endpoint` object and sharing it amongst all the tasks. This does require that `Endpoint` objects are thread safe. There was a change in botocore to protect the auth signing with a lock (https://github.com/boto/botocore/pull/142) so `Endpoints` are thread safe now.

Confirmed that we don't leak socket fd's with a large number of uploads, we're constrained to a maximum number of open connection being the thread pool size (default of 12). Verified this on a linux machine, in this snapshot, we're only using 11 connections:

```
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 0 -> /dev/pts/1
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 1 -> /dev/pts/1
lr-x------ 1 ubuntu ubuntu 64 Sep 30 19:55 10 -> /sync/file-98
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 11 -> socket:[50063]
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 12 -> socket:[50761]
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 13 -> socket:[50765]
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 14 -> socket:[50065]
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 15 -> socket:[50059]
lr-x------ 1 ubuntu ubuntu 64 Sep 30 19:55 16 -> /sync/file-98
lr-x------ 1 ubuntu ubuntu 64 Sep 30 19:55 17 -> /sync/file-98
lr-x------ 1 ubuntu ubuntu 64 Sep 30 19:55 18 -> /sync/file-98
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 2 -> /dev/pts/1
lr-x------ 1 ubuntu ubuntu 64 Sep 30 19:59 21 -> /sync/file-98
lr-x------ 1 ubuntu ubuntu 64 Sep 30 19:55 3 -> /usr/share/bash-completion/completions
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 4 -> socket:[50775]
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 5 -> socket:[50057]
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 6 -> socket:[50771]
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 7 -> socket:[50061]
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 8 -> socket:[50055]
lrwx------ 1 ubuntu ubuntu 64 Sep 30 19:55 9 -> socket:[50773]
```

Previously, you'd see the fd's leak until you hit the max fd's per process (typically 1024).

This is a large change, but most of is in the test cases, as they tend to create a lot of `FileInfo` classes. I think the long term change is going to be to separate the `FileInfo` class into two separate classes, one that describes a specific file operation, and a class that performs the s3 operations (a high level botocore abstraction).

Also, I've renamed `operation` to `operation_name`. The `operation` is typically used to refer to an `Operation` object in botocore, so to help disambiguate I've renamed this value appropriately, give that this change keeps this pattern with the `.service` and `.endpoint` names.