crowdmob/goamz — pull request #248 by paulmach (2014-10-16 21:34:55+00:00)
Title: Optionally reuse http.Client

This is my proposed fix to https://github.com/crowdmob/goamz/issues/247

Basically connections are not being reused between requests. This causes a few problems:

1. It's inefficient to renegotiate the SSL connection for every request.
2. Leave tcp connections in a TIME_WAIT as they've been closed on the client side, but the os is waiting to see if the server has any late packets before it officially closes client side port.

These issue come up in my use case where I would like to download S3 objects at 200+ qps. I run out of local tcp sockets to use for new connections since they are all in the TIME_WAIT state. This fix solved my problems.

This PR adds an ReuseHTTPClient parameter to the S3 struct and the http.Client will be created on first request. The main issue is that changes to the ConnectTimeout and ReadTimeout will have not effect if made after the first request. (maybe we should check if they've been changed on each request?)

I understand that people have tried to solve this problem before and would be interested in their insights as I'm not an expert on TCP connection reuse and weird states it can find itself in.