rdwilliamson/aws — pull request #10 by rpkamp (2014-09-19 19:51:17+00:00)
Title: Solved problem with "Too many open files"

I've built a glacier backup program in Go (https://github.com/rpkamp/gobackup) using your excellent library, but I ran into a problem that after a few hundred uploads I would get the error 'Too many open files' and the whole thing would come crumbling down and refuse to upload anything from that point forward.

I'd checked my code several times and I couldn't find a leak where I'd leave file descriptors hanging. So I dug in to it some more and found that `http.Client` may not re-use connections if you don't fully consume the response (see http://stackoverflow.com/questions/17948827/reusing-http-connections-in-golang in the first answer by Matt Self).

I saw in `aws.Connection.UploadArchive()` that you are currently not consuming the response of the request at all, so I was guessing that that was the reason for my 'Too many open files' problem.

So I added the code to drain the response (by simply copying it to `io.Discard` since I'm not interested in it), recompiled, and voila, I've got 2 connections open to glacier which are constantly being reused for new files, and no more 'Too many open files'. At least I've yet to see one :)