Fix leaking FD

See #1289 
Reviewers, @ddysher: Don't forget to consider, what happens if this crashes? Should probably have a "defer util.HandleCrash()" here.