Get rid of the multiple close problem and remove some overhead

Fixes #187.