Revert r183721. It caused cleanups to be delayed too long in some cases.
Testcase to follow.