Revert "Revert "Only delay removing containers. Fixes BUKKIT-5238""

This reverts commit c3a795c2c92acda6b00558d9bc4c99bed5f330e8.