Revert "Only delay removing containers. Fixes BUKKIT-5238"

This reverts commit 530fcb7da280e660fcbd09b116fa71368c258f42.