Revert "Defer GC and scrub ivars to speed up test suite"

This seems to be causing issues on CI - worth reverting this just in case.

This reverts commit 3fda09c7d6c713c721b05bf0ccea95f5fea973f9.