Remove unnecessary garbage collection deferring in specs

It might have made tests faster, but our test suite runs soo fast
already without any tweaks, so this hack was unnecessary.