Remove assertions that never trigger anyway

These were either commented out, or I've never seen them being hit when
investigating GC bugs. This removes them.