Skip sweeping old slots which don't contain any young object in the minor GC