Ensure remove_at is really removing things