Faster index lookup in RemoveInternal