Is the broadcast in `_index_to_arrays` insufficient?