This can be vectorized if `index_map` is a numpy array.