Fix sparse arrays to not insert into already used indices, and not remove already removed indices.