Don't create more than one buffer per ndarray