Make sure we don't prefetch more than once.

TODO: should lock access to self.prefetches.