Ensure that we aren't loading TOO much data to the prefetcher