There's a lot of complexity in this class. The rate-limiting could be achieved using a fixed thread pool of size 1, perhaps?

It would be interesting to see the performance benefit of pre-fetching vs. just-in-time fetching; if the improvement isn't great, then perhaps the extra complexity isn't worth it.

We definitely need some tests here.