Revert "Cache all the fragments up front"

This reverts commit d6474b55d67dc1f6b9bf4a290c22dd0003147f85.