Revert "Attempting to make caching fanouts faster"

This reverts commit ede084f680db3232eb7bbf683aabedb6a9fcafd7.