Improve caching to not throw all our metadata away