Only cache when wanted

Don't unload when leaving, as it might be needed later.