Stop caching special configs

@mklauber - this is working reliably for me.
@JLMcLaugh - do you have any problems or know anywhere else we're caching really obvious stuff that I missed?

All tests pass.