Create a new in-memory cache for the settings if we are in a unit test runner