Improve caching in AppConfig

For #6145 

cc @DeepDiver1975 @karlitschek