Create special config folder to hold andy configuration-related data