Revert "Jarv/config refactor wip"

Reverts edx/edx-platform#6096

Looks like this affected our CI environment. Re-running with this PR.