This should probably happen after running before_configuration hooks since we're setting configs here