Remove bad directories added by defaults from path

Always sourcing the defaults may result in invalid directories being
added to the path. These are removed after the machine specific config
is loaded.