Also remove extra directories that are cruft to the user