config: Only load .conf-files when loading directories
docs: Add directories to --config in man page