Do not visit files while loading

Visiting files has all sorts of side effects, most notably visited files get
added to the list of recent files (`recentf`).