Feature/improve find and scan code

Part of fixing #856 and #858. I'm stilling missing the follow config option, but rest should be more or less done, or at least ready for review.
True, with the exception of directories with permission problems showing up in the errors. So yes the logging is a bit off. I think the more sane approach is to pass in a function which is given each path and then decides if we should traverse it.

Though ideally I should just make sure the pipeline handles things like `.directory` files etc nicely and then we wouldn't need to skip anything.