Improve scan performance for remote filesystems

Old state this was 80s or so, new code completes in 14s for the same find operation and is not slowed further by later having to re-stat the same files for the mtimes.

Should further improve #614