rewrote to rescan files rather than try to hold all files in memory