refactoring to allow for mining of local files