rewrote the crawler to use multiple threads for loading files from repositories, considering files, and updating files; also added way better logging