Refactoring with focus on the Crawler

These changes work towards a resolution of #10 as well as API refinement