Add in crawler type enum. Integrate crawler type into the code. Refactoring