Made the crawler class extendable. This was to eliminate duplicate code.