added base functionality to crawler