Update the crawler functionality