Updated crawler with custom crawling rules