Sometimes sites make crawling hard. Selenium-crawler uses selenium automation to fix that.