modified the crawler with rspec test cases