adding integration test for crawler
fixing tests