Added integration test for deep crawling, fixed port in dir name bug and added some logging.