Adjusting recursive crawling logic and improving the logging