Updated the README.md file to refeclt the current changes and explained how the crawler is working