GitHub repository: mhenes/PythonCrawler
Primary language: Python
Description: Spyder Crawler using python using multi threading. It fetches top 10 results from Google/Bing search engine for any user specified keyword. User can choose the number of results required and number of thread. Crawler also checks for robot.txt files and avoids restricted urls.