Package: longinus
SaintLonginusVersion 1.1A experiment with web scrappingSaintLonginus is a web crawler that I developed while learning about web scrapping. It's functionality is to search for given words inside a set of pages.
While it looks for references to the searched words within the page, it also looks for links that lead to other pages, increasing its queue of pages to search in.It's built on top ofSeleniumandBeautifulSoupIt's name comes from a popular tradition which says that the catholic saintLonginuscan help you to find anything you are looking for.This project is open-source, so feel free to contribute, report issues or make suggestions.For any other cases, feel free to email me atmail@hugosouza.comDoumentationHere is how you can this crawlerLonginus classThe program's main class.Class Constructor:Longinus(name: str, threads: int = 4, wait_for_page_load_ms: int = 500, when_find: callable = write_to_file, save_bot: bool = False)This is the class constructor, it takes some arguments:name (string): The name of that crawler instancethreads (int): How many threads it's going to use while searchingwait_for_page_load_ms (int):  How many milisseconds the crawler is going to wait for the page to load and run all of its scripts until return the source of that page(this is important to make sure that pages built with React, Angular, Vue or even vanilla JavaScript can render everything it should before the crawler starts scrapping it)when_find (callable): A function that is going to be executed when there's a reference to a specific keyword on that page. The default function writes the url of that page in a text file calledresults.txt. It must take 3 arguments containing theurl of the page,keyword found at the pageandthe whole text of the page:defwrite_to_file(url,keyword,full_text):if"google.com"inurl:returnwithopen("results.txt","a+")asfile:file.write("{}:{}\n".format(url,keyword))file.write(full_text+"\n\n")file.close()save_bot (bool): Sets if the bot is going to be periodically saved into a file for further loading. The frequency of the savings can be defined at setup.Instance methods:setup(depth: int = 3, strategy=SHALLOW_LINKS, bonus_when_match: int = 1, saving_frequency = NORMAL)This method configures the process of searching that the bot is going to use. It takes the arguments:Depth (int):The amount of links beyond the root that are going to be scrappedstrategy (STRATEGY CONSTANT): The searching strategy that is going to be used (Read the strategy section)bonus_when_match (int): This bonus is a value that is added to the depth when the bot finds a reference to a keyword on a page. It's a way to "reward" the page for having a reference to a keyword because it's likely its links also lead to pages with references to a keyword.saving_frequency (FREQUENCY CONSTANT): Defines the frequency the bot is going to the saved (Read the saving frequency section)If you try to start the search without callingsetupfirst, it will raise aAssertionError.set_url(new_urls: list)Changes the urls that the crawler is going to take as root when it starts searching. The root urls are the first ones to be scrapped, and starting from them, the scrapper is going to find the next urls to be visited.When a Longinus object is instantiated, the urls' default value isNone, in that case, the crawler is going to perform a Google search for each keyword and gather a list of urls from the results to be used as root.set_callback(new_callback: callable):Sets what function is going to be executed when there's a reference to a specific keyword on that page. The default function writes the url of that page in a text file calledresults.txt. It must take 4 arguments containing theurl of the page,keyword found at the page,the whole text of the pageandthe title of the HTML document:defwrite_to_file(url,keyword,full_text,title):if"google.com"inurl:returnwithopen("results.txt","a+")asfile:file.write("{}({}):{}\n".format(title,url,keyword))file.write(full_text+"\n\n")file.close()set_filter(new_filter: callable)Sets a new filtering function. A filtering function is used to determinate which pages are going to be crawled and which ones aren't based on their url.The default function filters pages that contains"webcache.googleusercontent.com"on their url.If you want to create a custom filtering function, follow the structure:defcustom_filter(url:str):ifcondition_to_be_filtered(url):returnTrue# This page is going to be filtered (not crawled)returnFalse# This page is going to be crawled (not filtered)start(keywords: list)Starts the searching looking for the words described in thekeywordslist.The search follow the processes configured in thesetupmethod so it must be called after thesetup.If the bot as loaded from a save-file, there's no need to speciLoading a botIf you had previously saved a bot and you want to re-load it, you can do it with the:load_bot_from_save(save_filename: str)This function acts as a constructor, it reads the bot saved file and constructs another instance of the bot with the same specifications, history and searching queue.Thesave_filenameargument represents the name of the file generated when the previous bot was saved.StrategiesThere are several constants representing strategies that can be used to setup the crawler:ONLY_ORIGIN_DOMAIN: Doesn't follow links that lead to a different domain or subdomainONLY_SUBDOMAINS: Follow only origin domain and subdomainsFOLLOW_ALL_LINKS:Follows everything that comes up on the pageSHALLOW_LINKS:Follows links that lead to other domains with depth 0Saving frequencyThere are several constants representing the frequency a bot will be saved:ULTRA_FREQUENT: Will save the bot after each single link searched. (NOT RECOMENDED, IT MIGHT CAUSE THE BOT TO BE EXTREMELLY SLOW)VERY_FREQUENT: Will save the bot after each 10 links searched.FREQUENT: Will save the bot after each 25 links searched.NORMAL: Will save the bot after each 50 links searched.SOMETIMES: Will save the bot after each 100 links searched.RARELY: Will save the bot after each 500 links searched.