Package: bs4-web-scraper
bs4_web_scraperA web scraper based on the BeautifulSoup4 library with translation capabilities.View Project on PyPIDependenciesPython 3.11beautifulsoup4translatorsrequestslxmlhtml5libpyyamltomlSetup Local Development EnvironmentMake sure you have python3 installed on your local machine.Clone the repository to local machine into your project directory.Change directory into the repository "bs4_web_scraper" andpip install -r requirements.txt. You're ready to go if the installations were successful.For a quick demo, runexample.py.InstallationMake sure you have python3 installed on your local machine.Runpip install bs4-web-scraperon terminal.Import the module into your project and you're good to go.FeaturesWeb scrapingTranslationSaving scraped data to a fileDownloading data from a web page or URLLogging the scraping processUsageBefore using the scraper, make sure you have an internet connection. The scraper uses the internet to scrape web pages and translate scraped data.NOTE:The scraper is not a browser. It does not execute JavaScript. It only scrapes the HTML content of web pages.The scraper requires an internet connection to scrape web pages and translate scraped data.The scraper was built with HTML5 web pages in mind. It may not work well with older HTML versions.Some web pages may not be scraped properly due to the way they are structured. This is not a bug. It is a limitation of the scraper.The scraper only attempts to authenticate with the information provided. The success of the authentication largely depends on the information provided to the scraper. This is because the authentication process varies from website to website. It is up to the user to provide the correct authentication information.The scraper does not support authentication with CAPTCHA.The scraper does not support scraping dynamic web pages by default.Some methods of the scraper may not work properly if the scraper is not instantiated properly. Ensure to read the docstrings of the methods and class attributes before using them.A number of methods of the scraper use multithreading. This means that the scraper can run multiple threads at the same time. This is done to speed up the scraping process. The number of threads used by the scraper can be set by the user but it is advisable to leave it at its default value.Importing the modulefrombs4_web_scraper.scraperimportBS4WebScraperCreating a scraper objectThe following example shows how to instantiate and customize the scraper's settings. The default settings are used if no parameters are passed to the scraper object.# Here, the scraper object is created with the default settingsbs4_scraper=BS4WebScraper()# To customize the scraper's settings, pass a dictionary of the preferred instantiation parameters to the scraper object.params={"parser":"html.parser","markup_filename":"base.html","log_filepath":"./scrape_log/log.txt","no_of_requests_before_pause":30,# This should not exceed 50 to avoid high frequency requests. The upper limit is 100"scrape_session_pause_duration":20,# pause duration in seconds. It is advisable to leave this at its default, "auto"."max_no_of_retries":5,"base_storage_dir":"./scraped_data","storage_path":".",}bs4_scraper=BS4WebScraper(**params)Instantiation parametersTo read more about the instantiation parameters and class attributes, run the following command:>>>print(BS4WebScraper.__doc__)Scraping a web pageLet's say you want to download a website/page to your local machine along with its dependencies(like CSS files, scripts, images or fonts), thescrapemethod can be used. Below is an example of how to scrape a web site.# Scraping google.combs4_scraper.scrape(url="https://www.google.com",scrape_depth=0)In the above example, thescrape_depthparameter is set to 0. This means that the scraper will only scrape the web page at the given url. If thescrape_depthparameter is set to 1, the scraper will scrape the web page at the given url and all the web pages linked to it. If thescrape_depthparameter is set to 2, the scraper will scrape the web page at the given url and all the web pages linked to it and all the web pages linked to the web pages linked to it and so on.Translating scraped dataTo translate the scraped data, set thetranslate_toparameter to the language you want to translate to. The following example shows how to translate the scraped data to French. The translation is done using the translation engine specified in the instantiation parameters. The default translation engine is "google" (Google Translate). To change the translation engine, set thetranslation_engineparameter or attribute to the preferred translation engine. The following example shows how to translate the scraped data to French using the Bing translation engine.# Scraping a web page and translating the scraped data to Frenchbs4_scraper=BS4WebScraper(...)bs4_scraper.scrape(url="https://www.google.com",scrape_depth=0,translate_to="fr",translation_engine='google')To get a list of available translation engines you can use, do the following:frombs4_web_scraperimporttranslate# INTERNET CONNECTION REQUIREDprint(translate.translation_engines)Translation of the web pages is done immediately after scraping. Thetranslate_toparameter is set to "fr" in the above example. This means that the scraped data will be translated to French. Thetranslate_toparameter can be set to any of the languages supported by the scraper's translation engine.To get a list of the languages supported by the scraper's translation engine, do:print(bs4_scraper.translator.supported_languages)Scraping web sites or pages that require authenticationTo scrape websites or pages that require authentication, you can pass thecredentialsparameter to thescrapemethod.The following example shows how to scrape a web page that requires authentication.# Scraping a web page that requires authenticationcredentials={'auth_url':'https://www.websitewithauth.com/login_path/','auth_username_field':'usernamefieldname','auth_password_field':'passwordfieldname','auth_username':'yourusername','auth_password':'yourpassword','additional_auth_fields':{'fieldname':'fieldvalue','fieldname':'fieldvalue',}}bs4_scraper.scrape(url="https://www.websitewithauth.com",scrape_depth=0,credentials=credentials)You can also authenticate the scraper before scraping by passing thecredentialsparameter to theauthenticatemethod. The following example shows how to authenticate the scraper before scraping.# Authenticating the scraper before scrapingbs4_scraper.authenticate(credentials=credentials)bs4_scraper.scrape(url="https://www.websitewithauth.com",scrape_depth=0)# or in the case of downloading data from a web page that requires authenticationbs4_scraper.authenticate(credentials=credentials)bs4_scraper.download_url(url="https://www.websitewithauth.com/download/example.mp4",save_as="example.mp4",save_to="downloads")# run help(bs4_scraper.downloaded_url) for more information on the download_url methodNOTE:credentialsshould always take the form of a dictionary with the following keys:auth_url,auth_username_field,auth_password_field,auth_username,auth_password,additional_auth_fields. Theadditional_auth_fieldskey is optional. It is used to pass additional authentication fields that may be required by the website or page.To get a quick template for thecredentialsdictionary, do:importbs4_web_scraperprint(bs4_web_scraper.credentials_template)Other useful scraper methodsThe following are some useful methods for scraping web data using the scraper class.download_urldownload_urlsfind_urlsfind_all_tagsfind_tags_by_idfind_tags_by_classfind_commentsfind_linksfind_stylesheetsfind_scriptsfind_videosfind_imagesfind_audiosfind_fontsfind_patternfind_emailsfind_phone_numbersFor information on how to use these methods, do:>>>help(bs4_scraper.<method_name>)Other utility classes included in the moduleTranslatorFileHandlerLoggerRequestLimitSettingFor information on how to use these classes, do:frombs4_web_scraper.<module_name>import<class_name>>>>help(<class_name>)Scraper MethodsBefore proceeding, it is important to know what an 'rra' means. 'rra' stands for 'resource related attribute'. A resource related attribute is an attribute that is related to a file(resource) that the webpage is dependent on. For example, thehrefattribute of anatag is a resource related attribute because it is related to the resource (link) that theatag points to. Thesrcattribute of animgtag is also a resource related attribute because it is related to the resource (image) that theimgtag points to.download_urlThedownload_urlmethod is used to download files from a web page or url. The following example shows how to download a file from a web page. A simple example usage is shown below:# Downloading a file from a web pagefile_handler=bs4_scraper.download_url(url="https://www.example.com/download/example.mkv",save_as="example.mkv",save_to="downloads")# Returns FileHandler Object for the downloaded fileprint(file_handler.filename)download_urlsThedownload_urlsmethod is used to download multiple files from multiple web pages or urls. The following example shows how to download multiple files from a web page. A simple example usage is shown below:# Downloading multiple filesurls=["https://www.example.com/download/example1.mkv","https://www.example.com/download/example2.mkv","https://www.example.com/download/example3.mkv","https://www.example.com/download/example4.mkv","https://www.example.com/download/example5.mkv",]bs4_scraper.download_urls(urls=urls,save_to="downloads",fast_download=True)To define what each file should be saved as, you can pass thesave_asalongside the url as a dictionary -urlsbecomes a list of dictionaries, as parameter to thedownload_urlsmethod. The following example shows how to download multiple files from a web page and save them with different namesurls=[{"url":"https://www.example.com/download/example1.mkv","save_as":"example_1.mkv"},{"url":"https://www.example.com/download/example2.mkv","save_as":"example_2.mkv"},{"url":"https://www.example.com/download/example3.mkv","save_as":"example_3.mkv","save_to":"./downloads2/"# This will be treated as an absolute path and will not be saved in `self.base_storage_dir`.# This file will be saved in its own directory(determined by the path given).},# You can also add a normal string url. This will we saved as 'example4.mkv' by default."https://www.example.com/download/example4.mkv",]bs4_scraper.download_urls(urls=urls,save_to="downloads",fast_download=True)find_urlsThis method is used to get all resource related attribute(url or link) on elements that match a given tag name. This method only works for tags that have thesrcorhrefattribute. The following example shows how to find all urls on theimgelements in a web page:# Finding all urls on the img elements on a web page with a class of 'sub-image', saving them to a file and downloading themimg_urls=bs4_scraper.find_urls(url='https://example.com/',target="img",attrs={"class":"sub-image"})# save the urls to a file using the save_results methodbs4_scraper.save_results(results=img_urls,file_path="downloads/img_urls.txt")# download the urls using the download_urls methodbs4_scraper.download_urls(urls=img_urls,save_to="downloads")find_all_tagsThis method is used to get all elements that match a given tag name. A simple example usage is shown below:# Finding all small elements from a url going two levels deepsmall_elements=bs4_scraper.find_all_tags(url='https://example.com/',target="small",depth=2)print(small_elements)find_linksThis method is used to get thehrefon allaelement. A simple example usage is shown below:# Finding all links on a web pagelinks=bs4_scraper.find_links(url='https://example.com/')print(links)Allfind_*methods have the same usage as thefind_linksmethod. For example, thefind_stylesheetsmethod is used to get thehrefon alllinkelement with arelattribute ofstylesheet. A simple example usage is shown below:# Finding all stylesheets on a web pagestylesheets=bs4_scraper.find_stylesheets(url='https://example.com/')print(stylesheets)find_patternThis method is used to find all elements that match a given REGEX pattern. A simple example usage is shown below:# Finding all elements that match a email REGEX patternpattern=r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$"emails=bs4_scraper.find_pattern(url='https://example.com/',regex=pattern)print(emails)Utility ClassesTranslatorThis class is simply used to translate text, a file or soup from one language to another. The following example shows how to translate text from English to Yoruba:frombs4_web_scraper.translateimportTranslator# Translating text from English to Yorubatranslator=Translator(translation_engine="google")translated_text=translator.translate(content="Hello World",src_lang="en",target_lang="yo")print(translated_text)# If content is markuptranslated_text=translator.translate(content="<p>Hello World</p>",src_lang="en",target_lang="yo",is_markup=True)print(translated_text)To translate a file, you can pass the file path to thetranslatemethod as shown below:# Translating a file from English to Yorubatranslator=Translator(translation_engine="google")translated_file_handler=translator.translate_file(file_path="path/to/file.txt",src_lang="en",target_lang="yo")print(translated_file_handler.file_content)To translate a soup object, you can pass the soup object to thetranslatemethod as shown below:# Translating a soup object from English to Yorubatranslator=Translator(translation_engine="google")translated_soup=translator.translate(content=soup,src_lang="en",target_lang="yo")For specificity, you can use thetranslate_text,translate_markuportranslate_soupmethods to translate text, markup or a soup respectively.FileHandlerTheFileHandlerclass is used to handle files and perform basic read-write operations on supported file types.Example Usagefrombs4_web_scraper.file_handlerimportFileHandler# Instantiating a FileHandler objectfile_handler=FileHandler(file_path="path/to/file.txt")# Opening the filefile_handler.open_file()# Closing the filefile_handler.close_file()# Reading the filefile_content=file_handler.read_file(read_mode='r')print(file_content)# Writing to the filefile_handler.write_to_file(content="Hello World",write_mode='w')# Appending to the filefile_handler.write_to_file(content="Hello World",write_mode='a')# Copying the filefile_handler.copy_to(destination="path/to/copy.txt")# Moving the filefile_handler.move_to(destination="path/to/move.txt")# Clearing the file contentfile_handler.clear_file()# Deleting the filefile_handler.delete_file()# Getting the file typefile_type=file_handler.filetypeprint(file_type)# Getting the file namefile_name=file_handler.nameprint(file_name)# Getting the file pathfile_path=file_handler.filepathprint(file_path)LoggerThis class is used to log messages to a file and/or console. The following example shows how to log messages to a file:frombs4_web_scraper.loggingimportLogger# Instantiating a Logger objectlogger=Logger(name="example_logger",log_filepath="path/to/log.txt")# To log the message to the console alsologger.to_console=True# Set the log levellogger.set_base_level(level="DEBUG")# Log a messagelogger.log(message="Hello World")# Default log level is INFO# Log a message with a specific log levellogger.log(message="Hello World",level="DEBUG")The base log level can be set to any of the following:DEBUG,INFO,WARNING,ERRORorCRITICAL. The default log level isNOTSET. Theto_consoleattribute is set toFalseby default. To log messages to the console, set theto_consoleattribute toTrueas shown in the example above. You cannot log at levels below the base log level. For example, if the base log level is set toINFO, you cannot log atDEBUGlevel.CreditsBeautiful SoupRequestsTranslatorsContributions are welcome. Please fork the repository and submit a pull request.