Package: abstract-webtools
Abstract WebToolsProvides utilities for inspecting and parsing web content, including React components and URL utilities, with enhanced capabilities for managing HTTP requests and TLS configurations.Features:URL Validation: Ensures URL correctness and attempts different URL variations.HTTP Request Manager: Custom HTTP request handling, including tailored user agents and improved TLS security through a custom adapter.Source Code Acquisition: Retrieves the source code of specified websites.React Component Parsing: Extracts JavaScript and JSX source code from web pages.Comprehensive Link Extraction: Collects all internal links from a specified website.Web Content Analysis: Extracts and categorizes various web content components such as HTML elements, attribute values, attribute names, and class names.abstract_webtools.pyDescription:Abstract WebTools offers a suite of utilities designed for web content inspection and parsing. One of its standout features is its ability to analyze URLs, ensuring their validity and automatically attempting different URL variations to obtain correct website access. It boasts a custom HTTP request management system that tailors user-agent strings and employs a specialized TLS adapter for heightened security. The toolkit also provides robust capabilities for extracting source code, including detecting React components on web pages. Additionally, it offers functionalities for extracting all internal website links and performing in-depth web content analysis. This makes Abstract WebTools an indispensable tool for web developers, cybersecurity professionals, and digital analysts.Dependencies:requestssslHTTPAdapterfromrequests.adaptersPoolManagerfromurllib3.poolmanagerssl_fromurllib3.utilurlparse,urljoinfromurllib.parseBeautifulSoupfrombs4UrlManagerTheUrlManageris a Python class designed to handle and manipulate URLs. It provides methods for cleaning and normalizing URLs, determining the correct version of a URL, extracting URL components, and more. This class is particularly useful for web scraping, web crawling, or any application where URL management is essential.UsageTo use theUrlManagerclass, first import it into your Python script:fromabstract_webtoolsimportUrlManagerInitializing a UrlManager ObjectYou can create aUrlManagerobject by providing an initial URL and an optionalrequestssession. If no URL is provided, it defaults to 'www.example.com':url_manager=UrlManager(url='https://www.example.com')URL Cleaning and NormalizationTheclean_urlmethod takes a URL and returns a list of potential URL variations, including versions with and without 'www.', 'http://', and 'https://':cleaned_urls=url_manager.clean_url()Getting the Correct URLTheget_correct_urlmethod tries each possible URL variation with an HTTP request to determine the correct version of the URL:correct_url=url_manager.get_correct_url()Updating the URLYou can update the URL associated with theUrlManagerobject using theupdate_urlmethod:url_manager.update_url('https://www.example2.com')Extracting URL ComponentsTheurl_to_piecesmethod extracts various components of the URL, such as protocol, domain name, path, and query:url_manager.url_to_pieces()print(url_manager.protocol)print(url_manager.domain_name)print(url_manager.path)print(url_manager.query)Additional Utility Methodsget_domain_name(url): Returns the domain name (netloc) of a given URL.is_valid_url(url): Checks if a URL is valid.make_valid(href, url): Ensures a relative or incomplete URL is valid by joining it with a base URL.get_relative_href(url, href): Converts a relative URL to an absolute URL based on a base URL.Compatibility NoteTheget_domainmethod is kept for compatibility but is inconsistent. Use it only for "webpage_url_domain." Similarly,url_basename,base_url, andurljoinmethods are available for URL manipulation.ExampleHere's a quick example of using theUrlManagerclass:fromabstract_webtoolsimportUrlManagerurl_manager=UrlManager(url='https://www.example.com')cleaned_urls=url_manager.clean_url()correct_url=url_manager.get_correct_url()url_manager.update_url('https://www.example2.com')print(f"Cleaned URLs:{cleaned_urls}")print(f"Correct URL:{correct_url}")DependenciesTheUrlManagerclass relies on therequestslibrary for making HTTP requests. Ensure you have therequestslibrary installed in your Python environment.SafeRequestTheSafeRequestclass is a versatile Python utility designed to handle HTTP requests with enhanced safety features. It integrates with other managers likeUrlManager,NetworkManager, andUserAgentManagerto manage various aspects of the request, such as user-agent, SSL/TLS settings, proxies, headers, and more.UsageTo use theSafeRequestclass, first import it into your Python script:fromabstract_webtoolsimportSafeRequestInitializing a SafeRequest ObjectYou can create aSafeRequestobject with various configuration options. By default, it uses sensible default values, but you can customize it as needed:safe_request=SafeRequest(url='https://www.example.com')Updating URL and UrlManagerYou can update the URL associated with theSafeRequestobject using theupdate_urlmethod, which also updates the underlyingUrlManager:safe_request.update_url('https://www.example2.com')You can also update theUrlManagerdirectly:fromurl_managerimportUrlManagerurl_manager=UrlManager(url='https://www.example3.com')safe_request.update_url_manager(url_manager)Making HTTP RequestsTheSafeRequestclass handles making HTTP requests using thetry_requestmethod. It handles retries, timeouts, and rate limiting:response=safe_request.try_request()ifresponse:# Process the response hereAccessing Response DataYou can access the response data in various formats:safe_request.source_code: HTML source code as a string.safe_request.source_code_bytes: HTML source code as bytes.safe_request.source_code_json: JSON data from the response (if the content type is JSON).safe_request.react_source_code: JavaScript and JSX source code extracted from<script>tags.Customizing Request ConfigurationTheSafeRequestclass provides several options for customizing the request, such as headers, user-agent, proxies, SSL/TLS settings, and more. These can be set during initialization or updated later.Handling Rate LimitingThe class can handle rate limiting scenarios by implementing rate limiters and waiting between requests.Error HandlingTheSafeRequestclass handles various request-related exceptions and provides error messages for easier debugging.DependenciesTheSafeRequestclass relies on therequestslibrary for making HTTP requests. Ensure you have therequestslibrary installed in your Python environment:pipinstallrequestsExampleHere's a quick example of using theSafeRequestclass:fromabstract_webtoolsimportSafeRequestsafe_request=SafeRequest(url='https://www.example.com')response=safe_request.try_request()ifresponse:print(f"Response status code:{response.status_code}")print(f"HTML source code:{safe_request.source_code}")SoupManagerTheSoupManagerclass is a Python utility designed to simplify web scraping by providing easy access to the BeautifulSoup library. It allows you to parse and manipulate HTML or XML source code from a URL or provided source code.UsageTo use theSoupManagerclass, first import it into your Python script:fromabstract_webtoolsimportSoupManagerInitializing a SoupManager ObjectYou can create aSoupManagerobject with various configuration options. By default, it uses sensible default values, but you can customize it as needed:soup_manager=SoupManager(url='https://www.example.com')Updating URL and Request ManagerYou can update the URL associated with theSoupManagerobject using theupdate_urlmethod, which also updates the underlyingUrlManagerandSafeRequest:soup_manager.update_url('https://www.example2.com')You can also update the source code directly:source_code='<html>...</html>'soup_manager.update_source_code(source_code)Accessing and Parsing HTMLTheSoupManagerclass provides easy access to the BeautifulSoup object, allowing you to search, extract, and manipulate HTML elements easily. You can use methods likefind_all,get_class,has_attributes, and more to work with the HTML content.elements=soup_manager.find_all(tag='a')Extracting LinksThe class also includes methods for extracting all website links from the HTML source code:all_links=soup_manager.all_linksExtracting Meta TagsYou can extract meta tags from the HTML source code using themeta_tagsproperty:meta_tags=soup_manager.meta_tagsCustomizing ParsingYou can customize the parsing behavior by specifying the parser type during initialization or updating it:soup_manager.update_parse_type('lxml')DependenciesTheSoupManagerclass relies on theBeautifulSouplibrary for parsing HTML or XML. Ensure you have thebeautifulsoup4library installed in your Python environment:pipinstallbeautifulsoup4ExampleHere's a quick example of using theSoupManagerclass:fromabstract_webtoolsimportSoupManagersoup_manager=SoupManager(url='https://www.example.com')all_links=soup_manager.all_linksprint(f"All Links:{all_links}")LinkManagerTheLinkManagerclass is a Python utility designed to simplify the extraction and management of links (URLs) and associated data from HTML source code. It leverages other classes likeUrlManager,SafeRequest, andSoupManagerto facilitate link extraction and manipulation.UsageTo use theLinkManagerclass, first import it into your Python script:fromabstract_webtoolsimportLinkManagerInitializing a LinkManager ObjectYou can create aLinkManagerobject with various configuration options. By default, it uses sensible default values, but you can customize it as needed:link_manager=LinkManager(url='https://www.example.com')Updating URL and Request ManagerYou can update the URL associated with theLinkManagerobject using theupdate_urlmethod, which also updates the underlyingUrlManager,SafeRequest, andSoupManager:link_manager.update_url('https://www.example2.com')Accessing Extracted LinksTheLinkManagerclass provides easy access to extracted links and associated data:all_links=link_manager.all_desired_linksCustomizing Link ExtractionYou can customize the link extraction behavior by specifying various parameters during initialization or updating them:link_manager.update_desired(img_attr_value_desired=['thumbnail','image'],img_attr_value_undesired=['icon'],link_attr_value_desired=['blog','article'],link_attr_value_undesired=['archive'],image_link_tags='img',img_link_attrs='src',link_tags='a',link_attrs='href',strict_order_tags=True,associated_data_attr=['data-title','alt','title'],get_img=['data-title','alt','title'])DependenciesTheLinkManagerclass relies on other classes within theabstract_webtoolsmodule, such asUrlManager,SafeRequest, andSoupManager. Ensure you have these classes and their dependencies correctly set up in your Python environment.ExampleHere's a quick example of using theLinkManagerclass:fromabstract_webtoolsimportLinkManagerlink_manager=LinkManager(url='https://www.example.com')all_links=link_manager.all_desired_linksprint(f"All Links:{all_links}")##Overall Usecasesfromabstract_webtoolsimportUrlManager,SafeRequest,SoupManager,LinkManager,VideoDownloader# --- UrlManager: Manages and manipulates URLs for web scraping/crawling ---url="example.com"url_manager=UrlManager(url=url)# --- SafeRequest: Safely handles HTTP requests by managing user-agent, SSL/TLS, proxies, headers, etc. ---request_manager=SafeRequest(url_manager=url_manager,proxies={'8.219.195.47','8.219.197.111'},timeout=(3.05,70))# --- SoupManager: Simplifies web scraping with easy access to BeautifulSoup ---soup_manager=SoupManager(url_manager=url_manager,request_manager=request_manager)# --- LinkManager: Extracts and manages links and associated data from HTML source code ---link_manager=LinkManager(url_manager=url_manager,soup_manager=soup_manager,link_attr_value_desired=['/view_video.php?viewkey='],link_attr_value_undesired=['phantomjs'])# Download videos from provided links (list or string)video_manager=VideoDownloader(link=link_manager.all_desired_links).download()# Use them individually, with default dependencies for basic inputs:standalone_soup=SoupManager(url=url).soupstandalone_links=LinkManager(url=url).all_desired_links# Updating methods for manager classesurl_1='thedailydialectics.com'print(f"updating URL to{url_1}")url_manager.update_url(url=url_1)request_manager.update_url(url=url_1)soup_manager.update_url(url=url_1)link_manager.update_url(url=url_1)# Updating URL manager referencesrequest_manager.update_url_manager(url_manager=url_manager)soup_manager.update_url_manager(url_manager=url_manager)link_manager.update_url_manager(url_manager=url_manager)# Updating source code for managerssource_code_bytes=request_manager.source_code_bytessoup_manager.update_source_code(source_code=source_code_bytes)link_manager.update_source_code(source_code=source_code_bytes)LicenseThis project is licensed under the MIT License - see theLICENSEfile for details.Module Information-Author: putkoff
-Author Email:partners@abstractendeavors.com-Github:https://github.com/AbstractEndeavors/abstract_essentials/tree/main/abstract_webtools-PYPI:https://pypi.org/project/abstract-webtools-Part of: abstract_essentials
-Date: 10/10/2023
-Version: 0.1.4.54