Package: scrapdynamics
ScrapDynamicsTable of ContentsIntroductionInstallationGetting StartedCLILibraryAdvance UsageSettingsFeaturesExamplesIntroductionScrapDynamics is a powerful framework for exploring and crawling websites, extracting links and finding information using regex expressions, making it easy to automate the process of collecting any data on a web.InstallationTo install ScrapDynamics, you can use the following commands:gitclonehttps://github.com/GuyChahine/ScrapDynamics.gitpipinstall-rrequirements.txtGetting StartedCLITo use ScrapDynamics from the command line interface, you can use the following command:python-mscrapdynamics-uhttps://example.org-o./results.jsonIn this example it starts the crawler at the specified URL "https://example.org" and saves the results in a JSON file at the path "./results.json".You can also use the following options:usage:ScrapDynamics[-h][-v][-uURL][-oOUTPUT]options:-h--helpshowthishelpmessageandexit-v--versionshowversion-uURL--urlURLbaseurl-oOUTPUT--outputOUTPUTpath/filename.extentionLibraryWhen using ScrapDynamics as a library, you can import it into your Python code and use the following code snippet:importscrapdynamicsassdcrawler=sd.Crawler("https://example.org")crawler.start()crawler.to_json("./results.json")This code creates a Crawler object with the specified URL, starts the crawling process, and saves the results in a JSON file.Advance UsageSettingsYou can customize the behavior of ScrapDynamics by modifying the settings. Here's an example of how to create a Settings object and set various options:fromscrapdynamics.settingsimportSettingssettings=Settings()You can modify the following settings:link_findall:regex expression to find linkslink_findall="href=\"((?:https?|\/\w|\/\/\w).+?)\""link_relative_sub:regex expression to substitute relative linkslink_relative_sub=["(^\/\w.+?$)","https://{domain}\1"]link_schema_relative_sub:regex expression to substitute schema relative linkslink_schema_relative_sub=["(^(?:\/\/\w).*?$)","https:\1"]domain_findall:regex expression to find domain from a urldomain_findall="https?:\/\/(?:www\.)?([^\/\s\'\"]+)"search_expressions:dict of regex expression to look for in the html pagesearch_expressions={"title":"(?:<title>|<meta.*?property=\"og:title\".*?content=\")(.*?)(?:<\/title>|\".*?>)","emails":"[\w\-\.]+?\@[\w\-\.]+?\.[\w]+","phones":"(?:tel\:)(\+?[\d\-\ ]{6,20})(?!\d)",}restrict_to_domain:restrict future urls to the domain given at the startrestrict_to_domain=Truedepth:max depth to crawldepth=1simulate_human:use selenium webdriver to get html pagesimulate_human=Falsescroll_first_page:selenium webdriver scroll down the first url given at the startscroll_first_page=Falsescroll_all_page:selenium webdriver scroll down all the url foundscroll_all_page=Falseheadless:don't show the page of selenium webdriverheadless=Falseget_timeout:time in seconds of a GET timeoutget_timeout=3progress_bar:use progress bar or simple printsprogress_bar=Falserequest_header:header to add when doing a GET request with the module requestsrequest_header={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"}valid_content_type:content type of page content to allow the crawler to explorevalid_content_type=["text/html"]xpath_restrict_link_crawl:xpath where children elements will be used to find links for depth 1xpath_restrict_link_crawl="/html"Here's an example of how to use the Settings object with ScrapDynamics:importscrapdynamicsassdfromscrapdynamics.settingsimportSettingssettings=Settings(progress_bar=True)crawler=sd.Crawler("https://example.org",settings)crawler.start()print(crawler.show())This code creates a Settings object with the progress_bar option set to True, creates a Crawler object with the specified URL and settings, starts the crawling process, and displays the results.FeaturesRegex-based Information Extraction:ScrapDynamics supports the use of regular expressions to search for specific information within the explored website. In addition to the regular expression patterns already implemented, you can define custom regular expression patterns and extract any other structured information.Website Crawling:ScrapDynamics provides a robust web crawling functionality that allows you to navigate through a website and discover all its accessible pages. It follows links, collects URLs, and traverses the website structure efficiently.Customizable Scraping Rules:You have full control over the scraping process. You can define the starting URL, specify the depth of crawling, set exclusion rules for certain URLs, and fine-tune the behavior of the crawler according to your requirements.Data Export:The extracted information can be easily exported to various formats, such as EXCEL, CSV or JSON, allowing you to further analyze or integrate the scraped data into your existing workflows.ExamplesLinkedin Job ScrappingIn this example, ScrapDynamics is set for scraping job details. It include link of the job, job title, description of the job, company name, job location, time of posting, and save the results in an Excel file.