Package: polipy
polipyPoliPy is a Python library that provides a command-line interface (CLI) and an API to scrape, parse, and analyze privacy policies of different services. It is a maintained library developed as a collaborative effort of researchers at theBerkeley Lab for Usable and Experimental Security (BLUES)at theUniversity of California, Berkeley.Please read carefully to learn more about properlycitingthis library and theGNU GPLv3terms that govern the usage and modification of this software.PrerequisitesPoliPy usesseleniumlibrary to dynamically scrape webpages that host privacy policies. This allows PoliPy to download policies hosted on pages that, for instance, require JavaScript to be enabled.Seleniumwill open the webpage in a private window of Firefox Browser in the background using ageckodriverprocess, allowing the Python script to control the behavior of the browser without having the user interact with it directly.To ensure that PoliPy works as intended, first make sure to installthe latest version of the Mozilla Firefox browser. Afterwards, downloadthe latest verison of geckodriverfor your operating system. You will need to ensure that thegeckodriveris on the path and can be executed. Refer toselenium Python API documentationfor more information and troubleshooting.InstallationYou can easily install the library usingpip:pipinstallpolipyYou might encounter the following prompt the first time you run PoliPy on MacOS (or a similar prompt on another OS):This prompt appears as the library is making network requests to the webpages where privacy policy are hosted. SelectAllowto continue.ExampleThis library can either be used as a command-line interface (CLI):$catpolicies.txt
https://docs.github.com/en/github/site-policy/github-privacy-statement

$polipypolicies.txt-sor as an API imported by another module:importpolipyurl='https://docs.github.com/en/github/site-policy/github-privacy-statement'result=polipy.get_policy(url,screenshot=True)result.save(output_dir='.')Both of these result in the creation of the following output folder:├──docs_github_com_c0eb432555
│├──20210511.html
│├──20210511.png
│├──20210511.json
├──└──20210511.metawhere the base file name corresponds to the date the policy was scraped and the file extensions correspond to the following:.htmlcontains the (dynamic) source of the webpage where privacy policy is hosted.pngcontains the screenshot of the webpage.metacontains information such as the URL of the privacy policy and the date of last scraping.jsoncontains the content extracted from the privacy policy.For instance, thetextkey of the JSON in the.jsonfile contains the extracted text from the scraped privacy policy:GitHub Privacy Statement - GitHub Docs
  GitHub Docs
  All products
  GitHub.com
  Getting started
  ...
  Effective date: December 19, 2020
  Thanks for entrusting GitHub Inc. (“GitHub”, “we”) with your source code, your projects, and your personal information. Holding on to your private information is a serious responsibility, and we want you to know how we're handling it.
  All capitalized terms have their definition in
  GitHub’s Terms of Service , unless otherwise noted here.
  ...
  Contact GitHub
  Pricing
  Developer API
  Training
  AboutUsageCLI usage manual is available with the--helpor-hflag:$polipy--help
usage:__main__.py[-h][--output_dirOUTPUT_DIR][--timeoutTIMEOUT][--screenshot][--extractorsEXTRACTORS[EXTRACTORS...]][--workersWORKERS][--force][--raise_errors][--verbose]input_file

DownloadprivacypoliciesfromURLscontainedintheinput_file.

positionalarguments:input_filePathtofilecontainingalistofnewline-separatedURLsofprivacypoliciestoscrape.

optionalarguments:-h,--helpshowthishelpmessageandexit--output_dirOUTPUT_DIR,-oOUTPUT_DIRPathtodirectorywherepolicieswillbesaved(defaultis...).--timeoutTIMEOUT,-tTIMEOUTTheamountoftimeinsecondstowaitfortheHTTPrequestresponse(defaultis30).--screenshot,-sCaptureandsavethescreenshotoftheprivacypolicypage(defaultisFalse).--extractorsEXTRACTORS[EXTRACTORS...],-eEXTRACTORS[EXTRACTORS...]Extractorstousetocaptureinformationfromtheprivacypolicy(defaultis[text]).--workersWORKERS,-wWORKERSNumberofthreadingworkerstouse(defaultis1).--force,-fScrapeprivacypolicyagainevenifitisalreadyscrapedorhasnotbeenupdated(defaultisFalse).--raise_errors,-rRaiseerrorsthatoccurduringthescrapingandparsing(defaultisFalse).--verbose,-vEnableverboselogging(defaultisFalse).The following helper methods are available when the PoliPy library is imported:get_policy: Helper method that returns apolipy.Policyobject containing
information about the policy, scraped and processed from the given URL.download_policy: Helper method that scrapes, parses, and saves the privacy policy located at the provided .Additionally, you can directly createpolipy.Policyobjects supporting the following interface:Policy.__init__: Constructor method.Policy.scrape: Obtains the page source of the given privacy policy URL.Policy.extract: Extracts information from the scraped privacy policy.Policy.save: Saves the information contained in thePolicyobject.Policy.to_dict: Converts thePolicyobject to a dictionary.get_policyHelper method that returns apolipy.Policyobject containing information about the policy, scraped and processed from the given URL.Parameters:url (str): The URL of the privacy policy.screenshot (bool, optional): Flag that indicates whether to capture and save the screenshot of the privacy policy page (default isFalse).timeout (int, optional): The amount of time in seconds to wait for the HTTP request response (default is30).extractors (list of str, optional): Extractors to use to capture information from the privacy policy (default is["text"]).Returns:polipy.Policy: Object containing information about the privacy policy.Raises:polipy.NetworkIOException: Raised if an error has occurred while performing networking I/O.polipy.ParserException: Raised if an error occurred while extracting text from page source.download_policyHelper method that scrapes, parses, and saves the privacy policy located at the providedurlby creating the following directory structure:├── <output_dir>
│   ├── <policy URL domain>_<hash of policy URL>
│   │   ├── <current UTC date>.html
│   │   ├── <current UTC date>.json
└── └── └── <current UTC date>.metaParameters:url (str): The URL of the privacy policy.output_dir (str, optional)Path to directory where the policy will be saved (default is the current working directory).force (bool, optional): Flag that indicates whether to scrape privacy policy again even if it is already scraped or has not been updated (default isFalse).raise_errors (bool, optional): Flag that indicates whether to raise errors that occur during the scraping and parsing (default isFalse).logger (logging.Logger, optional): Alogging.Loggerobject to handle the logging of events (default isNone).screenshot (bool, optional): Flag that indicates whether to capture and save the screenshot of the privacy policy page (default isFalse).timeout (int, optional): The amount of time in seconds to wait for the HTTP request response (default is30).extractors (list of str, optional): Extractors to use to capture information from the privacy policy (default is["text"]).Raises:polipy.NetworkIOException: Raised if an error has occurred while performing networking I/O.polipy.ParserException: Raised if an error occurred while extracting text from page source.PolicyA class representing a privacy policy. Attributes:url (dict): Contains the URL to the privacy policy and additional information about the URL, such as domain, scheme, content-type, etc.source (dict): Contains the information scraped from the webpage where policy is hosted, such as the HTML (dynamic) source and the screenshot.content (dict): Contains the content extracted from the privacy policy website, such as the text of the policy.__init__Constructor method. Populates thePolicy.urlattribute. Parameters:url (str): The URL of the privacy policy.scrapeObtains the page source of the given privacy policy URL. Populates thePolicy.sourceattribute. Parameters:screenshot (bool, optional): Flag that indicates whether to capture and save the screenshot of the privacy policy page (default isFalse).timeout (int, optional): The amount of time in seconds to wait for the HTTP request response (default is30).Returns:polipy.Policy:Policyobject with the populated attribute.Raises:polipy.NetworkIOException: Raised if an error has occurred while performing networking I/O.extractExtracts information from the scraped privacy policy. Populates thePolicy.contentattribute. Parameters:extractors (list of str, optional): Extractors to use to capture information from the privacy policy (default is["text"]).Returns:polipy.Policy:Policyobject with the populated attribute.Raises:polipy.ParserException: Raised if an error occurred while extracting text from page source.saveSaves the information contained in thePolicyobject. Parameters:output_dir (str): Path to directory where the policy will be saved.to_dictConverts thePolicyobject to a dictionary. Returns:dict: Dictionary containing policy attributes as key-value pairs.ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.CitationsAny project that uses this library in part or in whole is required to acknowledge the usage of this library. For publications, the following citation can be used:Samarin, N., Kothari, S., Siyed, Z., Wijesekera, P., Fischer, J., Hoofnagle, C. and Egelman, S., Investigating the Compliance of Android App Developers with the CCPA.LicenseGNU GPLv3