Package: gdprCrawlerTest19
GDPRxiv Crawler (README)An efficient tool to crawl GDPR legal documents!About The ProjectWith the introduction of the Europeans Union's General Data Protection Regulation (GDPR), there has been an explosion in the number of legal
documents pertaining to case reviews, analyses, legal decisions, etc... that mark the enforcement of the GDPR.
Additionally, these documents are spread across over 30 Data Protection (DPA) and Supervisory Authorities. As a result, it is
cumbersome for researchers/legal teams to access and download a large quantity of GDPR documents at once.To address this, we have created GDPRxiv Crawler, a command-line tool that allows users to efficiently filter and
download GDPR documents. Users may select their desired DPA and document_type, and GDPRxiv Crawler will scrape the web
and download all up-to-date documents.Of course, it is impossible to entirely keep up with DPA website redesigns and newly added document categories.
However, we hope that this tool will eliminate the bulk of the workload and allow users to focus on more important tasks.Built WithBeautifulSoup4SeleniumAdd more later...Getting StartedPrerequisitesPython 3.9 is required. This python version includes the pip installer and the venv module, which is needed to create a
virtual environment.It is strongly recommended that users utilize a virtual environment when installing this package.
See below to create and activate one.In a directory:venv:virtualenv<virtualenvname>Activate the virtual environment:source<virtualenvname>/bin/activateInstallationAt any moment, use command 'pip3 list' to view all installed packages.Downloadrequirements.txtand place it in the directory that contains the virtual environment.Install package requirementspip3install-rrequirements.txtInstall the GDPRxiv Crawler packagepip3install-ihttps://test.pypi.org/simple/gdprCrawlerTest15UsageDownloaded documents will be organized into a set of folders based on DPA and document type.A file called visitedDocs.txt is always created upon an initial run within a new directory. This file records each downloaded document's
unique hash, which allows the tool to avoid overwriting existing documents (if desired) in future runs.Scrape desired documents:gdprCrawlerscrape--country<countryname>--document_type<documenttype>--path<directorytostoredocuments>The same directory can be used for multiple countries: the scraper automatically organizes documents based on country and document type.Optionally, the --overwrite argument can be included if users would like to overwrite existing documents:gdprCrawlerscrape--country<countryname>--document_type<documenttype>--path<directorytostoredocuments>--overwrite<True/False>Overwrite is False by default.Country and document type arguments should be written exactly as they appear below:SUPPORTED COUNTRIES:     DOCUMENTS TYPES:

        Austria                  Decisions
        Belgium                  Annual Reports, Decisions, Opinions
        Bulgaria                 Annual Reports, Opinions
        Croatia                  Decisions
        Cyprus                   Annual Reports, Decisions
        Czech Republic           Annual Reports, Completed Inspections, Court Rulings, Decisions, Opinions, Press Releases
        Denmark                  Annual Reports, Decisions, Permissions
        EDPB (Agency)            Annual Reports, Decisions, Guidelines, Letters, Opinions, Recommendations
        Estonia                  Annual Reports, Instructions, Prescriptions
        Finland                  Advice, Decisions, Guides, Notices
        France                   FUTURE UPDATE
        Germany                  N/A
        Greece                   Annual Reports, Decisions, Guidelines, Opinions, Recommendations
        Hungary                  Annual Reports, Decisions, Notices, Recommendations, Resolutions
        Ireland                  Decisions, Judgements, News
        Italy                    Annual Reports, Hearings, Injunctions, Interviews, Newsletters, Publications
        Latvia                   Annual Reports, Decisions, Guidances, Opinions, Violations
        Lithuania                Decisions, Guidelines, Inspection Reports
        Luxembourg               Annual Reports, Opinions
        Malta                    Guidelines, News Articles
        Netherlands              Decisions, Opinions, Public Disclosures, Reports
        Poland                   Decisions, Tutorials
        Portugal                 Decisions, Guidelines, Reports
        Romania                  Decisions, Reports
        Slovakia                 Fines, Opinions, Reports
        Slovenia                 Blogs, Guidelines, Infographics, Opinions, Reports
        Spain                    Blogs, Decisions, Guides, Infographics, Reports
        Sweden                   Decisions, Guidances, Judgements, Publications
        United Kingdom           Decisions, Judgements, NoticesContributingAll suggestions and contributions you make aregreatly appreciated.LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.ContactProject Link:https://github.com/GDPRxiv/crawlerAcknowledgmentsThank you to everyone who has supported the project in any way. We greatly appreciate your time and effort!Choose an Open Source LicenseImg Shields(back to top)