Package: pdpc-decisions
pdpc-decisionsThis package contains utilities which allow you to create a corpus of decisions from the
Personal Data Protection Commission of Singapore'sData Protection Enforcement Cases.The primary use of such a corpus is for studying, possibly using data science tools such as
natural language processing.It currently has the following features:Visit the Personal Data Protection Commission of Singapore'sData Protection Enforcement Casesand compile a table of decisions with information from the summaries provided by the PDPC for each case.Save this table of decisions as CSVDownload all the PDF files of the decisions from the PDPC's website.
If the decision is not a PDF, collects the information provided on the decision web page and saves it as a text file.Convert the PDF files into text filesFeatures provided by scraperPublished dateRespondentTitleSummaryURL of PDF of decisionThe features are discovered by passing--extrasto the command.[Extras]Citation[Extras]Basic enforcement information (Financial penalty, warning, directions)[Extras]References (referred by, referring to)What pdpc-decisions usesPython 3PDF MinerSeleniumChromespaCyInstallationDocker ImageI dockerised the application for my personal ease of use.
It is probably the easiest and most straight-forward way
to use the application and I recommend it too.
The dockerised application also contains all pre-requisites
so there is no need for any manual installs.You need to have docker installed.
Pull the image fromdocker hub.dockerpullhoufu/pdpc-decisionsAfter that you can run the image and pass commands and arguments to it.
For example, if you would like the application to do all actions.dockerrunhoufu/pdpc-decisionsallThis isn't clever because downloads will be stored in the docker image
and not easily accessed. Bind a volume in your filesystem and
use the--rootoption to direct the application
to save the files there. For example:dockerrun\--mounttype=bind,source="$(pwd)"/target,target=/code/download\# Target directory must exist!houfu/pdpc-decisions\all\--root/code/download/Local installInstall via PIPpipinstallpdpc-decisionsOnce the package is installed, used the command line toolpdpc-decisionsto use the script.If necessary, installChromeandChromeDriverfor Selenium to work.The main entry point for the script ispdpcdecision.pyUsageThe script accepts the following actions and options:Accepts the following actions."all"       Does all the actions (scraping the website, saving a csv,
downloading all files and creating a corpus)."corpus"    Converts PDF format of decisions into plain text files."csv"      Save the items gathered by the scraper as a csv file."files"     Downloads all the decisions from the PDPC website into a
folder.Options:--csv FILEFilename for saving the items gathered by scraper as a
csv file.  [default: scrape_results.csv]--download DIRECTORYDestination folder for downloads of all PDF/web pages
of PDPC decisions  [default: download/]--corpus DIRECTORYDestination folder for PDPC decisions converted to
text files  [default: corpus/]-r, --root DIRECTORYRoot directory for downloads and files  [default:
Your current working directory]--extras/--no-extrasAdd extra features to the data collected. This increases processing time. This feature is ignored if action isfilesordownloads.
(Experimental and requires reading of actual decisions)
[default:False, '--no-extras']--extra-corpus/--no-extra-corpusEnable experimental features for corpus.
This increases processing time.--verboseVerbose output--helpShow this message and exit.ContactFeel free to let me have your suggestions, comments or issues using the issue tracker or byemailing me.It would also be nice to hear how you have used this corpus by using the above contacts.