Package: scraparser
ScraparserA generic PDF table scraper and parser for data analysis.Originally written for scraping and parsing Hong Kong governmentCOVID-19related public data. Now generalize for hopefully other research
purposes as well.Package is available onpypi.org. The development is
onGitLab. You are welcome
to submitissueandmerge request.
And should you want to contribute, please read theDevelopmentsection.PrerequisitesTo use scraparser, you needPython 3installed to your system. You will also need to know how to use
terminal commands on your system.The instructions below assumes that Python 3 is available to
you through the commandpython3. If it is only available to
you as commandpythonor other name, simply usepythonor
the available name for the commandspython3described below.InstallThe recommended way is to installthe PyPi packagewith
the [pip] module:python3 -m pip install --upgrade scraparserExample UseBasic ScrapingTo scrap the latest location situation report:python3 -m scraparser scrap "https://www.chp.gov.hk/files/pdf/local_situation_covid19_tc.pdf" \
| python3 -m scraparser parse-pdf-to-csv --headers="個案編號,報告日期,發病日期,性別,年齡,入住醫院名稱,住院/出院/死亡,香港/非香港居民,個案分類,確診/疑似個案"The downloaded PDF file and the parsed CSV file will be stored in:./data/local_situation_covid19_tc.<time-string>.pdf
./data/local_situation_covid19_tc.<time-string>.csvThetime-stringwill be formated asYYYY-MM-DD-HHmmss.Parse Previously Downloaded PDF ReportTo parse pre-exist PDF file from your local computer:python3 -m scraparser scrap-location-situation-pdf --file=path/to/somename.pdfThe parsed CSV file will be stored to "path/to/somename.csv"Utility to Fix or Modify Parsed CSVIts highly difficult to correctly read tables from PDF files. Common errors include:Column underflow / overflowThe content of a cell spilled over to the last or next cellRow overflowThe content of a cell (usually with line wraped into multiple lines), spilled over
to create a phantom row with only 1 content-filled cell.To fix these issue, please use the following subcommands:sortThe command takes CSV filenames either from arguments or fromSTDIN(one filename)
per line:python3 -m scraparser sort --column=0 --sort-as-number --in-place ./data/local_situation_covid19_tc.<time-string>.csvThis command will:Read the file and parse 1st column (parameter--columnaccepts column definition start with 0, like in Python list index)Sort all rows by the 1st column.Save the fix result back to the input file.fix column-underflowThe command takes CSV filenames either from arguments or fromSTDIN(one filename)
per line:python3 -m scraparser fix column-underflow --column=5 --in-place ./data/local_situation_covid19_tc.<time-string>.csvThis command will:Automatically read all the valid contents in the 6th column (parameter--columnaccepts column definition start with 0, like in Python list index).Read every row and check if a cell in that column is empty (math.isnan()).If so, check the column before it (6th column for our case) and see if it is
suffixed by any valid content found in step (1).Split the content correctly for the 6th and 7th column.Save the fix result back to the input file.fix date-column-underflowThe command takes CSV filenames either from arguments or fromSTDIN(one filename)
per line:python3 -m scraparser fix date-column-underflow --column=1 --format=DD/MM/YYYY --in-place ./data/local_situation_covid19_tc.<time-string>.csvThis command will:Read every row and check if a cell in the 2nd column is empty (math.isnan()).If so, check the column before it (1st column for our case) and see if it is
suffixed by string that matches our specified date format.Split the content correctly for the 1st and 2nd column.Save the fix result back to the input file.fix date-column-overflowThe command takes CSV filenames either from arguments or fromSTDIN(one filename)
per line:python3 -m scraparser fix date-column-overflow --column=1 --format=DD/MM/YYYY --in-place ./data/local_situation_covid19_tc.<time-string>.csvThis command will:Read every row and check if a cell in the column before target column
(i.e. 1st column in our case) to see is empty (math.isnan()).If so, check the target column and see if it is some string
suffixed by date string of specified format.Split the content correctly for the 1st and 2nd column.Save the fix result back to the input file.fix empty-rowsThe command takes CSV filenames either from arguments or fromSTDIN(one filename)
per line:python3 -m scraparser fix empty-rows --in-place ./data/local_situation_covid19_tc.<time-string>.csvThis command will:Read every row and find all rows with all but 1 cell empty (math.isnan()).If so, append the content of that 1 cell to the cell directly above it.Drop all "phantom rows" found in step (1).Save the fix result back to the input file.Advanced Piping usageParse and Show Result DataTo correctly fix all the issue created from the parsed CSV file in local situation report:Linuxpython3 -m scraparser scrap "https://www.chp.gov.hk/files/pdf/local_situation_covid19_tc.pdf" \
| python3 -m scraparser parse-pdf-to-csv --headers="個案編號,報告日期,發病日期,性別,年齡,入住醫院名稱,住院/出院/死亡,香港/非香港居民,個案分類,確診/疑似個案" \
| python3 -m scraparser fix date-column-underflow --column=1 --in-place \
| python3 -m scraparser fix date-column-overerflow --column=1 --in-place \
| python3 -m scraparser fix column-underflow --column=6 --in-place \
| python3 -m scraparser fix column-underflow --column=5 --in-place \
| python3 -m scraparser fix empty-rows --in-place \
| python3 -m scraparser sort --in-place \
| xargs -i xdg-open "{}"macospython3 -m scraparser scrap "https://www.chp.gov.hk/files/pdf/local_situation_covid19_tc.pdf" \
| python3 -m scraparser parse-pdf-to-csv --headers="個案編號,報告日期,發病日期,性別,年齡,入住醫院名稱,住院/出院/死亡,香港/非香港居民,個案分類,確診/疑似個案" \
| python3 -m scraparser fix date-column-underflow --column=1 --in-place \
| python3 -m scraparser fix date-column-overerflow --column=1 --in-place \
| python3 -m scraparser fix column-underflow --column=6 --in-place \
| python3 -m scraparser fix column-underflow --column=5 --in-place \
| python3 -m scraparser fix empty-rows --in-place \
| python3 -m scraparser sort --in-place
| xargs -I{} open "{}"Parse Data then Update Google SheetThis will overwrite the current data specified in the range. If there are not enough rows in
the Google Sheet, the file will be expanded automatically.Presume you have defined the string$GOOGLE_SHEET_IDand the target sheet
'CHP/DH Local Situation Input' exists:python3 -m scraparser scrap "https://www.chp.gov.hk/files/pdf/local_situation_covid19_tc.pdf" \
| python3 -m scraparser parse-pdf-to-csv --headers="個案編號,報告日期,發病日期,性別,年齡,入住醫院名稱,住院/出院/死亡,香港/非香港居民,個案分類,確診/疑似個案" \
| python3 -m scraparser fix date-column-underflow --column=1 --in-place \
| python3 -m scraparser fix date-column-overerflow --column=1 --in-place \
| python3 -m scraparser fix column-underflow --column=6 --in-place \
| python3 -m scraparser fix column-underflow --column=5 --in-place \
| python3 -m scraparser fix empty-rows --in-place \
| python3 -m scraparser sort --in-place \
| python3 -m scraparser googlesheet "$GOOGLE_SHEET_ID" update --range="'CHP/DH Local Situation Input'!A2:Z"DevelopmentFirst clone this repository by:git clone https://gitlab.com/yookoala/scraparser.git
cd scraparserYou are recommended to usevenvfor the development environment.First you would need to initialize venv and install all the packages specified
in therequirements.txt:pip -m venv .venv
. ./bin/activate.sh
pip install -r requirements.txtOnce this is done, you are ready to run the package in the repository folder
as if the module was installed locally:python3 -m scraparser <command>You can change thescraparserfolder in this repository and this
command will run correctly.Build and SubmitShould you want to fork and create your ownscraparserpackage on Python Package index,
you may build and release your package (requiresmake) with commands.BuildingTo build the package for upload, you need to rename the package to something other
thanscraparser. Let's say you would suffix the package withYOURNAME:PYPI_PKG_NAME=scraparser-YOURNAME make clean distThe default version is specified by git commands. If it fails to work, you may
force a version string on it:PYPI_PKG_VERSION=0.5.0 PYPI_PKG_NAME=scraparser-YOURNAME make clean distPlease note that the version stringMUSTfollow thePEP 440convension
or it cannot be submitted.Submitting to test.pypi.orgPYPI_TEST_PASSWORD=<your-pypi-test-token> make upload-testSubmitting to pypi.orgPYPI_PASSWORD=<your-pypi-test-token> make uploadLicenseLicense underthe MIT License. You may obtain the license in this repository.