sc3/cookcountyjail — pull request #80 by nwinklareth (2013-07-21 22:14:06+00:00)
Title: Improved robustness and added parallel scrapping

I have tested the robustness improvements.

You need to add GNU Parallel to the server machine. Once you have then edit cron.sh, remove the current call to scrape_inmates and uncomment the following code that runs scrapping in parallel, also uncomment the line that call discharge_inmates. Note that the script assumes that there are 20 database connections allowed on your Postgres server if the number is less than need to either increase it or reduce the number of parallel jobs in the script.

I have tested parallel scrapping from the command line went from just under 5 hours to 46 minutes.

I have not tested the cron script, nor have I tested the inner logic of the discharge_inmate script, my code inspection indicates that it is correct. The inmate selection code has been tested.

I recommend after installation of GUN Parallel program and changing the cron.sh script and checking that database connections are correctly set, that you do the following:
1) comment out the line that calls discharge_inmates in cron.sh
2) run cron.sh from the command line
3) while that is running, do a code review of discharge_inmate command, fix any errors you find
4) once cron.sh is finished running check the database, use 26thandcalifornia webapp, in particular the
general stats options
5) If it looks good run discharge_inmates from the command line
6) if it runs correctly uncomment the line calling discharge_inmates and this is the new cron.sh
7) if it did not run correctly, restore the original call to scrape_inmates and comment out the new parallel code in
cron,sh and let me know what the errors are.