Rearranged some of the automatically generated mysql dump strategy related mysqld instructions to enable removal of Account table from Edx db after load.
Fixed very hidden duplicate key error. Occurred when JSON parsing errors happened, e.g. when the strings were cut off. A finally clause slipped a bad INSERT statement into the sql output.
Started working on log file pull, and added transformGivenLogFiles.sh
Added scripts/transformGivenLogfiles.sh again, b/c datastage doesn't see it.
Checkpoint: Two thirds of log file pulling from S3 done.
In pullLatestLogFiles.py, finding new S3 files works. Loading those down works. Call to transform written, not tested. To do: the load command, and a nice CLI with help to operate the script. Test as CRON job. Elsewhere, fixed some issues.
*Hope* to have recovered from somehow getting into a git 'no branch' state. Created branch Temp while in 'no branch state. Then checked out master, and merged Temp. I so hope this worked OK. Will run tests.