Package: energy-dashboard-client
Energy Dashboard Command Line Interface (edc)Command Line Interface for the Energy Dashboard.Status : PRE ALPHAStagesdownload  : doneparse     : doneinsert    : inproc (mostly working, need to test)notebook  : not yet startedWhile this is the master branch, this project is not ready for public
consumption. Stand by...All examples commands, install, etc. assume a linux (ubuntu) installation and
use theaptpackage manager, etc.PrerequisitesInstall basic depssudoaptinstallparallelrclonebuild-essentialgitp7zip-fullInstall git-lfs (git large file store)git-lfsis used for storing the database files, which are basically binary blobs
that are updated periodically. Database blob revisions are offloaded to git-lfs.For installation instructions, go here:https://git-lfs.github.com/Example:curl-shttps://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh|sudobashInstall conda/anacondaYou don't strictlyneedanaconda for this toolchain to work. If you prefer
mucking with python virtualenv directly, then go for it. I find that anaconda
works really well with other parts of this toolchain, namely Jupyter Notebooks.
All the examples and documentation will assume you are using anaconda.Example, see the website for current instructions:https://www.anaconda.com/distribution/#download-sectionExample:wgethttps://repo.anaconda.com/archive/Anaconda3-2019.07-Linux-x86_64.sh
chmod+xAnaconda3-2019.07-Linux-x86_64.sh./Anaconda3-2019.07-Linux-x86_64.shInstallationThis webpage has a great tutorial on how to use conda. It's what I use
when I forget the commands and concepts:https://geohackweek.github.io/Introductory/01-conda-tutorial/EnvironmentFirst, create a conda environment, it can be named anything, I'll call
thisedc-cli:condaupdateconda
condacreate-nedc-clipython=3numpyjupyterpandas
condaactivateedc-cliClientThen install the energy-dashboard-client:pipinstall-Uenergy-dashboard-clientSetupThe energy-dashboard-client has two commands to get you up and running with an
energy-dashboard:clone : this will literally use git to clone the energy-dashboard repo to your local machineupdate : this will pull down all the submodules to your local machineNote: if you only want a subset of the submodules installed on your local machine, then you
can use thegit submodule deinit data/[name-of-submodule-to-remove].As always, let me know if you need better tooling around this or any other aspect of this project.Clone and Updatemkdirfoocdfoo
edcclonecdenergy-dashboard
edcupdateVerify SetupAt this point you should have a working environment:Verify that you have files:$tree-L1.
├──data
├──docs
├──LICENSE
├──notebooks
├──README.md
└──run.shVerify thatedcworks:`$edc--help
Usage:edc[OPTIONS]COMMAND[ARGS]...CommandLineInterfacefortheEnergyDashboard.Thistoolingcollectsinformationfromanumberofdatafeeds,importsthatdata,transformsit,andinsertsitintoadatabase.

Options:--ed-dirTEXTEnergyDashboarddirectory(defaultstocwd)--log-level[CRITICAL|ERROR|WARNING|INFO|DEBUG]--helpShowthismessageandexit.

Commands:cloneCloneenergy-dashboardlocallyfeedManageindividual'feed'(singular).feedsManagethefullsetofdata'feeds'(plural).licenseShowthelicense(GPLv3).updateUpdatethesubmodulesVerify that you can list out the data feeds:$edcfeedslist|head
data-oasis-atl-ruc-zone-map
data-oasis-cbd-nodal-grp-cnstr-prc
data-oasis-cmmt-rmr-dam
data-oasis-atl-sp-tie
data-oasis-prc-mpm-cnstr-cmp-dam
data-oasis-trns-curr-usage-all-all
data-oasis-ene-baa-mkt-events-rtd-all
data-oasis-ene-eim-transfer-limit-all-all
data-oasis-as-results-dam
data-oasis-ene-wind-solar-summaryUsingfindwe can verify that we don't have any data files
such as .zip, .xml, or .sql in the tree, but that wedohave
the state files:$ find data/ | grep state | head
data/data-oasis-atl-ruc-zone-map/sql/state.txt
data/data-oasis-atl-ruc-zone-map/xml/state.txt
data/data-oasis-atl-ruc-zone-map/zip/state.txt
data/data-oasis-cbd-nodal-grp-cnstr-prc/sql/state.txt
data/data-oasis-cbd-nodal-grp-cnstr-prc/xml/state.txt
data/data-oasis-cbd-nodal-grp-cnstr-prc/zip/state.txt
data/data-oasis-cmmt-rmr-dam/sql/state.txt
data/data-oasis-cmmt-rmr-dam/xml/state.txt
data/data-oasis-cmmt-rmr-dam/zip/state.txt
data/data-oasis-atl-sp-tie/sql/state.txtNow verify what databases you have downloaded...$finddata/|grep"\.db$"|head
data/data-oasis-atl-gen-cap-lst/db/data-oasis-atl-gen-cap-lst_00.db
data/data-oasis-sld-adv-fcst-rtd/db/data-oasis-sld-adv-fcst-rtd_01.db
data/data-oasis-sld-adv-fcst-rtd/db/data-oasis-sld-adv-fcst-rtd_00.db
data/data-oasis-sld-sf-eval-dmd-fcst/db/data-oasis-sld-sf-eval-dmd-fcst_00.db
data/data-oasis-sld-sf-eval-dmd-fcst/db/data-oasis-sld-sf-eval-dmd-fcst_03.db
data/data-oasis-sld-sf-eval-dmd-fcst/db/data-oasis-sld-sf-eval-dmd-fcst_01.db
data/data-oasis-sld-sf-eval-dmd-fcst/db/data-oasis-sld-sf-eval-dmd-fcst_05.db
data/data-oasis-sld-sf-eval-dmd-fcst/db/data-oasis-sld-sf-eval-dmd-fcst_04.db
data/data-oasis-sld-sf-eval-dmd-fcst/db/data-oasis-sld-sf-eval-dmd-fcst_02.db
data/data-oasis-ene-flex-ramp-dc-rtd-all/db/data-oasis-ene-flex-ramp-dc-rtd-all_00.dbI'll go over this in more detail below, but the reason there are multiple database files
for a given data feed is because the feed has multiple formats (argh!) and I
have not yet sorted out how to deal with that. More on this later.Use CasesCreate Jupyter NotebookTODO : This is what most of the users of this project want to do.The code to automatically generate jupyter notebooks has not been written yet, but
it's essentially two things:look at the generated database and the tables in that database (or the .sql files)use Jinja2 templating to emit a Jupyter Notebook with the Pandas queries wired inWhile the code toautomaticallygenerate this hasn't been written yet, you can
always write your own Jupyter Notebook and manualy specify the database and
queries... just look at the samples in the energy-dashboard/notebooks directory.Curate Data FeedsAt a high level, a data feed is simply a url and some instructions for processing
it. The url is stored in themanifest.json, and the processing instructions
are stored in the./srcdirectory. The./srcdirectory contains python files
that handle downloading, parsing, constructing sql insert statements, and inserting
the data into a sqlite3 database. See the section onAdd New Data Feedfor more
details on the construction of a data feed.PipelineData feeds are processed in stages.StagesData moves horizontally across the stages, starting withdownloadand winding up
with a database as the final artifact.Here are the typical stages:DOWNLOAD->UNZIP->PARSE->INSERT->SAVEThese stages are represented by bi the edcproccommands:download | unzip | parse | insert | save. Eachproccommand processes the artifacts in the
previous stage into it's stage.DownloadThedownloadcommand looks at the current date/time, generates a list of urls
with start and end dates, and downloads all these artifacts into the./zipdirectory. The ./zip/state.txt file is updated with the urls of the downloaded
artifacts.edcfeedsome-feed-named-fooprocdownloadUnzipTheunzipstage extracts./zipfiles from the downloaded artifacts into the./xmldirectory. The ./xml/state.txt file is updated with the name of the zip
file that was unzipped.edcfeedsome-feed-named-fooprocunzipParseTheparsestage reads the xml files from./xmland writes out sql
statements to the./sqldirectory. The ./sql/state.txt file is updated with
the name of the xml file that was parsed.edcfeedsome-feed-named-fooprocparseInsertTheinsertstage reads the sql files from./sqland creates a database
and inserts the records into that database in the./dbdirectory. The ./db/state.txt
file is updated with the name of the sql file that was inserted.edcfeedsome-feed-named-fooprocinsertSaveThesavestage invokes git to save the repository.edcfeedsome-feed-named-fooprocinsertNote, that this procesing needs to be able to be re-started and continued, or
reset and started-from-scratch. That's what the state.txt files are for.State FilesDOWNLOAD->EXTRACT->PARSESQL->INSERT->*DATABASE*
./zip/./xml/./sql/./db/state.txtstate.txtstate.txtstate.txtEach state.txt contains a list of artifacts that have already been processed.
Originally I called these: ./zip/downloaded.txt, ./xml/unzipped.txt, etc. But
after working with this for a few days, it is easier to justcatout [dir]/state.txt
rather than remembering what each state file is named.Each stage in the pipeline looks at the artifacts in the previous stage and compares
that list with the list of previously processed artifacts in it'sstate.txtfile, and
then gives the delta of new files to the processing code.So, to restart a given stage, you just delete the stage directory. This deletes all
the generated artifactsandthe state file. Voila. You are ready to start over.Note: if you delete a stage, you may want to delete the subsequent stages, too.Here's the command that does this for you:edcfeed[feedname]reset[stage]Here's the scenario. I've been writing the code for this project on my laptop. But it does
not have the horsepower to crunch all this data into the various sqlite databases in
a reasonable amount of time. So I'm firing up a desktop machine to perform the heavy
lifting. Here's what that process looks like. This is the same process any researcher
that wanted to replicate my work would want to do.Clone and updatemkdirfoo
edcclonecdenergy-dashboard
edcupdateAt this point, we have the energy-dashboard project, but we don't want to re-download
all the previously downloaded files from their original source. In the case of CAISO
OASIS, that would simply take too long (I've calculated the upper bound as 152 days,
though in reality it took about 3 weeks to download the resources here). Instead, we
can pull these previously downloaded artifacts from one of the public S3 buckets that
I've mirrored them on...Example:edcfeeddata-oasis-atl-ruc-zone-maps3restoreTo grab the artifacts from the entire set of feeds:edcfeedslist|xargs-L1-I{}edcfeed{}s3restoreEven though these are replicated on S3 and the download times are reasonable, you'll want
to grab a coffee, take a walk, etc. while the raw dataset downloads.At the current time, only the original zip files will be downloaded.TODO: give the s3restore command more fidelity to control what get's restored. Currently
it is hardwired to just download the zip files, but it's possible someone would want to
select any of the stage artifacts [zip, xml, sql, db].Ok, so you had coffee, took a walk, and stuff is still downloading. No problem. Pulling stuff
off of S3 is compressing what it took me weeks to accumulate down to hour(s). Let's continue...We're downloading the feeds in this order, so the top few should have zip files:$edcfeedslist|head
data-oasis-atl-ruc-zone-map
data-oasis-cbd-nodal-grp-cnstr-prc
data-oasis-cmmt-rmr-dam
data-oasis-atl-sp-tie
data-oasis-prc-mpm-cnstr-cmp-dam
data-oasis-trns-curr-usage-all-all
data-oasis-ene-baa-mkt-events-rtd-all
data-oasis-ene-eim-transfer-limit-all-all
data-oasis-as-results-dam
data-oasis-ene-wind-solar-summarySo let's process the first feed in the list and make sure everything in the tool chain works...First, let's use theinvokecommannd to execute arbitrary commands from that directory:edcfeeddata-oasis-atl-ruc-zone-mapinvoke'ls zip'|head

oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130101T07_00-0000_edt_20130102T07_00-0000_v_1.zip
oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130102T07_00-0000_edt_20130103T07_00-0000_v_1.zip
oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130103T07_00-0000_edt_20130104T07_00-0000_v_1.zip
oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130104T07_00-0000_edt_20130105T07_00-0000_v_1.zip
oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130105T07_00-0000_edt_20130106T07_00-0000_v_1.zip
oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130106T07_00-0000_edt_20130107T07_00-0000_v_1.zip
oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130107T07_00-0000_edt_20130108T07_00-0000_v_1.zip
oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130108T07_00-0000_edt_20130109T07_00-0000_v_1.zip
oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130109T07_00-0000_edt_20130110T07_00-0000_v_1.zip
oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20130110T07_00-0000_edt_20130111T07_00-0000_v_1.zipedcfeeddata-oasis-atl-ruc-zone-mapinvoke'ls zip'|wc24512450205727Cool, so we have 2451 zip files to play with. That should match the number of lines
in the ./xml/state.txt file:edcfeeddata-oasis-atl-ruc-zone-mapinvoke'cat zip/state.txt'|wc24502449355106Close enough. Let's try to generate the database, which is done
with theproccommand:edcfeeddata-oasis-atl-ruc-zone-mapproc--help
Usage:edcfeedproc[OPTIONS]STAGEProcessthefeedthroughthestageprocesingfilesinthe'./src'directory,inlexicalorder.Stagesare:['download','unzip','parse','insert']Options:--helpShowthismessageandexit.TODO: add the 'all' stage and the 'save' stage.Let's run through them one at a time to make sure everything
works as expected...edcfeeddata-oasis-atl-ruc-zone-mapprocdownload{"ts":"09/18/2019 01:33:16 PM","msg":{"src":"data-oasis-atl-ruc-zone-map","action":"download","url":"http://oasis.caiso.com/oasisapi/SingleZip?queryname=ATL_RUC_ZONE_MAP&startdatetime=20190916T07:00-0000&enddatetime=20190917T07:00-0000&version=1","file":"oasis_SZ_q_ATL_RUC_ZONE_MAP_sdt_20190916T07_00-0000_edt_20190917T07_00-0000_v_1.zip"}}Ok, so we downloaded a single file. This make sense. I ran this yesterday so I'd only expect to see a single new file. Today is:date

WedSep1813:34:51PDT2019The start date on that download is: sdt_20190916T07_00-0000
The end date is: edt_20190917T07_00-0000'sdt' : abbreviation for 'start date'
'edt' : abbreviation for 'end date'So today is 9/18, so the latest it can grab a report for is 9/17. Looks good.What got modified on the local system?edcfeeddata-oasis-atl-ruc-zone-mapinvoke'git status'HEADdetachedatca440b3
Changesnotstagedforcommit:(use"git add <file>..."toupdatewhatwillbecommitted)(use"git checkout -- <file>..."todiscardchangesinworkingdirectory)modified:zip/state.txt

nochangesaddedtocommit(use"git add"and/or"git commit -a")Notice that the zip file, which is in the ./zip directory, is not showing up. In
fact it is explicitly excluded from the git repository. So are the other intermediate
files like .sql and .xml. These files are expected to be replicated to the S3 buckets,
but not stored in the git repository. Only the sqlite3 .db files are stored in
git using git-lfs, and even that might change in the future if it's a pain. Time will tell.So what changed in the zip/state.txt file?$edcfeeddata-oasis-atl-ruc-zone-mapinvoke'git diff'diff--gita/zip/state.txtb/zip/state.txt
index92e0f63..b394e8f100644---a/zip/state.txt
+++b/zip/state.txt
@@-2447,3+2447,4@@http://oasis.caiso.com/oasisapi/SingleZip?queryname=ATL_RUC_ZONE_MAP&startdatetihttp://oasis.caiso.com/oasisapi/SingleZip?queryname=ATL_RUC_ZONE_MAP&startdatetime=20190913T07:00-0000&enddatetime=20190914T07:00-0000&version=1http://oasis.caiso.com/oasisapi/SingleZip?queryname=ATL_RUC_ZONE_MAP&startdatetime=20190914T07:00-0000&enddatetime=20190915T07:00-0000&version=1http://oasis.caiso.com/oasisapi/SingleZip?queryname=ATL_RUC_ZONE_MAP&startdatetime=20190915T07:00-0000&enddatetime=20190916T07:00-0000&version=1+http://oasis.caiso.com/oasisapi/SingleZip?queryname=ATL_RUC_ZONE_MAP&startdatetime=20190916T07:00-0000&enddatetime=20190917T07:00-0000&version=1The zip/state.txt file contains the urls for the artifacts that have been downloaded.Let's move on to the next stage...edcfeeddata-oasis-atl-ruc-zone-mapprocunzip

$edcfeeddata-oasis-atl-ruc-zone-mapinvoke'ls xml'20190916_20190917_ATL_RUC_ZONE_MAP_N_20190918_13_33_16_v1.xml
state.txtThere is only 1 file in the ./xml directory. This is because thes3restorecommand only restored the
zip file artifacts. Then, theproc unzipcommand only processed the new file that's not in the
./xml/state.txt file. Normally, this is fine, but right now, we want to re-run the entire pipeline from
scratch (but using the zip files we restored from s3). What to do? Reset the stage(s)...$edcfeeddata-oasis-atl-ruc-zone-mapreset--help
Usage:edcfeedreset[OPTIONS]STAGE!!!USEWITHCAUTION!!!Resetastage.Thisisadestructiveaction,makebackupsfirst!.Thiswilldeletethestagedirectory,includingthestatefileandalltheprocessedresources.!!!USEWITHCAUTION!!!Stagesare:['download','unzip','parse','insert']Options:--confirm/--no-confirm--helpIn this case we want to reset the last 3 stages...edcfeeddata-oasis-atl-ruc-zone-mapresetunzip
Abouttodelete:/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/xml.Doyouwanttocontinue?[y/N]:y
/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/xml


edcfeeddata-oasis-atl-ruc-zone-mapresetparse
Abouttodelete:/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/sql.Doyouwanttocontinue?[y/N]:y
/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/sql


edcfeeddata-oasis-atl-ruc-zone-mapresetinsert
Abouttodelete:/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/db.Doyouwanttocontinue?[y/N]:y{"ts":"09/18/2019 01:44:54 PM","msg":{"name":"edl.cli.feed","method":"reset","path":"/home/toddg/proj/energy-dashboard","feed":"data-oasis-atl-ruc-zone-map","target_dir":"/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/db","ERROR":"failed to remove target_dir","exception":"[Errno 2] No such file or directory: '/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/db'"}}TODO: fix this error message when the directory does not exist.Let's examine what thestatusof this feed is now:edcfeeddata-oasis-atl-ruc-zone-mapstatus

feedname,downloaded,unzipped,parsed,inserted
data-oasis-atl-ruc-zone-map,2450,0,0,0We have 2450 records in the zip/state.txt file to play with.Let's re-unzip and see what happens to the state:edcfeeddata-oasis-atl-ruc-zone-mapprocunzip

edcfeeddata-oasis-atl-ruc-zone-mapstatus

feedname,downloaded,unzipped,parsed,inserted
data-oasis-atl-ruc-zone-map,2450,2450,0,0Ok, so now we have 2450 downloaded zip files and 2450 extracted xml files (as tracked
by the respective state files). Onwards...edcfeeddata-oasis-atl-ruc-zone-mapprocparse{"ts":"09/18/2019 04:06:53 PM","msg":{"src":"data-oasis-atl-ruc-zone-map","action":"parse_file","infile":"20140710_20140711_ATL_RUC_ZONE_MAP_N_20190811_04_27_04_v1.xml","outfile":"/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/sql/20140710_20140711_ATL_RUC_ZONE_MAP_N_20190811_04_27_04_v1.sql","total_ddl":8,"total_sql":7688}}{"ts":"09/18/2019 04:06:57 PM","msg":{"src":"data-oasis-atl-ruc-zone-map","action":"parse_file","infile":"20150714_20150715_ATL_RUC_ZONE_MAP_N_20190811_05_00_12_v1.xml","outfile":"/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/sql/20150714_20150715_ATL_RUC_ZONE_MAP_N_20190811_05_00_12_v1.sql","total_ddl":8,"total_sql":7688}}Now this is going to take awhile to process all of the xml files and generate these sql files. While that's running, let's take a look at
a generated .sql file:$cat/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/sql/20130708_20130709_ATL_RUC_ZONE_MAP_N_20190811_03_54_06_v1.sql|head

CREATETABLEIFNOTEXISTSoasismaster(xmlnsTEXT,PRIMARYKEY(xmlns));CREATETABLEIFNOTEXISTSmessageheader(timedateTEXT,sourceTEXT,versionTEXT,oasismaster_xmlnsTEXT,FOREIGNKEY(oasismaster_xmlns)REFERENCESoasismaster(xmlns),PRIMARYKEY(timedate,source,version));CREATETABLEIFNOTEXISTSmessagepayload(idTEXT,oasismaster_xmlnsTEXT,FOREIGNKEY(oasismaster_xmlns)REFERENCESoasismaster(xmlns),PRIMARYKEY(id));CREATETABLEIFNOTEXISTSrto(nameTEXT,messagepayload_idTEXT,FOREIGNKEY(messagepayload_id)REFERENCESmessagepayload(id),PRIMARYKEY(name));CREATETABLEIFNOTEXISTSatls_item(idTEXT,rto_nameTEXT,FOREIGNKEY(rto_name)REFERENCESrto(name),PRIMARYKEY(id));CREATETABLEIFNOTEXISTSatls_header(tzTEXT,systemTEXT,reportTEXT,atls_item_idTEXT,FOREIGNKEY(atls_item_id)REFERENCESatls_item(id),PRIMARYKEY(tz,system,report));CREATETABLEIFNOTEXISTSatls_data(pnode_nameTEXT,start_date_gmtTEXT,end_date_gmtTEXT,ruc_zone_nameTEXT,end_dateTEXT,start_dateTEXT,atls_item_idTEXT,FOREIGNKEY(atls_item_id)REFERENCESatls_item(id),PRIMARYKEY(pnode_name,start_date_gmt,end_date_gmt,ruc_zone_name,end_date,start_date));CREATETABLEIFNOTEXISTSdisclaimer_item(disclaimerTEXT,rto_nameTEXT,FOREIGNKEY(rto_name)REFERENCESrto(name),PRIMARYKEY(disclaimer));INSERTORIGNOREINTOoasismaster(xmlns)VALUES("http://www.caiso.com/soa/OASISMaster_v1.xsd");INSERTORIGNOREINTOmessageheader(source,timedate,version,oasismaster_xmlns)VALUES("OASIS","2019-08-11T10:54:06-00:00","v20131201","http://www.caiso.com/soa/OASISMaster_v1.xsd");So a couple of things to notice here. First, the DDL is included at the top of every file. This allows the creation of the sql tables at any point
in the pipeline. So if, for example, the data feed structurally changes from one file to the next, then the DDL will change. Later in the pipeline,
when we process this .sql file, if it fails b/c the schema has changed, then a new table will be created.For now, let's verify that the DDL and associated SQL insert statements are correct. This command will process this file into an in-memory
sqlite3 db instance. If it fails, then we know we have an error in either the DDL or the SQL.$cat/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/sql/20130708_20130709_ATL_RUC_ZONE_MAP_N_20190811_03_54_06_v1.sql|sqlite3Now, it might take some time to process all the .xml files into the .sql files. So CTRL-C out of the running process and let's move on to
the next stage, 'insert'.$edcfeeddata-oasis-atl-ruc-zone-mapstatus

feedname,downloaded,unzipped,parsed,inserted
data-oasis-atl-ruc-zone-map,2450,2450,98,0Ok, so only 98 out of the 2450 xml files have been turned into sql files. No problem. Let's insert the 98 sql files into the database and see what we have.$edcfeeddata-oasis-atl-ruc-zone-mapprocinsert{"ts":"09/18/2019 04:13:44 PM","msg":{"name":"edl.resources.db","src":"data-oasis-atl-ruc-zone-map","method":"insert","sql_dir":"/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/sql","db_dir":"/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/db","new_files":99}}{"ts":"09/18/2019 04:13:44 PM","msg":{"name":"edl.resources.db","src":"data-oasis-atl-ruc-zone-map","method":"insert","sql_dir":"/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/sql","db_dir":"/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/db","db_name":"data-oasis-atl-ruc-zone-map_00.db","file_idx":0,"sql_file":"/home/toddg/proj/energy-dashboard/data/data-oasis-atl-ruc-zone-map/sql/20190204_20190205_ATL_RUC_ZONE_MAP_N_20190811_06_56_57_v1.sql","depth":0,"message":"started"}}Again, CTRL-C after a few of these have processed and let's take a look at the database...$edcfeeddata-oasis-atl-ruc-zone-mapdb--help
Usage:edcfeeddb[OPTIONS]COMMAND[ARGS]...Manageafeed'sdatabase(s).

Options:--helpShowthismessageandexit.

Commands:consolelaunchsqlite3databaseconsolecreateddlGenerateSQLDDLfortablecreation.insertsqlGenerateSQLfordatainsertionintotable.listListthefeeddatabasesOk, so what databoses do we have?$edcfeeddata-oasis-atl-ruc-zone-mapdblist
data-oasis-atl-ruc-zone-map_00.dbAh, good. Just one. Let's take a peek...$edcfeeddata-oasis-atl-ruc-zone-mapdbconsoledata-oasis-atl-ruc-zone-map_00.db

SQLiteversion3.29.02019-07-1017:32:03
Enter".help"forusagehints.
sqlite>Bingo, we are in! What's in it?sqlite>.tables
atls_dataatls_itemmessageheaderoasismasteratls_headerdisclaimer_itemmessagepayloadrtoOk, so I'm going to guess that we want to look at 'atls_data':sqlite>pragmatable_info(atls_data);0|pnode_name|TEXT|0||11|start_date_gmt|TEXT|0||22|end_date_gmt|TEXT|0||33|ruc_zone_name|TEXT|0||44|end_date|TEXT|0||55|start_date|TEXT|0||66|atls_item_id|TEXT|0||0And what's the data look like?sqlite>select*fromatls_datalimit10;PRKWAY_2_LD1|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_PGAE|2030-12-31T23:59:59|2009-03-31T00:00:00|74466c6d-55fc-4978-9c0d-7ea67a87f9eb
MRYSVL_6_LD2|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_PGAE|2030-12-31T23:59:59|2009-03-31T00:00:00|e481dd77-d2e4-42f3-b985-06195618f5ee
SNTABL_6_LD30|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_SDGE|2030-12-31T23:59:59|2009-03-31T00:00:00|b654a7a4-5ac9-44b2-9f3b-7bd20005da37
PALMER_1_LD1|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_PGAE|2030-12-31T23:59:59|2009-03-31T00:00:00|b5a6e60d-61c7-4baa-a98e-e2bdfa176c80
POWAY_6_LD31|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_SDGE|2030-12-31T23:59:59|2009-03-31T00:00:00|17d495fc-4317-44b0-82a3-ce669aefc869
PRCTVY_1_LD42|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_SDGE|2030-12-31T23:59:59|2009-03-31T00:00:00|632fde0d-ccf8-4643-b759-87340ddd9a50
ORTGA_6_LD1|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_PGAE|2030-12-31T23:59:59|2009-03-31T00:00:00|5accad7b-06b9-43d4-a6ce-77edc5334e87
SOBAY_6_GN-A1XA2|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_SDGE|2030-12-31T23:59:59|2009-03-31T00:00:00|ab398edd-e8b0-4d81-aaab-683043e97dd0
CAWELO_6_LD-B|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_PGAE|2030-12-31T23:59:59|2009-03-31T00:00:00|4de47a31-34ec-4785-8c09-d30d3a0247a4
LNTREE_2_LD1|2009-03-31T07:00:00-00:00|2031-01-01T07:59:59-00:00|RUC_PGAE|2030-12-31T23:59:59|2009-03-31T00:00:00|0eb6b341-a5ad-4bcf-b2b4-727b0a1e92d4Note that the guid looking things are, in fact, guids. Because there's no way to generate the .sql filesandknow the rowid of a previously inserted
item in a parent table, I'm using the 'time-worn'(tm) strategy of generating uuids out-of-band of the database for table relationships.At this point, we have a functioning system. Now's a good time to dial up the processing and try and use all the horsepower of my desktop machine.
You see, because each stage is restartable, and each data feed is separate, I can use an sort of parallel processing strategy to orchestrate.edcfeedslist|parallel"edc feed {} reset unzip --no-confirm"edcfeedslist|parallel"edc feed {} reset parse --no-confirm"edcfeedslist|parallel"edc feed {} reset insert --no-confirm"Ok, so now we are in a clean state. Files are still downloading on the process we launched before. But while that's running, we can start
processing the currently downloaded files. It's basically an eventually consistent sort of thing...$edcfeedslist|parallel"edc feed {} status --no-header"data-oasis-atl-ruc-zone-map,2450,0,0,0
data-oasis-ene-wind-solar-summary,2449,0,0,0
data-oasis-cbd-nodal-grp-cnstr-prc,2449,0,0,0
data-oasis-cmmt-rmr-dam,2449,0,0,0
data-oasis-atl-sp-tie,2449,0,0,0
data-oasis-ene-eim-transfer-limit-all-all,2449,0,0,0
data-oasis-prc-mpm-cnstr-cmp-dam,2449,0,0,0
data-oasis-ene-baa-mkt-events-rtd-all,2449,0,0,0
data-oasis-prc-cd-rtm-nomogram-rctm-all,2449,0,0,0So there is plenty of downloaded zip files from the s3restore command to start processing.
Let's go...First, we unzip...edcfeedslist|parallel--max-procs70%"edc feed {} proc unzip"Then, we parse...edcfeedslist|parallel--max-procs80%"edc feed {} proc parse"Then, we insert...edcfeedslist|parallel--max-procs80%"edc feed {} proc insert"Then, we check the databases...TODOMaintainersMaintainers will need to install 'rclone' in addition to the stuff mentioned above.sudoaptinstallrclone'rclone' is necessary for project maintainers that need to upload resources to
s3 buckets. For users that just want to use the databases, git-lfs is all you
need.  For those users that want the original and intermediate artifacts such
as .zip, .xml, and .sql files, the python 'requests module' is all you need,
and that's included with the installation of the energy-dashboard-client.Add New Data FeedTODOShow HelpedcUsage:edc[OPTIONS]COMMAND[ARGS]...CommandLineInterfacefortheEnergyDashboard.Thistoolingcollectsinformationfromanumberofdatafeeds,importsthatdata,transformsit,andinsertsitintoadatabase.

Options:--config-dirTEXTConfigfiledirectory--debug/--no-debugEnabledebuglogging--helpShowthismessageandexit.

Commands:configManageconfigfile.feedManageindividual'feed'(singular).feedsManagethefullsetofdata'feeds'(plural).licenseShowthelicense(GPLv3).configUsage:edcconfig[OPTIONS]COMMAND[ARGS]...Manageconfigfile.

Options:--helpShowthismessageandexit.

Commands:showShowtheconfigupdateUpdateconfigfeedUsage:edcfeed[OPTIONS]COMMAND[ARGS]...Manageindividual'feed'(singular).

Options:--helpShowthismessageandexit.

Commands:archiveArchivefeedtotar.gzcreateCreatenewfeeddownloadDownloadfromsourceurlinvokeInvokeashellcommandinthefeeddirectoryprocProcessafeedthroughthestagesresetResetfeedtoreprocessstagerestoreRestorefeedfromtar.gzs3archiveArchivefeedtoS3buckets3restoreRestorefeedzipfilesfromfromS3bucketstatusShowfeedstatusfeedsUsage:edcfeeds[OPTIONS]COMMAND[ARGS]...Managethefullsetofdata'feeds'(plural).

Options:--helpShowthismessageandexit.

Commands:listListfeedssearchSearchfeeds(NYI)licenseedc:EnergyDashboardCommandLineInterfaceCopyright(C)2019ToddGreenwood-Geer(EnviroSoftwareSolutions,LLC)Thisprogramisfreesoftware:youcanredistributeitand/ormodifyitunderthetermsoftheGNUGeneralPublicLicenseaspublishedbytheFreeSoftwareFoundation,eitherversion3oftheLicense,or(atyouroption)anylaterversion.Thisprogramisdistributedinthehopethatitwillbeuseful,butWITHOUTANYWARRANTY;withouteventheimpliedwarrantyofMERCHANTABILITYorFITNESSFORAPARTICULARPURPOSE.SeetheGNUGeneralPublicLicenseformoredetails.YoushouldhavereceivedacopyoftheGNUGeneralPublicLicensealongwiththisprogram.Ifnot,see<https://www.gnu.org/licenses/>.##UsageExamplesedcfeedinvokedata-oasis-atl-lap-all"git st"edcfeedinvokedata-oasis-atl-lap-all"ls"edcfeedinvokedata-oasis-atl-lap-all"cat manifest.json"edcfeedinvokedata-oasis-atl-lap-all"head manifest.json"edcfeedslist
edcfeedslist|grepatl
edcfeedslist|grepatl|edcfeedinvoke"head manifest.json"edcfeedslist|grepatl|edcfeedinvoke"head manifest.json"-
edcfeedslist|grepatl|xargs-L1-I{}edcfeedinvoke{}"head manifest.json"edcfeedslist|grepatl|xargs-L1-I{}edcfeedinvoke{}"jq . < manifest.json"edcfeedslist|grepatl|xargs-L1-I{}edcfeedinvoke{}"jq .url < manifest.json"edcfeedslist|grepmileage|xargs-L1-I{}edcfeedinvoke{}"echo {}; sqlite3 db/{}.db 'select count(*) from oasis'"edcfeedslist|grepatl|xargs-L1-I{}edcfeedinvoke{}"jq .url < manifest.json"edcfeedslist|xargs-L1-I{}edcfeedinvoke{}"echo {}; sqlite3 db/{}.db 'select count(*) from oasis'"edcfeedslist|grepatl|xargs-L1-I{}edcfeedstatus{}edcfeedslist|grepatl|xargs-L1-I{}edcfeedstatus--header{}edcfeedslist|grepatl|xargs-L1-I{}edcfeedstatus--header{}edcfeedslist|grepmileage|xargs-L1-I{}edcfeedstatus--header{}edcfeedslist|xargs-L1-I{}edcfeedinvoke{}"./src/10_down.py"edcfeedarchivedata-oasis-as-mileage-calc-all
edcfeedarchivedata-oasis-as-mileage-calc-all|xargs-L1-I{}tar-tvf{}edcfeedresetdata-oasis-as-mileage-calc-all--stagexml--stagedb
edcfeeds3restoredata-oasis-as-mileage-calc-all--outdir=temp--service=wasabi
edcfeeds3archivedata-oasis-as-mileage-calc-allOnboardingSome quick notes on how I onboarded 'data-oasis-as-mileage-calc-all':edcfeedprocdata-oasis-as-mileage-calc-all
edcfeeds3archivedata-oasis-as-mileage-calc-all--servicewasabi
edcfeeds3archivedata-oasis-as-mileage-calc-all--servicedigitalocean
edcfeedstatusdata-oasis-as-mileage-calc-all--header
edcfeedinvokedata-oasis-as-mileage-calc-all"git st"edcfeedinvokedata-oasis-as-mileage-calc-all"git log"edcfeedinvokedata-oasis-as-mileage-calc-all"git show HEAD"AuthorTodd Greenwood-Geer (Enviro Software Solutions, LLC)NotesThis project uses submodules, and this page has been useful:https://github.blog/2016-02-01-working-with-submodules/