dmwm/deployment — pull request #114 by vidma (2014-02-23 11:43:13+00:00)
Title: Das mongodb separation  (it is rebased on top of valentin's request)

**This would provide a complete separation between DAS and MongoDB and simplified dasmap/das-metadata update procedure.**

**Major novelties:**
- All DB updates are now initiated from DAS which connect to a database (instead from MongoDB manage scripts) .
- a script to automatically download and stage the metadata (including dasmaps) updates from a repository (das/manage fetchdata). download dasmaps "schema" is validated before staging them.
- a script to reset the DB to initial state shipped with RPM (das.manage resetdata)

In more details, I propose to proceed something like this:
* **das/deploy:**
- on deploy call manage/resetdata to stage default data
- schedule cronjobs:
* das/manage updatedata - to periodically load stagedir, similarly as it has been up to now
* ~~das/manage fetchdata (and there should be a switch whether this automatic updating is enabled)~~
- the dasmap & keylearning updates must be initiated semi-manually, to have some supervision.
* **das/manage new commands**:
- fetchdata: download data from repo and stage it for updating - to be run manually
- updatedata: check staging area for updates, is run periodically via cron (call existing das_db_import)
- resetdata: stage the default data contained in DAS's RPM
* **mongo/deploy and mongodb/manage - does nothing specific for DAS**
- optionally, on reboot possible power failure, could consider resetting dasdb to the default state which either was included in DAS rpms, or in the last successful update (not implemented), but if we had real separation this would be harder to signal / identify. now DB is updated from DAS side (server containing DAS), not from mongodb.

----

This is how it works currently:

```bash
bash-3.2$ sudo -H -u _das bashs -lc "/data/current/config/das/manage fetchdata 'I did read documentation'"
Downloading into tmp dir: /data/state/das/metadata_tmp
Get https://raw.github.com/dmwm/DASMaps/master/js/das_maps_dbs_prod.js
Get https://raw.github.com/dmwm/DASMaps/master/js/kws/update_keylearning_db.js
Get https://raw.github.com/dmwm/DASMaps/master/js/kws/update_inputvals_datatype_name.js
Get https://raw.github.com/dmwm/DASMaps/master/js/kws/update_inputvals_group_name.js
Get https://raw.github.com/dmwm/DASMaps/master/js/kws/update_inputvals_primary_dataset_name.js
Get https://raw.github.com/dmwm/DASMaps/master/js/kws/update_inputvals_release_name.js
Get https://raw.github.com/dmwm/DASMaps/master/js/kws/update_inputvals_site_name.js
Get https://raw.github.com/dmwm/DASMaps/master/js/kws/update_inputvals_status_name.js
Get https://raw.github.com/dmwm/DASMaps/master/js/kws/update_inputvals_tier_name.js
Validating and Staging the downloaded files to be imported into MongoDB
Validating DASMaps: /data/state/das/metadata_tmp/das_maps_dbs_prod.js
Validating metadata (keylearning, inputvals)
validating: /data/state/das/metadata_tmp/update_inputvals_datatype_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_group_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_primary_dataset_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_release_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_site_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_status_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_tier_name.js
Validating metadata (keylearning, inputvals)
validating: /data/state/das/metadata_tmp/update_inputvals_datatype_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_group_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_primary_dataset_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_release_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_site_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_status_name.js
validating: /data/state/das/metadata_tmp/update_inputvals_tier_name.js
copying initial metadata from: /data/state/das/metadata_tmp
```
if all went fine it will be updated (this command is also run as a cron job):

```bash
bash-3.2$ sudo -H -u _das bashs -lc "/data/current/config/das/manage updatedata 'I did read documentation'"
update_db(mapping)
Clean mapping
MongoDB shell version: 2.4.8
connecting to: 127.0.0.1:8230/test
Updating db: mapping col: db
connected to: 127.0.0.1:8230
Thu Feb 20 16:57:07.622 check 9 191
Thu Feb 20 16:57:07.627 imported 191 objects
update_db(keylearning)
- no changes needed.
update_db(inputvals)
- no changes needed.
```

you can always stage data to be reset to the default one:
```bash
bash-3.2$ sudo -H -u _das bashs -lc "/data/current/config/das/manage resetdata 'I did read documentation'"
Validating DASMaps: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/services/cms_maps/das_maps_dbs_prod.js
Validating metadata (keylearning, inputvals)
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_datatype_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_group_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_primary_dataset_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_release_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_site_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_status_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_tier_name.js
Validating metadata (keylearning, inputvals)
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_datatype_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_group_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_primary_dataset_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_release_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_site_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_status_name.js
validating: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps/update_inputvals_tier_name.js
copying initial metadata from: /data/hg1312a/sw.pre.zemleris/slc5_amd64_gcc461/cms/das/2.5.0.pre2-1/lib/python2.6/site-packages/DAS/kws_data/db_dumps
```

--
P.S. this is a more clean copy of canceled: https://github.com/dmwm/deployment/pull/108