Package: export-manager
export_managerThis tool helps manage automatic exports/backups of personal data (for example, from cloud services like Todoist or Goodreads).It handles:Running exports according to a schedule (note: export_manager itself must be run via a scheduler like cron/launchd/etc)Keeping data and logs organizedCleaning up old dataGathering metrics and generating reports to help you see whether your backups are workingTerminology:dataset: a collection of parcels from a particular source, and related config and metadataparcel: data exported on a single occasion (e.g. your Todoist data retrieved on 2020-0215T01:02:03Z) and related metadataGetting StartedInstall python3 and pippip install export_managerUse the tool to set up directories for each dataset:mkdir~/exports
export_managerinit~/exports/todoist
export_managerinit~/exports/goodreads# etc. The names can be anything you want, those are just examples.This will create a directory structure like this:~/exports/
   todoist/
       config.toml
       metrics.csv
       data/
       incomplete/
       log/
   goodreads/
       config.toml
       metrics.csv
       data/
       incomplete/
       log/Edit theconfig.tomlfile in each subdirectory to specify the schedule and the command to run.
For example, if you installexporteer_todoistyou could use the following config to export data from Todoist:# Command to invoke. $PARCEL_PATH will be set to ~/exports/todoist/data/DATETIME# $DATASET_PATH is also available and will be set to ~/exports/todoistcmd="TODOIST_API_TOKEN=your_token exporteer_todoist full_sync > $PARCEL_PATH.json"# Only get a new export if the last one is at least 1 day old.interval="1 day"# Only keep the most recent 5 successful exports.# If you don't specify this, export_manager will not delete old exports.keep=5Runexport_manager process ~/exports/*to run all the exports and cleanups that are due.
Set up a cron or launchd job to run this periodically.Metrics and ReportsYou shouldn't trust your backups unless you're testing them.
Testing has to involve some manual action - if it were fully automated, you'd never know if the automation broke.
But ideally, you'd automate the process of collecting all the evidence to prove that a backup is working, so that all you have to do is periodically look at it and say "yep, looks good."export_manager helps with this by providing a report you can generate:export_managerreport~/exports/*This will:Warn you of any obvious problems, such as datasets that are overdue or failing.Tell you when each dataset was most recently exported successfully.Show you metrics about each dataset's most recent successful parcel, and compare them with metrics from 7 days and 180 days ago.By default, the metrics include the number of bytes and number of files in the parcel when it was produced.You can also define custom metrics for each dataset in theconfig.tomlfile.
For example, for json data, you might usejqto count some elements of the json.
The following config creates a metric named "tasks" to track the number of tasks in the todoist exports configured above:metrics.tasks.cmd="jq '.items | length' $PARCEL_PATH"Theprocesscommand gathers all the configured metrics every time a new parcel is produced and stores them inmetrics.csv.Tracking Parcels in GitIf you make your dataset directory a git repo, and setgit = trueinconfig.toml, then the data files of successful exports, as well as the metrics.csv file, will be committed after each change.Ingesting Files DirectlySome services are prohibitively difficult to export from automatically, but do provide a way to manually export your data.
export_manager can keep track of those manual exports for you.
To minimize the amount of manual work, you can configure it to automatically pull in any files that match a certain path pattern, so that all you need to do is dump the manually exported file into that location.
For example, if you put the following inconfig.toml:ingest.paths=["/Users/me/Dropbox/*.opml"]Then every time theprocesscommand is run, export_manager will check for files with that suffix in/Users/me/Dropbox.
If it finds any, it willmovethem into the dataset directory, assign them a parcel ID, run metrics on them, etc.(By default, the modification time of the file is used as the parcel ID.
You can putingest.time_source = "now"in yourconfig.tomlif you'd rather use the current date/time as the parcel ID.)Additional DocumentationHelp for each command is available on the command line and in thedoc folder.You can use export_manager programmatically.
The most important module isexport_manager.dataset.DevelopmentSetup:Install python3 and pipClone the repoI recommend creating a venv:cdexport_manager
python3-mvenvvenvsourcevenv/bin/activateInstall dependencies:pipinstall.
pipinstall-rrequirements-dev.txtTo run unit tests:PYTHONPATH=srcpytest(Overriding PYTHONPATH as shown ensures the tests run against the code in the src/ directory rather than the installed copy of the package.)To run the CLI:PYTHONPATH=srcpython-mexport_manager...ContributingBug reports and pull requests are welcome on GitHub athttps://github.com/brokensandals/export_manager.LicenseThis is available as open source under the terms of theMIT License.