Package: sheetsite
# sheetsite: sheets for sites[![Build Status](https://travis-ci.org/paulfitz/sheetsite.svg?branch=master)](https://travis-ci.org/paulfitz/sheetsite)[![PyPI version](https://badge.fury.io/py/sheetsite.svg)](http://badge.fury.io/py/sheetsite)Keep a website or directory in sync with a google sheet.Features:* Copy a google spreadsheet locally, as json or excel format.* Can strip specified tabs, columns, or cells from the spreadsheet,in case not all of it should be copied along.* Can push a filtered json copy out to a git repository, handy formaintaining a website based on a private shared spreadsheet.* Can augment the sheet with geocoding, adding latitude and longitude basedon address fields for example.* Can notify people by email with a summary of updates.## InstallationFor the basics:```pip install sheetsite```For all bells and whistles, when automating a sheet-to-site workflow:```pip install sheetsite[queue]```## Specifying the source and destinationThe `sheetsite` utility, when run without any arguments, will expectto find all necessary options in a `_sheetsite.yml` file.  A simpleexample of such a file is:```yamlsource:name: google-sheetskey: 15Vs_VGpupeGkljceEow7q1ig447FJIxqNS1Dd0dZpFccredential_file: service.jsondestination:file: sheet.xlsx```The file should have two stanzas, `source` specifying where to getdata from, and `destination` specifying where to put it.  Thisexamples reads a private google spreadsheet and saves it as`sheet.xlsx`.  The key comes from the url of the spreadsheet.The credentials file is something you [get from google](https://pygsheets.readthedocs.io/en/stable/authorizing.html).Here's an example that outputs json:```yamlsource:name: google-sheetskey: 15Vs_VGpupeGkljceEow7q1ig447FJIxqNS1Dd0dZpFccredential_file: service.jsondestination:file: _data/directory.json```You could now build a static website from that `.json`, seehttp://jekyllrb.com/docs/datafiles/ for how, or see an exampleat https://github.com/datacommons/commonersHere's an example that adds some geocoded fields and directlyupdates a git repository:```yamlsource:name: google-sheetskey: 19UaXhqPQ0QHEfSWS_adDEtPwYstq8llK2YijpvFZcKAcredential_file: service.jsonflags:add:directory:- LAT- LNG- COUNTRY- STREET- REGION- LOCALITYdestination:name: gitrepo: git@github.com:datacommons/commonersfile: _data/directory.json```## Strip private sheets, columns, or cellsBy default, sheetsite will strip:* Any columns whose name is in parentheses, e.g. `(Private Notes)`* Any cells or text within cells surrounded by double parentheses, e.g. `((private@email.address))`* Any sheets whose name is in double parentheses, e.g. `((secret sheet))`## GeocodingIf you have a table with a column called `address`, sheetsite can geocode it foryou and pass along the results.  Just add the following in your yaml:```flags:add:table_name_goes_here:- latitude- longitude- country- state- city- street- zip```You can add just the columns you want.  Geocoding results are cached in a `_cache`directory by default so they do not need to be repeated in future calls to sheetsite.The full list of columns (with synonyms) available is:* latitude / lat* longitude / lng* latlng* country* state / province / region* city / locality* street* zip / postal_codeNormally you won't actually have a stand-alone `address` column.  More usually,information will be spread over multiple columns, or some will be implicit (e.g.the state/province and country).  You can tell sheetsite how to construct addressesfor geocoding by listing columns and constants to build it from.  For example:```flags:address:table_name_goes_here:- street_address1- street_address2- city- Manitoba- Canadaadd:table_name_goes_here:- postal_code```This tells sheetsite to produce addresses of the form:```<street_address1> <street_address2> <city> Manitoba Canada```And add a `postal_code` column populated by geocoding.It is possible to request columns directly in the spreadsheet.  Justwrap the column name in square brackets, like `[state]` or `[zip]`.Any blank cells in such columns will be filled using geocoding basedon the address given in that row.  If the address columns have not beenconfigured in `flags` then the address must be present in a single columnliterally called `address`.## Row uuidsThere's a random feature to add uuids to rows.   Just add a columncalled `dccid` for some reason:```flags:add:table_name_goes_here:- dccid```A uuid will be added to each row.  A good faith effort will be madeto keep that uuid constant across updates, keeping it linked to therow where it first appeared.## Grouping locationsIf there are several rows of a sheet that will give locations that shouldbe thought of as a single unit (e.g. an organization with multiple locations),you can tell `sheetsite` about that.  To do so, give it a `group` key.Every row for which the `group` is the same (and not blank) will be boundtogether.  When geocaching, blank cells in address cells  will be filledin with information from the first row in this group.  For example, with thisconfiguration:```flags:group: WEBSITE```Then for a table like the following:```STREET,   CITY,   STATE,    WEBSITE...17 N St,  Foo,    Utopia,   joe.ut16 S St,  ,       ,         joe.ut...```During geocoding, `16 S St` would be assumed to be in `Foo, Utopia`.## Renaming columnsColumns can be renamed.  This will occur before any other operation.```flags:rename:table_name:old_column_name1: new_column_name1old_column_name2: new_column_name2```## Getting credentials[Obtain credentials for accessing sheets from the Google Developers Console](https://pygsheets.readthedocs.io/en/latest/authorizing.html).Make sure you share the sheet with the email address in the credentials file.  Read-only permission is fine.## ExamplesFor example, the map at http://datacommons.coop/tap/ is a visualizationof data pulled from a google spreadsheet, styled usinghttps://github.com/datacommons/tap via github pages.## sheetwatchIt can be useful to automate and forget `sheetsite`, so that updatesto a google spreadsheet propagate automatically to their finaldestination.  The `sheetwatch` utility does this.  It requires a queueserver to operate.  To install, do:```pip install sheetsite[queue]```Install any queue server supported by `celery`.  For example, `redis`:```sudo apt-get install redis-serverredis-server```We need to set some environment variables to let `sheetwatch` knowwhere to find the queue server:```export SHEETSITE_BROKER_URL=redis://localhostexport SHEETSITE_RESULT_BACKEND=redis://localhost```The `sheetwatch` program needs a cache directory for its operations.```export SHEETSITE_CACHE=$HOME/cache/sites```Finally, it needs to know where there is a directory full of `yml`files describing any sheets to monitor and their corresponding sites:```export SHEETSITE_LAYOUT=$PWD/sites/enabled```We now start a worker:```sheetwatch worker```The last thing we need to do is check a mailbox from time to timefor sheet change notifications from Google, and kick off site updatesas needed:```export GMAIL_USERNAME=*****export GMAIL_PASSWORD=*****sheetwatch ping --delay 60```## Licensesheetsite is distributed under the MIT License.