Package: ontodev-cogs
COGS Operates Google SheetsWARNINGThis project is work in progress.COGS takes a set of TSV files on your local file system and allows you to edit them using Google Sheets.SetupCOGS is distributed onPyPI. To install:pip install ontodev-cogsTo see a list of all commands:cogs -hFor help with a specific command:cogs [command] -hDevelopmentFor local development on Unix (Linux or macOS), we suggest these install instructions:$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install -r requirements.txt
$ pip install -e .
$ cogs -hOverviewSince COGS is designed to synchronize local and remote sets of tables,
we try to follow the familiargitinterface and workflow:cogs initcreates a.cogs/directory to store configuration data and creates a spreadsheet for the projectcogs add foo.tsvstarts tracking thefoo.tsvtable as a sheetcogs rm foo.tsvstops tracking thefoo.tsvtable as a sheetcogs pushpushes changes to local sheets to the project spreadsheetcogs fetchfetches the data from the spreadsheet and stores it in.cogs/cogs lsshows a list of currently-tracked sheet names and their local namescogs statussummarizes the differences between tracked files and their copies in.cogs/cogs diffshows detailed differences between local files and the spreadsheetcogs mergeoverwrites local files with the data from the spreadsheet, if they have changedcogs pullcombines fetch and mergeThere are some other commands that do not correspond to anygitactions:cogs applyapplies attributes from standardized tables to one or more sheetscogs clearremoves formatting, notes, and/or data validation rules from one or more sheetscogs connectinitiates a new COGS project by connecting an existing Google Spreadsheetcogs deletedestroys the spreadsheet and configuration data, but leaves local files alonecogs ignorebegins ignoring a tracked sheet so that the local copy is no longer updatedcogs mv foo.tsv bar.tsvupdates the path to the local version of a spreadsheet fromfoo.tsvtobar.tsvcogs opendisplays the URL of the spreadsheetcogs shareshares the spreadsheet with specified usersThere is no step corresponding togit commit.We recommend runningcogs pushafter updating a local tracked sheet to keep the remote sheets in sync.When updating a remote sheet, we recommend the following to keep the local sheets in sync:cogs pullLoggingTo print info-level logging messages (error and critical level messages are always printed), run any command with the-v/--verboseflag:cogs [command & opts] -vOtherwise, most commands succeed silently.DefinitionsSpreadsheet: the remote Google Sheets spreadsheet - each COGS project corresponds to one spreadsheetSheet: a tab in the spreadsheet - each sheet corresponds to one local TSV or CSV tableRemote: data from Google SheetsLocal: data from your local working directoryCached: data stored in.cogs/which is fetched from the remote spreadsheetFormat: the format applied to a cell in a sheet (e.g., background color, font family, etc.)Note: a note (a.k.a. comment) on a cell that appears when hovered over in the sheetCommandsaddRunningaddwill begin tracking a local TSV or CSV table. The table details (path, name/title, and description) get added to.cogs/sheet.tsv.cogs add [path] -d "[description]"The-d/--descriptionis optional. You can also specify a number of rows and or columns to freeze:cogs add [path] -r [freeze_row] -c [freeze_column]If you specify-r 2 -c 1, then the first two rows and the first column will be frozen once the sheet is pushed to the remote Google Spreadsheet. If these options are not included, no rows or columns will be frozen.By default, the sheet title is created from the path (e.g.tables/foo.tsvwill be namedfoo). If a sheet with this title already exists in the project, the task will fail.You can also specify a sheet title that is different from the path with the-t/--titleoption:cogs add [path] -t "[title]"This does not immediately change the Google Sheet -- usecogs pushto push all tracked local tables to the project spreadsheet.If an added path does not exist, COGS will create an empty file at that location.Adding an Ignored SheetBy default, COGS ignores any untracked remote sheets. If you wish to start tracking a new remote sheet, you can do this by passing the sheet title toadd:cogs add [title]The default path for pulling changes from this newly-added sheet will be the current working directory (the same directory as.cogs/is in) and will be a lowercase, space-replaced version of the title (e.g.My Sheetbecomesmy_sheet.tsv). If you already have a tracked sheet at this location, the date & time will be appended to the path (e.g.,my_sheet_20200922_103045.tsvfor a sheet fetched at 10:30:45 on 2020/09/22). This path can be updated withcogs mv. Alternatively, if you want to specify a path immediately, you can:cogs add [path] -t [title]... where the "title" is the sheet title of the existing ignored sheet.You can also addallignored sheets using the-a/--allflag:cogs add --allWe recommend runningcogs fetchorcogs pullimmediately after adding ignored sheets to sync your.cogsdirectory.applyRunningapplyapplies the details of one or moremessage tablesordata validation tablesto the spreadsheet as cell formatting and notes.cogs apply [table1.tsv table2.tsv ...]Message TablesMessage tables provide a standard table output for logging messages (info, warn, or error) that can be converted into formatting and notes in the spreadsheet usingapply. As long as the table follows the format described below, any type of message can be applied to the sheets. One example is the errors fromROBOT template.These tables are applied to the sheets as formats and notes. The three levels of logging will be formatted with a black border and the following backgrounds:error: light red backgroundwarn/warning: light yellow backgroundinfo: light blue backgroundThe notes and formats will be added to any existing, but will take priority over the existing notes and formats.These tables must have the following headers:table: name of the table that the problem occurs incell: A1 format of problematic cell locationThe following fields are optional:level: severity of the problem; error, warn, or info - if not included, this will default to "error"rule ID: an IRI or CURIE to uniquely identify the problemrule: descriptive name of the problem - this is converted to the cell notemessage: detailed instructions on how to fix the problemsuggestion: a suggested value to replace the problematic valueThe only required fields aretableandcell. If no other fields are provided, the cell will be highlighted red with a note that says "ERROR". We strongly recommend using either arule IDorruleto identify the message, andmessagetext to provide more details.Data Valildation TablesThe data validation tables are applied to the sheets as data validation rules. These tables must have the following headers:table: name of the table to add data validation rules torange: A1 format of the cell or range of cells to apply data validation rules tocondition: the condition (see below) for the data validation rulevalue: the allowed value or values (see bellow)for the data validation ruleFor example:tablerangeconditionvalueSheet1A2:ATEXT_EQfooSheet1B2:BONE_OF_LISTfoo, bar, bazFor full descriptions of each condition type, please seeGoogle Sheets API ConditionType.Any value greater than one should be specified as a comma-separated list (e.g., forNUMBER_BETWEEN, the value could be1, 10). For "0" values, leave thevaluecolumn empty.Number ConditionsConditionValuesNUMBER_GREATER1NUMBER_GREATER_THAN_EQ1NUMBER_LESS1NUMBER_LESS_THAN_EQ1NUMBER_EQ1NUMBER_NOT_EQ1NUMBER_BETWEEN2NUMBER_NOT_BETWEEN2Text ConditionsConditionValuesTEXT_CONTAINS1TEXT_NOT_CONTAINS1TEXT_STARTS_WITH1TEXT_ENDS_WITH1TEXT_EQ1TEXT_IS_EMAIL0TEXT_IS_URL0Date ConditionsDates can be supplied in whatever format you like, but we recommentYYYY-MM-DD. You can also specify the exact day, a month (MM-YYYY), or just a year (YYYY). Relative dates (e.g. "today") are not currently supported.ConditionValuesDATE_EQ1DATE_BEFORE1DATE_AFTER1DATE_ON_OR_BEFORE1DATE_ON_OR_AFTER1DATE_BETWEEN2DATE_NOT_BETWEEN2DATE_IS_VALID0One-of ConditionsONE_OF_LISTvalues should be supplied as a comma separated list. There should be at least two values in the list. For single values, useTEXT_EQinstead. If an item in yourONE_OF_LISTcontains a comma, escape it with a backslash (e.g.,single value 1, multi-value 2\, 3will resolve to two values in the list).ConditionValuesONE_OF_RANGE1ONE_OF_LIST2+Other ConditionsTheCUSTOM_FORMULAvalue must be a formula that evaluates to TRUE or FALSE.ConditionValuesBLANK0NOT_BLANK0CUSTOM_FORMULA1BOOLEAN0clearclearremoves applied attributes (either fromapplyor manually added to the sheet remotely) from one or more sheets:cogs clear [keyword] [sheet-title-1] [sheet-title-2] ...The keyword must be one of:formats: sheet formattingnotes: sheet notesvalidation: data validation rulesall: formats, notes, and rulesAfter the keyword, you can supply zero or more sheet titles to remove attributes from. If no sheet titles are provided, the attribute(s) specified by the keyword will be removed fromallsheets.connectconnectis similar toinitin that it creates a new COGS project in the current directory. Instead of creating a new Google Spreadsheet, though, it connects to an existing one that you have already created. In order to runconnect, you must not have an existing COGS project in the directory. We also recommend that you are the owner of the Spreadsheet you are connecting, as you will need to transfer ownership to the service account defined in your credentials.cogs connect -k [spreadsheet-url-or-key] -c [path-to-credentials]The--key/-kargument accepts either the full sheet URL or just the key. You can find the spreadsheet key in the URL of the Google Spreadsheet:https://docs.google.com/spreadsheets/d/[SPREADSHEET-KEY]/edit#gid=0As withinit, you may exclude the-cargument if you have your credentials defined in the environment variableGOOGLE_CREDENTIALS. Note that this variable must be thecontentsof the credentials file, not the path to the file.After connecting to the existing spreadsheet, COGS will pause to ask you to share the Spreadsheet with the service email. It will provide a link to the sheet and the service email to share with. You can either give the service email "Editor" permissions, or transfer ownership.To give "Editor" access:Open the Google Spreadsheet in your browserClick "Share" in the upper right cornerEnter in the provided service email, uncheck "Nofity people", and click "Send"Return to the terminal and press ENTER to continueTo transfer ownership, click "Share" again and click the drop-down next to the service email and select "Make owner"Please be aware that if you do not transfer ownership to the service account,cogs deletewill not work. All other commands will work with just "Editor" access. If you do transfer ownership, you can always transfer ownership back to yourself using thesharecommand.connectwill automatically fetch the existing sheets from the remote Google Spreadsheet. To get the local copies, just runcogs merge.deleteRunningdeletereads the configuration data in.cogs/config.tsvto retrieve the spreadsheet ID. This spreadsheet is deleted in Google Sheets and the.cogsdirectory containing all project data is also removed. Any local TSV/CSV tables specified as sheets in the spreadsheet are left untouched.cogs deletediffRunningdiffwill display all file changes between local and remote sheets after runningcogs fetchor after updating a local sheet (before pushing or pulling those changes).cogs diffBy default, this will display all files with changes. If you wish to just see the changes for one or more paths, you can provide those paths:cogs diff path1.tsv path2.tsv ...diffopens a responsive scrolling window. To scroll down, press the down arrow. To scroll up, press the up arrow (see all navigation below). For large files with many columns, you can also scroll to the right with the right arrow and back to the left with the left arrow.The start of a diff for a sheet begins in bold with the file name (local and remote versions).The displayed text below has formatting for changes between the local and remote sheets:Lines that have beenremovedfrom the remote version of the sheet are in red text and begin with---.Lines that have beenaddedto the local version of the sheet are in green text and begin with+++.Lines that have beenchangedbetween the remote and local versions are in blue text and begin with->.In cells withchangedvalues, the cell is formatted as so:old text -> new textIf a sheet has been newly created or deleted, these changes will not appear indiff. Instead, usecogs status.To navigate the diff:↑: move one line up↓: move one line down→: move 20 characters right←: move 20 characters leftq: quitt: go to top (first line)b: go to bottom (last line)r: go to rightmost characters (last column)l: to to leftmost characters (first column)fetchRunningfetchwill sync the local.cogs/directory with all remote spreadsheet changes.cogs fetchThis will download all sheets in the spreadsheet to that directory as{sheet-title}.tsv- this will overwrite the existing sheets in.cogs/tracked/, but will not overwrite the local versions specified by their path. Any sheets that have been added withaddand then pushed to the remote sheet withpushwill be given their IDs in.cogs/sheet.tsv..cogs/format.tsvand.cogs/note.tsvare also updated for any cell formatting or notes on cells, respectively. Each unique format is given a numerical ID and is stored asCellFormat JSON.If a new sheet has been added to the Google spreadsheet, this sheet will be added to.cogs/sheet.tsvas an "ignored" sheet. While it appears in the sheets, it will not be downloaded and has no local path. If you wish to add an ignored sheet to tracking, usecogs add.To sync the local version of sheets with the data in.cogs/, runcogs merge.Note that if a sheet has beenrenamedremotely, the old sheet title will be replaced with the new sheet title. Any changes made to the local file corresponding to the old title will not be synced with the remote spreadsheet. Instead, once you runcogs merge, a new sheet{new-sheet-title}.tsvwill appear in the current working directory (the same as if a new sheet were created). It is the same as if you were to delete the old sheet remotely and create a new sheet remotely with the same contents. Usecogs mergeto write the new path - the old local file will not be deleted.ignoreRunningignoreon a given sheet title will start ignoring that sheet. This means that the cached copy of the sheet in the.cogs/directory is deleted, and the local version will no longer be updated when runningcogs pull.cogs ignore [sheet-title]You may only runignoreon a sheet that you are already tracking. This willnotremove either your local copy or the remote sheet.initRunninginitcreates a.cogsdirectory containing configuration data. This also creates a new Google Sheets spreadsheet and stores the ID. Optionally, this new sheet may be shared with users.cogs init -c [path-to-credentials] -t [project-title] -u [email] -r [role]gspreadneeds credentials to create a service account; you can either provide these with a file (-c [path]) or with an environment variable (GOOGLE_CREDENTIALS). The environment variable should be a string containing the contents of the credentials file. You must surround the contents with single quotes when setting this variable:export GOOGLE_CREDENTIALS='{...}'Options:-t/--title:required, title of the project which will be used as the title of the Google spreadsheet-c/--credentials: path toGoogle API credentialsin JSON format-u/--user: email of the user to share the sheet with (if a--roleis not specified, this user will be a writer)-r/--role: role of the user specified by--user:writerorreader-U/--users: path to TSV containing emails and roles for multiple users (header optional)The following files are created in the.cogs/directory when runninginit:config.tsv: COGS configuration, including the spreadsheet detailsformat.tsv: Sheet ID, cell location or range, and format IDs (the format for each format ID is stored as a JSON dictionary in.cogs/formats.json)note.tsv: Sheet ID, cell location, and note for all notessheet.tsv: Sheet names in spreadsheet and details (empty) - the sheets correspond to local tablesvalidation.tsv: Data validation conditionsAll other tasks will fail if a COGS project has not been initialized in the working directory.lsRunninglsdisplays a list of tracked sheet names and their local paths, even if the local path does not yet exist.cogs lsopenRunningopendisplays the URL of the spreadsheet.cogs openpullRunningpullwill sync local sheets with remote sheets. This combinescogs fetchandcogs mergeinto one step.cogs pullNote that if you make changes to a local sheet without runningcogs push, then runcogs pull, the local changeswill be overwritten.pushRunningpushwill sync the spreadsheet with your local changes. This includes creating new sheets for any added tables (cogs add) and deleting sheets for any removed tables (cogs rm). Any changes to the local tables are also pushed to the corresponding sheets.cogs pushThis will also push all notes and formatting from.cogs/format.tsvand.cogs/note.tsv.mergeRunningmergewill sync local sheets with remote sheets after runningcogs fetch.cogs mergeNote that if you make changes to a local sheet without runningcogs push, then runcogs fetch && cogs merge, the local changeswill be overwritten.mvRunningmvwill update the path of a local sheet.cogs mv [old_path] [new_path]This will only change the local path, not the sheet title. If you also wish to rename the sheet, you can do this with the-t/--titleoption:cogs mv [old_path] [new_path] -t [new_title]We recommend runningcogs pushaftercogs mvto keep the remote spreadsheet in sync.rmRunningrmwill stop tracking one or more local sheets. This will delete all local copies of the included paths, unless you include the optional-k/--keepflag.cogs rm path [path...] [-k]This does not delete the sheet(s) from the spreadsheet - usecogs pushto push all local changes to the remote spreadsheet and remove cached data about the sheet.shareRunningshareshares the spreadsheet with the specified user(s).cogs share -r [reader-email] -w [writer-email]There are three options:-r/--reader: email of the user to give read access to-w/--writer: email of the user to give write access to-o/--owner: email of the user to transfer ownership toPlease be aware that transfering ownership of the Spreadsheet prevent COGS from performing any administrative actions (e.g.,cogs delete). If you do transfer ownership and wish to delete the project, you should simply remove the.cogs/directory and then go online to Google Sheets and manually delete the project.statusRunningstatusshows the difference between local and remote copies of tracked sheets.cogs statusThere are five kinds of statuses (note that any changes to the remote spreadsheet will not be accounted for until you runcogs fetch)Modified locally: the sheet exists both locally and remotely (cached), but the local version has been edited since the last timecogs fetchorcogs pushwere runusecogs diff [path]to see detailsusecogs pushto sync local changes to remote version (overwriting any changes to remote not yet pulled)Modified remotely: the sheet exists both locally and remotely, butcogs fetchhas been run and returned a modified sheet since the last time the local version was editedusecogs diff [path]to see detailsusecogs pullto sync remote changes to local version (overwriting any changes to local not yet pushed)Added locally: the sheet exists locally and has been added to tracking, but is not yet pushed to the remote spreadsheetusecogs pushto add the sheet to the remote spreadsheetAdded remotely: the sheet exists remotely and has been added to tracking, but is not yet pulled to a local copyusecogs pullto add the sheet to locallyRemoved locally: the sheet exists remotely but has been removed from tracking usingcogs rmusecogs pushto remove the sheet from the remote spreadsheetRemoved remotely: the sheet exists locally but has been removed from remote spreadsheetusecogs pullto remove the sheet locally