Package: tinuous
GitHub|PyPI|Anaconda|Issues|Changelogtinuousis a command for downloading build logs and (for GitHub
only) artifacts & release assets for a GitHub repository from GitHub Actions,
Travis-CI.com, and/or Appveyor.See <https://github.com/con/tinuous-inception> for an example setup that usestinuouswith GitHub Actions to fetch the CI logs fortinuousitself.Installationtinuousrequires Python 3.8 or higher.  Just usepipfor Python 3 (You have pip, right?) to installtinuousand its dependencies:python3 -m pip install tinuoustinuouscan also optionally integrate withDataLad.  To install DataLad
alongsidetinuous, specify thedataladextra:python3 -m pip install "tinuous[datalad]"tinuousis also available for conda!  To install, run:conda install -c conda-forge tinuousUsagetinuous [<global options>] <command> [<args> ...]Global Options-cFILE,--configFILERead configuration from the given file [default
value:tinuous.yaml]-EFILE,--envFILELoad environment variables from the given.envfile.  By default, environment
variables are loaded from the first file named
“.env” found by searching from the current
directory upwards.Warning: Care must be taken when this file
is located in a Git repository so as not to
publicly expose it: either list the file in.gitignoreor, if using DataLad or
git-annex, configure git-annex to prohibit
public sharing of the file.-lLEVEL,--log-levelLEVELSet the log level to the given value.  Possible
values are “CRITICAL”, “ERROR”,
“WARNING”, “INFO”, “DEBUG” (all
case-insensitive) and their Python integer
equivalents.  [default value: INFO]fetchCommandtinuous [<global options>] fetch [<options>]tinuous fetchreads a configuration file telling it what repository to
retrieve logs & artifacts for, where to retrieve them from, and where to save
them, and then it carries those steps out.Options--sanitize-secretsSanitize secrets from log files after
downloading-SFILE,--stateFILEStore program state (e.g., timestamps before
which all asset are known to have been fetched)
in the given file [default value:.tinuous.state.json]sanitizeCommandtinuous [<global options>] sanitize <path> ...Sanitize the given files, replacing all strings matching a secret regex with a
series of asterisks.ConfigurationThe configuration file is a YAML file containing a mapping with the following
keys:repo(required)The GitHub repository to retrieve assets for, in the formOWNER/NAMEvarsA mapping defining custom path template placeholders.  Each key is the name
of a custom placeholder, without enclosing braces, and the value is the
string to substitute in its place.  Custom values may contain standard path
template placeholders as well as other custom placeholders.ci(required)A mapping from the names of the CI systems from which to
retrieve assets to sub-mappings containing CI-specific configuration.
Including a given CI system is optional; assets will only be fetched from a
given system if it is listed in this mapping.The CI systems and their sub-mappings are as follows:githubConfiguration for retrieving assets from GitHub Actions.  Subfields:pathsA mapping givingtemplate stringsfor the
paths at which to save various types of assets.  If this is empty
or not present, no assets are retrieved.  Subfields:logsA template string that will be instantiated for each workflow
run to produce the path for the directory (relative to the
current working directory) under which the run’s build logs
will be saved.  If this is not specified, no logs will be
downloaded.artifactsA template string that will be instantiated for each workflow
run to produce the path for the directory (relative to the
current working directory) under which the run’s artifacts will
be saved.  If this is not specified, no artifacts will be
downloaded.releasesA template string that will be instantiated for each
(non-draft, non-prerelease) GitHub release to produce the path
for the directory (relative to the current working directory)
under which the release’s assets will be saved.  If this is not
specified, no release assets will be downloaded.workflowsA specification of the workflows for which to retrieve assets.
This can be either a list of workflow basenames, including the file
extension (e.g.,test.yml, not.github/workflows/test.yml)
or a mapping containing the following fields:includeA list of workflows to retrieve assets for, given as either
basenames or (whenregexis true)Python regular
expressionsto match against basenames.  Ifincludeis
omitted, it defaults to including all workflows.excludeA list of workflows to not retrieve assets for, given as either
basenames or (whenregexis true)Python regular
expressionsto match against basenames.  Ifexcludeis
omitted, no workflows are excluded.  Workflows that match bothincludeandexcludeare excluded.regexA boolean.  If true (default false), the elements of theincludeandexcludefields are treated asPython
regular expressionsthat are matched (unanchored) against
workflow basenames; if false, they are used as exact namesWhenworkflowsis not specified, assets are retrieved for all
workflows in the repository.travisConfiguration for retrieving logs from Travis-CI.com.  Subfield:pathsA mapping givingtemplate stringsfor the
paths at which to save various types of assets.  If this is empty
or not present, no assets are retrieved.  Subfield:logsA template string that will be instantiated for each job of
each build to produce the path for the file (relative to the
current working directory) in which the job’s logs will be
saved.  If this is not specified, no logs will be downloaded.appveyorConfiguration for retrieving logs from Appveyor.  Subfields:pathsA mapping givingtemplate stringsfor the
paths at which to save various types of assets.  If this is empty
or not present, no assets are retrieved.  Subfield:logsA template string that will be instantiated for each job of
each build to produce the path for the file (relative to the
current working directory) in which the job’s logs will be
saved.  If this is not specified, no logs will be downloaded.accountName(required)The name of the Appveyor account to which the
repository belongs on AppveyorprojectSlugThe project slug for the repository on Appveyor; if not specified,
it is assumed that the slug is the same as the repository namesinceA timestamp (date, time, & timezone); only assets for builds started after
the given point in time will be retrieved.  If not specified, the cutoff
set bymax-days-backwill be used.As the script retrieves new build assets, it keeps track of their starting
points.  Once the assets for all builds for the given CI system &
configuration have been fetched up to a certain point, the timestamp for
the latest such build is stored in the state file and used as the newsincevalue for the respective CI system on subsequent runs.  If thesincesetting in the configuration file is then updated to a newer
timestamp, the configuration will override the value in the state file, and
the nexttinuousrun will only retrieve assets after the new setting.max-days-backAn integer specifying the maximum number of days back to look for builds;
defaults to 30.Ifsinceis earlier than the date indicated by this value, and if the
timestamp for a given CI system saved in the state file is either missing
(i.e., if this is the first run oftinuousagainst the CI system) or
older thansince, thentinuouswill use the explicitly-specifiedsincevalue as the cutoff and ignoremax-days-back.untilA timestamp (date, time, & timezone); only assets for builds started before
the given point in time will be retrievedtypesA list of build trigger event types; only assets for builds triggered by
one of the given events will be retrieved.  If this is not specified,
assets will be retrieved for all recognized event types.The recognized event types are:cronA build run on a schedulemanualA build triggered manually by a human or through the CI system’s APIprA build in response to activity on a pull requestpushA build in response to new commitssecretsA mapping from names (used in log messages) toPython regular
expressionsmatching secrets to sanitizeallow-secrets-regexAny strings that match asecretsregex and also match this regex will
not be sanitized.  Note thatallow-secrets-regexis tested against just
the substring that matched asecretsregex without any surrounding
text, and so lookahead and lookbehind will not work in this regex.dataladA sub-mapping describing integration oftinuouswithDataLad.
Subfields:enabledA boolean.  If true (default false), DataLad must be installed, the
current directory will be converted into a DataLad dataset if it is not
one already, the assets will optionally be divided up into subdatasets,
and all new assets will be committed at the end of a run oftinuous fetch.pathtemplate strings may contain//separators
indicating the boundaries of subdatasets.cfg_procProcedure to run on the dataset & subdatasets when creating themA sample config file:repo:datalad/dataladvars:path_prefix:'{year}//{month}//{day}/{ci}/{type}'build_prefix:'{path_prefix}/{type_id}/{build_commit[:7]}'ci:github:paths:logs:'{build_prefix}/{wf_name}/{number}/logs/'artifacts:'{build_prefix}/{wf_name}/{number}/artifacts/'releases:'{path_prefix}/{release_tag}/'workflows:-test_crippled.yml-test_extensions.yml-test_macos.ymltravis:paths:logs:'{build_prefix}/{number}/{job}.txt'appveyor:paths:logs:'{build_prefix}/{number}/{job}.txt'accountName:mihprojectSlug:dataladsince:2021-01-20T00:00:00Zmax-days-back:14types:[cron,manual,pr,push]secrets:github:'\bgh[a-z]_[A-Za-z0-9]{36,}\b'docker-hub:'\b[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}\b'appveyor:'\b(v2\.)?[a-z0-9]{20}\b'travis:'\b[a-zA-Z0-9]{22}\b'aws:'\b[a-zA-Z0-9+/]{40}\b'datalad:enabled:truecfg_proc:text2gitPath TemplatesThe path at which assets for a given workflow run, build job, or release are
saved is determined by instantiating the appropriate path template string given
in the configuration file for the corresponding CI system.  A template string
is a filepath containing placeholders of the form{field}, where the
available placeholders are:PlaceholderDefinition{year}The four-digit year in which the build was started or
the release was published{month}The two-digit month in which the build was started or
the release was published{day}The two-digit day in which the build was started or the
release was published{hour}The two-digit hour at which the build was started or
the release was published{minute}The two-digit minute at which the build was started or
the release was published{second}The two-digit second at which the build was started or
the release was published{timestamp}The date & time at which the build was started or the
release was published.  This is a Pythondatetimevalue; it can be formatted with astrftime()format
string by writing{timestamp:FORMAT}, e.g.,{timestamp:%Y-%b-%d}will produce a string of the
form “2021-Jun-14”.  If written as just{timestamp}, the date & time will be formatted in
ISO 8601 format.{timestamp_local}The date & time at which the build was started or the
release was published, in the local system timezone.
This is formatted in the same way as{timestamp}.{ci}The name of the CI system (github,travis, orappveyor){type}The event type that triggered the build (cron,manual,pr, orpush), orreleasefor
GitHub releases{type_id}Further information on the triggering event; forcronandmanual, this is a timestamp for the
start of the build; forpr, this is the number of
the associated pull request, orUNKif it cannot be
determined; forpush, this is the escaped[1]name
of the branch to which the push was made (or possibly
the tag that was pushed, if using Appveyor)[2]{release_tag}(``releases_path`` only)The release tag{build_commit}The hash of the commit the build ran against or that
was tagged for the release.  Note that, for PR builds
on Travis and Appveyor, this is the hash of an
autogenerated merge commit.{commit}The hash of the original commit that triggered the
build or that was tagged for the release.  For pull
request builds, this is the head of the PR branch, orUNKif it cannot be determined.  For other builds
(along with PR builds on GitHub Actions), this is
always the same as{build_commit}.{number}The run number of the workflow run (GitHub) or the
build number (Travis and Appveyor)[2]{status}The success status of the workflow run (GitHub) or job
(Travis and Appveyor); the exact strings used depend on
the CI system[2]{common_status}The success status of the workflow run or job,
normalized into one ofsuccess,failed,errored, orincomplete[2]{wf_name}(GitHub only)The escaped[1]name of the workflow[2]{wf_file}(GitHub only)The basename of the workflow file
(including the file extension)[2]{run_id}(GitHub only)The unique ID of the workflow run[2]{job}(Travis and Appveyor only)The number of the job,
without the build number prefix (Travis) or the job ID
string (Appveyor)[2]{job_index}(Travis and Appveyor only)The index of the job in
the list returned by the API, starting from 1[2]{job_env}(Appveyor only)The escaped[1]environment
variables specific to the job[2]{job_env_hash}(Appveyor only)The SHA1 hash of{job_env}before
escaping[2][1](1,2,3)Escaping consists of percent-encoding the characters\/<>:|"?*%and
replacing each whitespace character with a space.[2](1,2,3,4,5,6,7,8,9,10,11)These placeholders are only available forpathandartifacts_path, notreleases_pathA placeholder’s value may be truncated to the firstncharacters by writing{placeholder[:n]}, e.g.,{commit[:7]}.All timestamps and timestamp components (other than{timestamp_local}) are
in UTC.Path templates may also contain custom placeholders defined in the top-levelvarsmapping of the configuration.AuthenticationNote that environment variables can be loaded from a.envfile as an
alternative to setting them directly in the environment.GitHubIn order to retrieve assets from GitHub, a GitHub OAuth token must be specified
either via theGITHUB_TOKENenvironment variable or as the value of thehub.oauthtokenGit config option.TravisIn order to retrieve logs from Travis, a Travis API access token must be either
specified via theTRAVIS_TOKENenvironment variable or be retrievable by
runningtravis token--com--no-interactive.A Travis API access token can be acquired as follows:Install theTravis command line client.Runtravis login--comto authenticate.If your Travis account is linked to your GitHub account, you can
authenticate by runningtravis login--com--github-token$GITHUB_TOKEN.If the script will be run on the same machine that the above steps are
carried out on, you can stop here, and the script will retrieve the token
directly from thetraviscommand.Runtravis token--comto retrieve the API access token.The Travis integration also requires a GitHub OAuth token in order to look up
information on pull requests that the Travis API does not report; this token
must be specified in the same way as for the GitHub integration.AppveyorIn order to retrieve logs from Appveyor, an Appveyor API key (for either all
accessible accounts or just the specific account associated with the
repository) must be specified via theAPPVEYOR_TOKENenvironment variable.
Such a key can be obtained at <https://ci.appveyor.com/api-keys>.Cron IntegrationIf you want to set up scheduled runs oftinuouson a Linux server, one way
is as follows:Create a new directory andcdinto it.Create a file namedtinuous.yamlin this directoryas described aboveCreate a file named.envin this directory containing any needed
authentication tokens.  Entries are of the formNAME=value, e.g.:GITHUB_TOKEN=ghp_abcdef0123456789
TRAVIS_TOKEN=asdfghjkl
APPVEYOR_TOKEN=v2.qwertyuiopCreate a Pythonvirtualenvto provide an isolated environment to installtinuousinto:python3 -m venv venvInstalltinuousinside the virtualenv:venv/bin/pip install tinuousIf you want to use DataLad withtinuous, you need to install it as well,
even if it’s already installed outside the virtualenv:venv/bin/pip install dataladRuntinuousto fetch your first logs and test your configuration:venv/bin/tinuous fetchOnce you’re satisfied with yourtinuousconfig, set up scheduled runs by
creating a cronjob of the form:0 0 * * * cd /path/to/directory && chronic flock -n -E 0 .tinuous.lock venv/bin/tinuous fetchThis job runs once a day at midnight; adjust the cron expression to taste.
We usechronic(frommoreutils) to suppress output unless the command
fails, thus preventing e-mails full of log messages for every run.flockis used to ensure that no more than one instance oftinuousis
running at a time.If you want to commit your logs to a Git repository, first make sure that.env,venv/, and.tinuous.lockare included in the repository’s.gitignore.  Consider setting up the repository withDataLad; when the
DataLad integration is enabled,tinuouswill automatically commit any
new logs at the end of a run.If you’re using a regular Git repository instead, you can commit any new
logs at the end of a run by adding the following script to yourtinuousdirectory:#!/bin/bashset-exvenv/bin/tinuousfetchgitadd--allif!gitdiff--cached--quietthengitcommit-m"Ran tinuous"# Uncomment if you want to push the commits to a remote repository:#git pushfiand changing your cronjob to:0 0 * * * cd /path/to/directory && chronic flock -n -E 0 .tinuous.lock bash name-of-script.shIf you ever need to upgradetinuous, run the following command inside
yourtinuousdirectory:venv/bin/pip install --upgrade tinuousEnjoy your collection of logs, build artifacts, and/or release assets!