Package: tc-admin
Thetc-adminlibrary supports administration of the runtime configuration of a Taskcluster deployment.
This means creation and maintenance of resources such as roles, hooks, and worker pools, based on version-controlled specifications..UsageThis library is used as a dependency of a Python application containing code and configuration specific to the taskcluster deployment(s) being administered.
The project should contain atc-admin.pythat serves to define the application configuration.Assuming that is in place, the tool is easy to use:After installing the app, runtc-admin generateto generate the expected set of resources (use--jsonto get JSON output).
This will requireTASKCLUSTER_ROOT_URLto be set in the environment, to know which deployment to talk to.
Similarly,tc-admin currentwill generate the current set of resources (optionally with--json).
To compare them, runtc-admin diff.If the configuration includes secrets, you may want to pass the--without-secretsoption.
This option skips managing the content of secrets, and thus needs neither access to secret values nor Taskcluster credentials to fetch secrets.Runtc-admin applyto apply the changes.
Note that onlyapplywill require Taskcluster credentials, and it's a good practice to only set TC credentials when running this command.Seetc-admin <command> --helpfor more useful options.ChecksChecks are a way to double-check that purpose-specific invariants are satisfied in a Taskcluster deployment.
For example, it may be important to check that only specific repository roles have scopes to create tasks in critical worker pools.
Checks are defined as normal Python tests, and have access to the current and generated configurations.If the app has checks set up, thentc-admin checkwill run those checks.Quick Guide to Library OperationThe operation of this tool is pretty simple: it generates a set of expected Taskcluster resources, downloads existing resources from the Taskcluster API, and compares them.
A collection of resources also specifies the set of "managed" resources -- this allows deletion of resources that are no longer expected, without risk of deletingeverythingin the Taskcluster API.After generation, resources can be "modified".
This is typically used to make minor changes to resources depending on environment.
For exmaple, in a staging environment, hook schedules might be removed.App Configurationtc-adminandtc-admin.pyA tc-admin app is configured bytc-admin.py.
This is a Python file which is responsible for creating and customizing anAppConfigobject.fromtcadmin.appconfigimportAppConfigappconfig=AppConfig()# .. customizeThetc-admincommand looks fortc-admin.pyin the current directory, or in the directory given by$TC_ADMIN_PYor command-line argument--tc-admin-py.
Like most Python modules, the global__file__is set whentc-admin.pyis executed, and can be used to determine relative paths.Beforetc-admin.pyis executed, the current working directory is changed to the directory containing it.
This enables relative imports as well as loading files with relative paths (such as with LocalLoader, below).Programmatic InterfaceThis library can also be used programmatically.
Simply create an AppConfig object directly and calltcadmin.main.mainwith it:fromtcadmin.appconfigimportAppConfigfromtcadmin.mainimportmaindefboot():appconfig=AppConfig()# .. customizemain(appconfig)if__name__=="__main__":boot()Note that the current directory is not automatically set in this case.AppConfigThe AppConfig object contains a number of properties that can be customized, described below.
During execution, the current AppConfig object is available fromAppConfig.current().
This can be useful when generators or modifiers are defined in separate Python modules.GeneratorsGenerators generate expected Taskcluster resources and defined the managed resource names.
Each generator is an async function that is given aResourcesobject and is expected to callresources.manageandresources.update.Generators are registered withappconfig.generators.register, most easily with a decorator:@appconfig.generators.registerasyncdefupdate_resources(resources):# modify in place ...When generating secrets, respect the--with-secretsoption, and generate secrets without values when it is false.
You can also use this option to determine whether the generation process requires access to secret values.
This allows generation runs with--without-secretsto occur without any credentials or access to secret values.@appconfig.generators.registerasyncdefupdate_resources(resources):# modify in place ...ifappconfig.options.get('--with-secrets'):secretstore=load_secret_values()resources.add(Secret(name="top-secret/cookie-recipe",secret=secretstore.decrypt('recipes/double-chocolate-chip')))else:resources.add(Secret(name="top-secret/cookie-recipe"))ModifiersModifiers are responsible for modifying an existing set of resources.
Since resources are immutable, the signature differs slightly from generators:@appconfig.modifiers.registerasyncdefmodify_resources(resources):# return new set of resourcesreturnresources.map(..)Modifiers are called sequentially in the order in which they were registered.CallbacksCallbacks are external function from your own application that can be executed at specific times during thetc-admin applyexecution:Abefore_applycallback will run before a resource is created, updated or deleted,Aafter_applycallback will run after a resource has beencreated, updated or deleted.Supported actions are :createupdatedeleteBy default all actions are used.All resources are supported by callbacks, and enabled by default. If you want to limit your callback to some resources, you need to specify them using their class (not a string).You can declare your callbacks as:fromtcadmin.resourcesimportSecretasyncdefmy_action(action,resource):print("Got a callback on",action,resource)# Will call your function when a secret has been updated or deletedappconfig.callbacks.add("after_apply",my_action,actions=["update","delete"],resources=[Secret,])Command-Line OptionsApps can add additional command-line options, the values of which are then available during resource generation.To register an option, callappconfig.options.add, with the full option name and any of the following keyword options:required- if True, the option is requiredhelp- help string to be shown intc-admin generate --helpdefault- default value for the optionTo retrieve the option value during generation, callappconfig.options.get(name).
All together, then:appconfig.options.add("--branch",help="configuration branch to read from")@appconfig.generators.registerasyncdefupdate_resources(resources):branch=appconfig.options.get("--branch")# ...As a special case, the--with-secretssecret is available through this same mechanism.ChecksTheappconfig.check_pathproperty gives the path of the checks to run fortc-admin check, relative to the current directory.
This directory is a "normal" pytest directory.To help distinguish checks from tests, include apytest.iniin this directory:[pytest]python_classes=Check*python_files=check_*.pypython_functions=check_*description_prefixTheappconfig.description_prefixproperty allows the users to customize the prefix of the description.
This can be customized in thetc-admin.pyas follows:fromtcadmin.appconfigimportAppConfigappconfig=AppConfig()appconfig.description_prefix="YOUR_CUSTOM_PREFIX"The DEFAULT value of the description_prefix is*DO NOT EDIT* - This resource is configured automatically.\n\nRoot URLFor the common case of a configuration that applies to only one Taskcluster deployment, specify that deployment's root URL intc-admin.py:fromtcadmin.appconfigimportAppConfigappconfig=AppConfig()appconfig.root_url="https://taskcluster.example.com"To support more complex cases, this value can also be an async callable.
It will be invoked once, after theclickoptions have been processed, so it can accessappconfig.optionsif necessary.The current root URL is available from an async helper function:fromtcadmin.util.root_urlimportroot_urlasyncdeffoo():print(awaitroot_url())This will retrieve the value from the AppConfig or, if that is not set, fromTASKCLUSTER_ROOT_URL.
If both are set, and the values do not match, it will produce an error message.Loading Config SourcesMost uses of this library load configuration data from some easily-modified YAML files.
Thetcadmin.util.configpackage provides some support for loading and parsing these files.
All of this is entirely optional; use what is appropriate to the purpose.LoadersFirst, define a loader that can load data from files.fromtcadmin.util.configimportLocalLoaderloader=LocalLoader()The LocalLoader class knows how to load configuration from files relative totc-admin.py.
It has aloadmethod that will load data, optionally parsing it as YAML:data=loader.load("data.bin")aliases=awaitloader.load("aliases.yml",parse="yaml")You can also define your own loader class.
Just implement theload_rawmethod to return bytes, given a filename.ConfigYAML data is inconvenient to deal with in Python, introducig a lot of[".."]noise.
Commonly, config files are either a top-level array, or a top-level object with named "stanzas" of configuration.
The ConfigList and ConfigDict classes support these formats.
We suggest using these with the Pythonattrslibrary.Define a class that inherits from either of these classes, specifies the filename to load from, and has anItemclass for the items in the collection:fromtcadmin.util.configimportConfigListclassWorkers(ConfigList):filename="workers.yml"@attr.sclassItem:workerId=attr.ib(type=str)bigness=attr.ib(type=int,default=1)Then simply callawait Workers.load(loader)to load aworkers.ymlthat looks something like-workerId:smallbigness:5-workerId:hugebigness:5000The ConfigDict class is similar, but parses files likesmall:bigness:5huge:bigness:5000ConfigList creates newIteminstances from array elementsitemwithItem(**item).
ConfigDict creates newIteminstances fromk: itemwithItem(k, **item).
This approach is compatible withattrs, where in the latter casekshould be the first attribute defined.If array elements or object values are not themselves YAML objects, add a class method namedtransform_itemto transform the data in the YAML file into a Python dictionary.
For example:classWorkers(ConfigList):@classmethoddeftransform_item(cls,item):# given a simple string, assume that is the workerId and apply defaultsifisinstance(item,str):return{"workerId":item}returnitem...ResourcesThetcadmin.resourcespackage contains clasess for defining Taskcluster resources and collections.fromtcadmin.resourcesimportResourcesTheResourcesclass defines a collection of resources and tracks what resources are managed.
Resources found in the Taskcluster deployment that match the "managed" patterns but are not generated will be deleted onapply.
The class has the following methods:resources.add(resource)- add a resource to the collection.  The resource must be managed.resources.update(iterable)- add an iterable full of resources to the collection.  All resources must be managed.resources.manage(pattern)- consider reources matching regular expression stringpatternto be managedresources.is_managed(id)- return true if the given resource is managedresources.filter(pattern)- return a new Resources object containing only resources matching the given regular expression stringresources.map(functor)- return a new Resources object, with fuctor applied to each resource.  This is typically used in modifiers.Resources must be unique -- tc-admin cannot manage multiple hooks with the same name, for example.
However, some resource kinds support merging, where adding a resource with the same identity as one that already exists "merges" it into the existing resource.
See the description of roles, below.The remaining classes represent individual resources.
Each has anidformed from its kind and the unique identifier for the resource in the Taskcluster.
For example,Hook=release-hooks/beta-release.
Resources are immutable once created, but can be "evolved" (returning a new resource) withrsrc.evolve(**updates).Resources with descriptions automatically prepend a "DO NOT EDIT" prefix to dissuade users from editing them in the Taskcluster UI.Hookfromtcadmin.resourcesimportHook,Bindinghook=Hook(hookGroupId=..,hookId=..,name=..,description=..,owner=..,emailOnError=..,schedule=(..,..),bindings=(..,..),task={..},triggerSchema={..})Most of these fields correspond directly to the Taskcluster definition.
Bothscheduleandbindingsmust be tuples, not lists (as lists are mutable).
The items inscheduleare cron-like strings.
The items inbindingsare instances ofBinding(exchange=.., routingKeyPattern=..).Secretfromtcadmin.resourcesimportSecretsecret=Secret(name=..,secret=..)# or, when not managing secret valuessecret=Secret(name=..)Secrets are managed using the Secret resource type.
While Taskcluster supports expiration times on secrets, this library sets those times the far future, effectively creating non-expiring secretsThis library is careful to not display secret values in its output.
Instead, it displays<unknown>when not managing secret values, and displays a salted hash of the secret value when managing secret values.
The salted hash allowstc-admin diffto show that a secret value has changed, without revealing the value of that secret.
The salt includes a per-run salt, and the name of the secret, with the result that even if two secrets have the same value, they will be shown with different hashes intc-admin generate.Rolefromtcadmin.resourcesimportRolerole=Role(roleId=..,description=..,scopes=(..,..))As with hooks,scopesmust be a tuple (not a list) of strings.Roles can be merged if their descriptions match.
The resulting role contains the union of the scopes of the input roles.
This functionality makes management of roles easier in cases where different parts of the generation process may add scopes to the same role.For example:resources.add(Role(roleId="my-role",description="My Role",scopes=["scope1"]))resources.add(Role(roleId="my-role",description="My Role",scopes=["scope2"]))This will result in a single Role with scopes["scope1", "scope2"].Clientfromtcadmin.resourcesimportClientclient=Client(clientId=..,description=..,scopes=(..,..))Clients work much like roles.
As with roles,scopesmust be a tuple (not a list) of strings.
This library does not manage access tokens: it discards them from the response toauth.createClient.
The expectation is that project admins who need credentials for the managed clients will callauth.resetAccessTokenand use the returned token.Clients configured by this library have an expiration date far in the future.
Like roles, the clients managed here last "forever".WorkerPoolfromtcadmin.resourcesimportWorkerPoolworkerPool=WorkerPool(workerPoolId=..,providerId=..,description=..,owner=..,config={..},emailOnError=..)All attributes of this class match the Taskcluster definition.UtiliitesThe library provides a number of utilities for common application requirements.NOTE: only functions described in this README are considered stable.
Other functions defined by the library may change without notice.ScopesAs an aid to writing checks, tc-admin supplies local implementations of various scope-related algorithms.fromtcadmin.util.scopesimportsatisfies,normalizeScopes,ResolverThesatisfiesfunction determines scope satisfaction, without any expansion.
Satisfaction means that the first argument contains all scopes in the second argument.assertsatisfies(['balloons:*','cake:birthday'],['baloons:mylar:happy-birthday'])ThenormalizeScopesfunction normalizes a scopeset, removing redundant scopes and sorting.assertnormalizedScopes(['balloons:*','balloons:mylar:*'])==['baloons:*']Finally,Resolvercan perform scope expansion.
It is initialized with a dictionary mapping roleIds to scope lists.
Alternately, it can be initialized from aResourcesinstance usingResolver.from_resources(resources).ItsexpandScopesmethod behaves identically to the remote callauth.expandScopes.resolver=Resolver.from_resources(resources)assertresolver.expandScopes(['assume:clown:grimaldi'])==['assume:clown:grimaldi','ruffle:full']aiohttp sessionThe library usesaiohttpto communicate with Taskcluster, and establishes a single session for efficiency.
Applications can use the same session for any other HTTP operations.fromtcadmin.util.sessionimportaiohttp_sessionasyncdeffoo():# ...asyncwithaiohttp_session().get(url)asresponse:response.raise_for_status()result=awaitresponse.read()Tests and checks can set this value usingwith_aiohttp_session:fromtcadmin.util.sessionsimportwith_aiohttp_sessionimportpytest@pytest.mark.asyncio@with_aiohttp_sessionasyncdeftest_something():# ...MatchListA MatchList is a list of regular expressions that can determine whether a given
string matches one of those patterns.  Patterns are rooted at the left, but
should use$where required to match the end of the string.fromtcadmin.util.matchlistimportMatchListml=MatchList()ml.add("fo+$")ml.add("ba+r$")assertml.matches("foo")This functionality is used to track managed resources, but may be useful otherwise.DevelopmentTo install for development, in a virtualenv:pip install -e [path]And to run flake8 and tests:python setup.py flake8
python setup.py testThe library usesBlackto format code.pip install black
black tcadminReleasingTo release:update version insetup.pyandgit commit -m "vX.Y.Z"git tag vX.Y.zpush those changes tomain./release.sh --realand enter your pypi credentials when prompted (omit the--realto try it against the testing pypi, if you're not sure)Find the tag inhttps://github.com/taskcluster/tc-admin/releasesand create a new release with a brief description of the changes