Package: timid
Timid is a command line tool for running tests.  It differs from
Python tools like tox in that it is not limited to Python.  It uses a
YAML file to describe how to build the environment to run the test
in.Why Timid?Timid is intended to provide a very flexible test description
language.  It provides functionality for setting up various aspects of
the test environment, as well as for actually invoking the test
command.  While not too dissimilar from a Python tool like tox, Timid
does not make any assumptions about what that environment should look
like; in particular, it does not create a virtual environment unless
the test description includes the appropriate commands to do so.  This
makes it suitable for running any set of tests.Another aspect of Timid is the ability to reference any subset of test
steps from other files.  This enables easy reuse for complicated test
descriptions, and even means a library of test description fragments
may be easily established and used.  Timid also allows the working
directory to be directly set from the command line, allowing the test
descriptions to be separated from the actual code to test.  Finally,
Timid is extremely extensible; new test step actions and modifiers may
be created by including a class in the appropriate entrypoint
namespaces (“timid.actions” and “timid.modifiers”, respectively), and
extensions (namespace “timid.extensions”) may also be created that can
perform specific tasks under control of the command line–for
instance, an extension could allow a Timid test to run on a Github
pull request, setting test status information using the Github status
API.Basic Test Description SyntaxTest descriptions are YAML files consisting of a list of dictionaries,
where each dictionary describes astepin the testing process.  Each
step consists of oneactionand zero or moremodifierswhich alter
the action in some way.  A step may also have aname, which is used
to identify the step in the output, and adescription.  The action
and the modifiers are identified by the keys of the step dictionary;
the values of those keys identify the options for that action or
modifier.  The options may be any legal YAML entity, such as a string,
integer, boolean, list, or a dictionary; the documentation for each of
the actions and modifiers will describe what that action or modifier
expects.The test description could also be a smaller component of a YAML file
containing a dictionary; each value of this dictionary must be a list
of dictionaries, as described above.  This could be used to describe
several different but related tests for a single project (e.g., style
tests, unit tests, functional tests, and integration tests), or it
could be used to provide a library of test steps that can be included
using the “include” action.Templating and ExpressionsMany actions and modifiers allow Jinja2-style templates to be
specified for values, which enhances reusability of test description
components.  Jinja2-style expressions can also be used; an example
would be the “when” modifier, which provides simple conditional
control of an action.  Template variables can be set on the command
line, read from a YAML file, or set up directly in the test
description.SecurityTimid provides a way to mark both template variables and environment
variables as being “sensitive”.  This is to allow security-sensitive
data, such as usernames and passwords, to be used, while ensuring that
that sensitive data is scrubbed from any verbose or debugging output
from Timid itself.  For template variables, this can only be done from
the test description file, but environment variables can also be
marked sensitive by listing them in theTIMID_SENSITIVEenvironment variable, separated by your system’s path separator
character.  (On UNIX and Linux systems, this character would be the
“:” character.)  TheTIMID_SENSITIVEenvironment variable will
also be present in the environment of any subordinate processes,
updated with any additional environment variables marked as sensitive
by the test description; this can be used by test scripts to omit
sensitive information from the environment in debugging output.Extending TimidAs mentioned previously, Timid uses Python entrypoints for simple
extensibility.  Each action or modifier in a test description is
looked up for in the “timid.actions” or “timid.modifiers” namespaces,
respectively.  These entrypoints must map to subclasses oftimid.Actionortimid.Modifier, as appropriate.  Timid also
provides extensions, which allow extending the actual command line
interface and other per-step behavior; these are listed in the
“timid.extensions” interface, and the entrypoints must map to
subclasses oftimid.Extension.Creating a New ActionActions perform the actual test step.  Creating a new action is a
matter of extendingtimid.Action.  In the new action class, theschemaclass attribute must be set to a JSONSchema description of
the expected configuration; and the__call__()method, taking as
its sole argument acontextobject, must be defined to implement the
actual action; it should return an instance oftimid.StepResult.
Thetimid.Actionclass declares a__init__()method taking
four arguments (a context object, the name of the action (the key read
from the test description), the configuration for the action (the
value for that key), and astep addressobject); it validates the
configuration, then stores the last three arguments in thename,config, andstep_addrattributes, respectively.  (The context
object should not be stored; it will be passed in to the__call__()method.)There are two types of actions.  By default, alltimid.Actionsubclasses are instantiated while reading the test description, then
their__call__()methods are invoked in order during the actual
test run–or not invoked at all, if a syntax check is being
performed.  However, it is possible to create a “step action”, an
action invoked immediately after it is read from the test description;
this is used, for instance, to implement the “include” step, which
reads steps from another file and inserts them in place of the
“include” step.  These are implemented by setting thestep_actionclass attribute toTrueand having__call__()return a list oftimid.Stepobjects, instead of atimid.StepResultobject.Creating a New ModifierModifiers modify a step in some fashion, such as by running the step
in a loop or applying a conditional prior to invoking the step.
Creating a new action is a matter of extendingtimid.Modifier.
Like actions, the new subclass must have aschemaclass attribute
set to a JSONSchema description of the expected configuration, and a__init__()method identical to that fortimid.Actionis also
implemented; however, modifiers do not have a__call__()method,
and the class attributeprioritymust be set to an integer value.
Thepriorityattribute controls the order in which modifiers are
applied while running a step, with lower values invoked before higher
values.A modifier actually consists of a set of hook functions.  Thetimid.Modifiersuperclass contains default implementations of
these hook functions, so a developer need only override the ones
needed to implement the modifier.The first hook is theaction_conf()hook, which takes 5 arguments:
a context object, the class implementing the modified action, the name
of the action (key read from the test description), the configuration
for theaction(the__init__()method receives the configuration
for themodifier), and astep addressobject.  The hook function
must return the configuration that should be passed to the action
class, giving the modifier the opportunity to alter the configuration.The remaining two hooks are thepre_call()andpost_call()methods, which are invoked prior to and after calling the action’s__call__()method, respectively.  Thepre_call()method can
return atimid.StepResultobject, which aborts further processing
(including the call to the action) and proceeds with invoking anypost_call()methods.  Thepost_call()method receives thetimid.StepResultobject and can modify it or even replace it
entirely, by returning a different object.  Thepre_call()method
takes 4 arguments: a context object, a “pre_mod” list, a “post_mod”
list, and the instance of thetimid.Actionsubclass.  Thepost_call()method gets 5 arguments: a context object, the result
of the call (an instance oftimid.StepResult), the instance of thetimid.Actionsubclass, a “post_mod” list, and a “pre_mod” list.
The “pre_mod” and “post_mod” lists are lists oftimid.Modifierinstances that have lower priority and higher priority, respectively.
It should also be noted thatpost_call()is called in the inverse
order ofpre_call().Context ObjectsThe context object passed to the actions and modifier methods provides
several services throughout Timid.  Theverboseattribute contains
an integer value controlling the verbosity of Timid’s output (0 means
no output at all), anddebugis a boolean indicating whether
debugging is enabled.  Thevariablesattribute contains a
dictionary of template variables, andenvironmentcontains the
environment variables.  (The environment dictionary-like object also
allows control of the current working directory, by setting itscwdattribute, and itscall()method should be used to invoke
external programs.)Timid provides an interface to Jinja2, and two utility methods on the
context object facilitate this: thetemplate()method takes a
string and returns a callable of one argument that will render the
template, andexpression()works similarly for Jinja2
expressions.  (It is safe to pass objects other than strings to these
two methods as well; the result will still be a callable of one
argument, but no template expansion will be performed.)  The context
object should be passed to the callable returned bytemplate()andexpression().The usual way to use thetemplate()andexpression()methods
is to override the__init__()method of thetimid.Actionortimid.Modifiersubclass; the method should invoke the superclass’s
version of__init__()(using asuper()expression), and would
then process the configuration, saving the callables produced by
callingtemplate()andexpression().  Then, where the values
are used in the action’s__call__()or the modifier’s hook
methods, simply pass the context to the callable and use the result as
the actual value to use.Step AddressesTo aid debugging, each action or modifier has astep addressobject
associated with it.  The address has three attributes: the filename
from which the step was read (fname); the 0-based index of the
step within the file (idx); and the key for the list containing
the steps (key).  (This latter attribute will beNoneif the
file was a simple list of steps.)  The object also has a
straightforward string representation which includes the filename,
key, and step index (1-based; that is, ifidxis 3, the string
will identify the step as step 4).Creating an ExtensionExtensions are the most powerful extensibility mechanism in Timid.
Creating one is a matter of extendingtimid.Extensionand
implementing the desired hook methods, similar to creating a new
modifier, except that a Timid extension must implement anactivate()class method if it actually intends to do anything.
Additionally, atimid.Extensionsubclass must set thepriorityclass attribute to a numerical value, just like atimid.Modifiersubclass; extension hook functions will be called in the order
dictated by the priorities.The first hook method that an extension may implement is theprepare()method.  This must be a class method, and will receive
as its sole argument anargparse.ArgumentParserinstance, which
the extension may use to declare new command line options.  All
extensions will have theirprepare()method called during Timid
initialization.Once the command line has been processed byargparse.ArgumentParser, each extension’sactivate()method
will be called with a context object and anargparse.Namespacecontaining the results of the command line processing.  This method
must also be a class method, and must return eitherNoneor an
instance of thetimid.Extensionsubclass; if it returnsNone,
the extension is treated as inactive and no other hook methods will be
called.The remaining hook methods are all instance methods, called on the
object returned by theactivate()method.  Theread_steps()method is called with a context object and a list oftimid.Stepinstances; the extension may perform any in-place modifications to the
list of steps that are appropriate.  Thepre_step()andpost_step()methods are called before and after executing a step,
respectively;pre_step()is called with a context object, thetimid.Stepinstance, and the index of the step, and may return aTruevalue to cause the step to be skipped.  Thepost_step()method is called with the same arguments, and a fourth argument, which
will be atimid.StepResultobject, which it may alter in place;
the return value ofpost_step()is ignored.  Note thatpost_step()is called in extension order, in contrast to thepost_call()method oftimid.Modifierinstances.The final hook function is thefinalize()method, which is called
just before the command line tool exits.  It is called with a context
object and the result, which will typically beNonefor success,
or a text string indicating an error.  (It could also be called with
anExceptioninstance if an error occurred.)  This method’s return
value will replace the result.Debugging ExtensionsThe implementation of extensions explicitly ignores exceptions raised
by a given extension.  This would make it difficult to debug a newly
developed extension, so Timid provides a debugging mechanism: theTIMID_EXTENSION_DEBUGenvironment variable may be set to an
integer value, with larger values resulting in more verbose debugging.
IfTIMID_EXTENSION_DEBUGis present in the environment with no
value, or with a non-integer value, the debugging level will be set to
1; a debugging level of 0 (or any negative value) is exactly the same
as ifTIMID_EXTENSION_DEBUGwas not present in the environment at
all.Note that this environment variable is checked directly from the
environment, unlike theTIMID_SENSITIVEenvironment variable.
This means that the value used by extension debugging cannot be
altered by any instructions in the test description; only child
processes can be affected by such instructions.  Even command line
environment variable manipulations are ignored for the purposes of
extension debugging.  This design decision was made so that debugging
could be enabled before even calling the extensionprepare()method, which is called before any argument processing is done, and
thus prior to reading any test description files.