Package: python-matrix-runner
Matrix Runner for PythonAllows easy top level command line interface generation for matrix configurations.InstallInstallation using pip:pip install python-matrix-runnerThe appearance of matrix runner can be configured by placing a copy of thedefault.confinto the users Python site base directory (user global
configuration) or next to a build script (local configuration). Figuring out
the right locations is assisted by the following command:$ python -m matrix_runner config -hPreferencesThe following preferences can be configured:globalGlobal settings.loglevelThe default log level, one of:
DEBUG, INFO, WARNING, ERROR, CRITICALcolorsSettings for colorized log output.Colors can be one of: black, red, green, yellow, blue, purple, cyan, whiteStyle prefix can be one of: bold_, thin_Background color prefix: bg_DEBUGDebug level log output.INFOInfo level log output.WARNINGWarning level log output.ERRORError level log output.CRITICALCritical level log output.configMatrix configuration column.actionAction column.summary_configSummary table matrix configuration column.summary_successSummary table success cell.summary_unstableSummary table unstable cell, i.e. failing test cases.summary_skipSummary table skip cell.summary_failSummary table fail cell.summary_errorSummary table error cell.Example UsageWriting the following tobuild.py:from enum import Enum
from matrix_runner import main, matrix_axis, matrix_action, matrix_command, matrix_filter

@matrix_axis("alpha", "a", "A configuration axis")
class MyAlphaAxisValue(Enum):
    Value1 = ('value1', 'v1')
    Value2 = ('value2', 'v2')
    Value3 = ('value3', 'v3')

@matrix_action
def dump(config):
    """Dump configuration to console"""
    yield dump_config(config)

@matrix_command()
def dump_config(config):
    return ["sh", "-c", f"echo '{config}'"]

if __name__ == "__main__":
    main()Can be executed as:$ ./build.py --help
usage: build.py [-h] [--silent | --verbose | --debug] [--pairwise] [--slice <HERE>/<TOTAL>]
                [--alpha ALPHA] action [action ...]

positional arguments:
  action                Action(s) to run: dump

optional arguments:
  -h, --help            show this help message and exit
  --silent              Silent mode, only errors are shown.
  --verbose             Verbose log output.
  --debug               Debug log output.
  --pairwise, -2        Reduce number of combinations using pairwise algorithm.
  --slice <HERE>/<TOTAL>
                        Cut set of combinations into <TOTAL> number of slices and run ony <HERE>th one.
  --alpha ALPHA, -a ALPHA
                        A configuration axis: value1|v1, value2|v2, value3|v3

$ ./build.py -a v[23] dump
[value2](dump:dump_config) /usr/bin/sh -c echo 'Config(alpha=<MyAlphaAxisValue.Value2: ('value2', 'v2')>)'
[value2](dump:dump_config) (Hello, World): Config(alpha=<MyAlphaAxisValue.Value2: (value2, v2)>)
[value2](dump:dump_config) /usr/bin/sh succeeded with exit code 0
[value3](dump:dump_config) /usr/bin/sh -c echo 'Config(alpha=<MyAlphaAxisValue.Value3: ('value3', 'v3')>)'
[value3](dump:dump_config) Config(alpha=<MyAlphaAxisValue.Value3: (value3, v3)>)
[value3](dump:dump_config) /usr/bin/sh succeeded with exit code 0

Matrix Summary
==============

alpha    dump
-------  -------
value2   success
value3   successMatrix Runner automatically generates a comprehensivecommand linehelp.
Giving at least one action the commands for all permutations of configured
matrix axes are automatically executed one by one. In this example the
execution is limited by a regex patternv[23].Detailed UsageThe Matrix Runner constructs what gets executed by defining these elements:AxisActionCommandFilterAxisAxes build up the actual configuration matrix to be rolled out. Each degree
of freedom in ones set of configuration can be mapped to a single axis by
annotating anEnumwith@matrix_axis:@matrix_axis("alpha", abbrev="a", desc="A configuration axis")
class MyAlphaAxisValue(Enum):
    Value1 = ('value1', 'v1')
    Value2 = ('value2', 'v2')
    Value3 = ('value3', 'v3')This adds an axis namedalphato the current builds configuration matrix.
The optional argumentabbrevcan be used to specify a shortcut to be
used from the command line, i.e.--alphaor-a. And the optional
argumentdescenriches the argument parser with a help string.The values in such an Enum can either be plain strings or tuples of strings.
Tuples can be used to provide multiple string representations for a single
value. In the example one can provideValue2on the command line as--alphavalue2or just as-av2. Alternatively axis values can be
selected by providing afnmatch.fnmatchpattern, e.g.-av[23]. Be aware
that such a pattern selects all values which have at least one matching string
representation in its tuple.The overall configuration matrix is calculated by permuting all values from all
specified axes. In a typical scenario it easily happens that some combinations
are known to be inappropriate. These can be filtered by specifyingFilter’s.NoteDon’t use the keywordshelporpairwiseas axis names. Same
applies to abbreviationshand2for those keywords. These
are already used for staticCommand Linearguments.ActionActions are used to capture different workflow steps, such ascompileandrunfor unit tests. A step is simply defined by decorating a function
with@matrix_action:@matrix_action
def dump(config: Config <, results: List[Result]>):
    """Dump configuration to console"""
    pre_process()
    yield dump_config(config, 'Hello', 'World')
    post_process(<results>)The function itself needs to return aGeneratorgenerating a list ofCommand’s. The function is called once in preparation for each matrix
configuration.Theconfigargument gives access to the selected _Config permutation. It
can be used to generate the commands depending on the actual configuration.The optionalresultsargument can be used to gain access to the list ofCommandresults gathered so far, e.g. for adding post-processing.The function needs toyieldCommand’s, i.e.dump_configneeds to be
an annotated command function. Pre and post processing code can be added
around.CommandCommands are actual command lines to be executed while forwarding their
standard output and standard error streams. A command is defined by decorating
a function with@matrix_command:@matrix_command()
def dump_config(*args, **kwargs):
    return ["sh", "-c", f"echo '{args}: {kwargs}'"]The decorator takes optional keyword arguments to fine control how the returned
command line should be executed throughsubprocess.Popen:exit_code: Union[int, Iterable[int],Callable[[int],bool]] = 0Specifies exit codes denoting successful execution of the command, this can
either bea specific integer exit code, by default0, ora sequence of integer exit codes, e.g. a range, ora function that returns True for successful exit codes.needs_shell: bool = FalseSet this toTrueif the command requires a shell environment.encoding: str ='utf-8'Set this to another character encoding if the command’s output does not
use an UTF-8 compatible character set.exclusive: bool = FalseSome command are known to be prone to concurrency issues. Setting this toTruewill block concurrent invocations of the command, e.g. when using
Matrix Runner concurrently on the same machine.timeout: Optional[float] = NoneCommands prone to hick-ups (i.e. getting stuck) can be automatically killed
when a specified time span has elapsed [in seconds].retry: Optional[int] = 1Commands prone to occasional failures can be automatically repeated.
The result will reflect the latest return code. The output from all
tries is captured.rest_period: Optional[float] = NoneSome commands are known to be prone to concurrency issues. Setting this to
a value larger than0adds a rest period (i.e.time.sleep) before the
command actually gets executed.test_report: Optional[ReportFilter] = NoneBy default the result of a command only depends on theexit_code. The
commands output is captured but not processed by Matrix Runner. By
specifying a “recipe” how to evaluate results the output can be post
processed into a detailedReport.The actual command function can take arbitrary arguments which can be provided
during the enclosingAction. All arguments bound to a Command instance can
be inspected by reading the according object attribute, e.g.:cmd = dump_config(*args, **kwargs)  # Retrieve the Command object with bound arguments
print(cmd.args, cmd.kwargs)         # Inspect the Command object's bound argumentsCommand ResultFor each single invocation of aCommandthe entire output is captured in aCommandResultobject. These objects can be used to inspect the details
of the execution using the following properties:command: CommandA back-reference to the executedCommandwith bound arguments.cmdline: ListThe actual command line returned from the command function.exit_status: Optional[Union[int, TimeoutError]]The exit status returned by the command, or a TimeoutError exception if
occurred. ANoneindicates the command has run to completion, yet.success: boolIndicator for an overall successful execution. This value is calculated
from the actualexit_statusand the expectedCommand’sexit_code.output: StringIOThe capturedstdoutandstderrfrom the execution.test_report: Optional[ReportFilter.Result]The generatedReport Resultfrom applying theCommand’stest_report.start_time: floatThe Unix Epoch the command was launched retrieved fromtime.time().end_time: floatThe Unix Epoch the command was finishedtime.time().start_perf_counter: floatThe value retrieved fromtime.perf_counter()before running the command.end_perf_counter: floatThe value retrieved fromtime.perf_counter()after running the command.ReportA report can capture and post process the output of a command after execution.
This can be used to gather a test report from the commands standard output
into a standardized format. Though, this feature is not limit to test reports.In order to attach report generation to aCommandassign a pipe chain of
report generators totest_report. The available report generators are:ConsoleReport()Captures the console output of the command.CropReport(first: AnyStr, last: AnyStr)Crops a section out of the input. All lines between first and last
(inclusive) are kept, everything else is dropped.TransformReport(xslt)Applies the provided XSLT script to the input.JUnitReport(title)Treats the input as JUnit XML and provides a detailed summary. Settitleto a static string or a functionlambda(title: str <, result:CommandResult]>)->strto update the test suite names.Report generators can be chained in pipe like manner:ConsoleReport() |
CropReport("<report>", "</report>") |
TransformReport("tojunit.xsl") |
JUnitReport(title = lambda title, report: title)This chainCaptures the console output of the command.Crops all content between<report>and</report>, inclusive.Transforms the custom XML to JUnit.Considers the JUnit results for summary.TheReport Resultis appended to theCommand’s resulttest_reportproperty to be accessible, e.g. inActionfunctions.The report can
be written to a file using itswrite(file: AnyStr)function.Report ResultTheReportFilter.Resultgives access to the final output of aReportfilter chain applied to a specificCommandresult. The following properties
and functions can be used to access the report:stream->StringIODirect access to the text stream.getvalue()->strFinal string output.write(file: AnyStr)Write the output into a file.The mixinReportFilter.Summaryadds a detailed summary output in addition
to the defaultpassedorfailedone:summary->Tuple[int, int]returns a tuple ofpassedandexecutedtest cases.Custom ReportsCustom report generators can be added by subclassingReportFilteroverwriting its inner classReportFilter.Resultand giving a customstream->StringIOproperty. The preceding generator can be accessed
through_otherproperty. In case of an error while processing the report a
RuntimeError should be raised:class CustomReport(ReportFilter):
    class Result(ReportFilter.Result):
        @property
        def stream(self) -> StringIO:
            if not self._stream:
                try:
                    self._stream = StringIO()
                    input = self._other.stream
                    args = self._report.args
                    output = input              # add some modification
                    self._stream.write(output)
                except SomeException as e:
                    self._stream = e
            if isinstance(self._stream, Exception):
                raise RuntimeError from self._stream
            else:
                return self._stream

    def __init__(self, *args):
        super(CustomReport, self).__init__()
        self.args = argsAdditional report formats can be supported by mixing inReportFilter.Summaryand providing a customsummary->Tuple[int, int]property. The summary
shall return a tuple with numbers test cases(passed, executed):class CustomSummary(ReportFilter):
    class Result(ReportFilter.Result, ReportFilter.Summary):
        @property
        def summary(self) -> Tuple[int, int]:
            passed = ...   # calculate passed test cases
            executed = ... # calculate executed test cases
            return passed, executedFilterA filter can be used to remove inappropriateConfig’s from the configuration
matrix. This is achieved by defining a function annotated with@matrix_filterreturningTruefor configuration to be dropped:@matrix_filter
def filter(config):
    return config.alpha.match('value3')ConfigA single matrix configuration with specific values for each matrix axis is
denoted by anConfigobject. TheConfigobject contains a property
for each axis containing the value.In order to check if an axis value matches a condition one can use thematchfunction providing afnmatch.fnmatchpattern.Command LineRunning one or more configurations from the command line using Matrix Runner is
trivial. The generated interface looks like this:$ ./build.py --help
usage: build.py [-h] [--silent | --verbose | --debug] [--pairwise] [--slice <HERE>/<TOTAL>]
                [[--<axis> <AXIS>] ...] action [action ...]The positional argumentactioncan be one or multiple defineAction’s to
be executed in the given order, e.g. eitherbuildandrunseparately or
both in a sequence.The optional arguments are a combination of static ones used to parametrize
Matrix Runner itself and dynamic ones generated from the definedAxis:-h,--helpshows the help message--silentSilent mode, only errors are shown.--verboseVerbose log output.--debugDebug log output.-2,--pairwisereduces number of combinations using pairwise
algorithm. This enables combinatorialall-pairs testingto reduce the
overall number of configuration in logarithmic manner while retaining a high
probability of detecting issues.--slice<HERE>/<TOTAL>cuts the set of combinations intoTOTALnumber
of slices and executes only theHERE’th one. This can be used to run the
overall set of combinations in parallel. Slicing is applied after--pairwisereduction.-<a><AXIS>,--<axis><AXIS>reduce number of combinations to
selectedAXISvalues foraxisAxis values can be given as one of
their string representationsoranfnmatch.fnmatchpattern matching at
least one of these. In case of pattern matchingallmatching values are
selected. This argument can be given multiple time which adds the values in a
cumulative way.The console output has two parts. While executing the actions the output from
the associated commands is forwarded like this:[<AXIS>](<ACTION>) <pre processing log>
[<AXIS>](<ACTION>:<COMMAND>) <command line>
[<AXIS>](<ACTION>:<COMMAND>) <stdout/stderr from command>
[<AXIS>](<ACTION>) <post processing log>Each line of output is prefixed with a[<AXIS>]per axis giving the
matrix configuration values the output belongs to. Followed by a(<ACTION>:<COMMAND>)tag denoting the action and command currently
executed. On a color terminal there is a clear distinction between stdout
(green) and stderr (red) output. Intermediate warning messages from Python
are colored as yellow.After all actions have been executed for all selected configurations a
comprehensive summary is displayed like this:Matrix Summary
==============

<axis>   <ACTION>
-------  -------
<AXIS>   success
<AXIS>   FAILED
<AXIS>   (skip)
...      ...The Matrix Summary gives a table with a column per axis and action. Each line
denotes all actions result for a specific configuration:success(green) denotes all commands have been executed successfully (i.e. with expected exit codes)FAILED(red) denotes a command has returned a failure exit code and execution got stalled(skip)(yellow) denotes that this action has not been performed for that configuration.Skipping configuration happens due toFilter’s.NoteBy default Matrix Runner scripts are expected to be run from their
local directory. Calling a script from anywhere else will create a
warning and the execution is likely to fail.UtilitiesInspectorThe command line utilitymatrix-runner-inspectcan be used to inspect a
Matrix Runner build:$ matrix-runner-inspect -h
usage: matrix-runner-inspect [-h] scriptIt takes only a single positional argument denoting the script to be inspected,
for example:$ matrix-runner-inspect -- example.py
{
  "axes": {
    "alpha": {"abbrev": "a", "values": ["value1", "value2", "value3"],
              "desc": "A configuration axis"}
  },
  "matrix": [{"alpha": "value1"}, {"alpha": "value2"}, {"alpha": "value3"}]
}The output gives the axes definition and the actual configuration matrix. The
script take all the usual axis arguments as shown above. These can be used to
reduce the matrix reported by inspect:$ matrix-runner-inspect -- demo/example.py -a v[23]
{
  "axes": {"alpha": {"abbrev": "a", "values": ["value1", "value2", "value3"],
           "desc": "A configuration axis"}},
  "matrix": [{"alpha": "value2"}, {"alpha": "value3"}]
}