Package: pexen
Python EXecution ENvironmentThis is a (fairly) simple framework for building anything that requires some
kind of ordered execution, such as a build system, a test suite, or similar.How to use itSee further topics (in this order):SchedulerTask metadataFactoryCurrent stateThe project currently sits idle until I find a use for it. It is not under
very active development, but it's also not abandonware.Feel free to fork it and make enhancements if you wish.No APIs, names or anything is currently declared as stable, if you use this
project in your projects, expect having to rename/rework things some time
in the future.BackgroundThe idea came from limitations of various (non-Python) test suites that used
static (ini) files for configuration, limiting options for dynamic execution,
not having parallel execution support (or a crappy make-job-like one), etc.Thus the goal was to create a system which would supportTree-based structure of tests, no single (flat) or double (main/sub) levelsDependency tracking and resolution between testsParallel execution of tests with resolved dependenciesDynamic runtime configuration (selecting tests based on OS, etc.)Multiple results per one test (ie. 'pass', but with problems detected)Persistent state keeping (for re-runs of tests, for OS reboot)Some watchdog for keeping test runtime in checketc.and the idea was to utilize python as a "configuration" language.Generalizing this, the above formed intoA "factory-time" logic, which would traverse a tree structure and generate
the tests (a.k.a. execution units, python callables) based on some inputAn "execution-time" logic, which would use a scheduler to efficiently run
the collected testsFurther generalizing, it ultimately turned into an universal execution engine
info which you can feed any python callables and, given the right metadata,
they get scheduled and run in the right order, with the "factory-time" being
essentially just an example way of how to generate execution units.LimitationsNeeds python 3.6+, mostly due to quality-of-life features. Functionally,
it could be ported to python 2.7 as it currently uses only basicthreadingandmultiprocessingfeatures (eg. nofutures), at the expense of code
readability.Parallel result pipeliningWishA thread-safe process-safe interface to get results from running tasksThe user can then spawn a separate custom pool of threads/processes that
query this interface and post-process results returned from tasks, in
parallelWhile other tasks are still running, not after everything finishesWhy NotThere are process-unsafe metadata returned to the user, namely the
callable objects that were executed, exception objects, etc.These are safe to pass only to the thread/process that added the taskThere is a simple workaround: simply take care of the metadata in the
thread that added the tasks and pass on picklable data (ie. retvals
from the executed callables) to the custom process pool for further
post-processingWishlistScheduling groups of exclusivityPass groups of callables instead of just callables to the schedulerHave only one group running at a timeAs it finishes, destroy the worker pool, create a new one for the
new groupAllows to specify nr. of workers/spare and pool type (thread/mp) used
on a per group basis!Useful when a set of tasks is very different from the rest, ie. testing
virtual machines (5 workers) versus testing syscalls (100 workers), and one
global setting cannot cover both.Also useful to logically split mutually exclusive things that need
uninterrupted / unconditional access to the OS without defining "claims"
to lots of tests.Also useful for tasks that should logically run "together", possibly
needing sequential execution due to formal requirements.Adding new tasks on-the-flyWhile processing results from theSched.run()iterator, add new tasksUseful when the overall task set depends on result(s) from some taskspexen.sched.poolsupport already in placeBut properSchedsupport would need heavy rewrites, removal of sanity
checks, etc.I did the rewrite; took 6-8 hours and the code was incomprehensibleRefactoring of the wholepexen.schedwould be neededA post-processor for factoriesA task that requires some generic resource, ie.httpdThe post-processor would find (somehow) another, best suitable, task that
provides it, adding it to the tasks for executionSimilar to how Debian packages can depend on a virtual package likemail-transport-agentand have multiple packages (exim4,postfix, etc.)
provide it"UseCases" that would glue Factory and SchedulerIe. "test suite UseCase" for running a test suite, collecting and formatting
results, storing xUnit results in a file, supporting resume from
an interrupted execution, etc.