Package: eventor
Table of ContentsOverviewProgram Run FileEventor InterfaceRecoveryDelayed AssociationsResourcesDistributed StepsTODOsChange logAdditional InformationOverviewEventorprovides programmer with interface to create events, steps and associations of these artifacts with to create a flow.It would be easier to show an example.Simple Example1importeventorasevr2importlogging3fromacrilibimportLoggerAddHostFilter45appname=os.path.basename(__file__)6logger=logging.getLogger(appname)78defconstruct_and_run():910defprog(progname):11logger.info("doing what%sis doing"%progname)12returnprogname1314ev=evr.Eventor(name=appname)1516ev1s=ev.add_event('run_step1')17ev2s=ev.add_event('run_step2')18ev3s=ev.add_event('run_step3')1920s1=ev.add_step('s1',func=prog,kwargs={'progname':'prog1'},21triggers={evr.StepStatus.success:(ev2s,),})22s2=ev.add_step('s2',func=prog,kwargs={'progname':'prog2'},23triggers={evr.StepStatus.success:(ev3s,),})24s3=ev.add_step('s3',func=prog,kwargs={'progname':'prog3'},)2526ev.add_assoc(ev1s,s1)27ev.add_assoc(ev2s,s2)28ev.add_assoc(ev3s,s3)2930ev.trigger_event(ev1s,1)31ev.run()32ev.close()333435if__name__=='__main__':36construct_and_run()Example OutputThe above example with provide the following log output.[ 2016-11-30 10:07:48,612 ][ INFO ][ Running step s1[1] ][ main.task_wrapper ]
[ 2016-11-30 10:07:48,612 ][ INFO ][ Step completed s1[1], status: success, result 'prog1' ][ main.task_wrapper ]
[ 2016-11-30 10:07:50,649 ][ INFO ][ Running step s2[1] ][ main.task_wrapper ]
[ 2016-11-30 10:07:50,649 ][ INFO ][ Step completed s2[1], status: success, result 'prog2' ][ main.task_wrapper ]
[ 2016-11-30 10:07:52,688 ][ INFO ][ Running step s3[1] ][ main.task_wrapper ]
[ 2016-11-30 10:07:52,689 ][ INFO ][ Step completed s3[1], status: success, result 'prog3' ][ main.task_wrapper ]
[ 2016-11-30 10:07:53,700 ][ INFO ][ Processing finished with: success ][ main.loop_session_start ]Note: actual logging includeshostnameand andprocname, e.g., “[ mbp02 ][ MainProcess ]”. These information was omitted from logging herein.Example HighlightsEventor(line 10) defines eventor using default database configuration (see bellow).add_event(e.g., line 12) adds an event namedrun_step1to the respective eventor object.add_step(e.g., line 16) adds steps1which when triggered would run predefined functionprogwith key words parametersprogname=’prog1’.
Additionally, when step would end, if successful, it would trigger eventevs2add_assoc(e.g., line 22) links eventevs1and steps1.trigger_event(line 26) marks eventevs1; when triggers, event is associated with sequence. This would allow multiple invocation.ev()(line 27) invoke Eventor process that would looks for triggers and tasks to act upon. It ends when there is nothing to do.loggerandEventornames (lines 6 and 14), Eventor is using hierarchical logger based on Eventor name argument. As such naming convention needs to be aligned among all the files participating with the run.Program Run FileOne important artifact used in Eventor is program’s runner file.  Runner file database (sqlite) will be created at execution, if not directed otherwise, at the location of the run (UNIX’s pwd).
This file contains information on tasks and triggers that are used in the run and in recovery.Eventor InterfaceEventor Class InitiatorEventor(name='',store='',run_mode=RUM_RESTART,recovery_run=None,run_id='',config={})Argsname: string id for Eventor object initiated.store: Eventor mechanism is built to work with SQLAlchemy. If store is provided, Eventor first check if store is a tag within config underEVENTOR.DATABASE(or whatever the environment variablesEVENTOR_CONFIG_TAGandEVENTOR_DB_CONFIG_TAGpoints to) section. If the tag exists, it will pick its configuration as database configuration. If store is empty, Eventor will try to look fordefaultdatabase configuration. Otherwise,storewill be considered as a path to file that would store runnable (sqlite) information; If not provided, calling module path and name will be used with ‘.db’ extension instead of ‘.py’.run_mode: can be eitherRUN_RESTART(default) orRUN_RECOVER; in restart, new instance or the run will be created. In recovery, ifshared_dbis set, run_id or the recovered program must be provided.recovery_run: ifRUN_RECOVERis used,recovery_runwill indicate specific instance of previously recovery run that would be executed.If not provided, latest run would be used.run_id: unique ID for the program run (excluding recovery_run).  It is mandatory inshared_dbmode, and if not provided, will be generated.config: keyword dictionary of default configurations.  Available keywords and their default values:NameDefault
ValueDescriptionworkdir/tmpplace to create necessary artifacts (not in use)logdir/tmpplace to create debug and error log filestask_constructmp.Processmethod to use for execution of stepsmax_concurrent1maximum concurrent processing, if value <1, no
limit will be posestop_on_exceptionTrueif an exception occurs in a step, stop
all processes.  If True, new processes will not
start.  But running processes will be permitted
to finishsleep_between_loops1seconds to sleep between iteration of checking
triggers and tasksshared_dbFalseif set, db must not be in memory. signals that
multiple programs will use the same database
tables.envvar_prefixEVENTORset prefix for naming environment variabledefined for each step:{envvar_prefix}STEP_NAME{envvar_prefix}STEP_SEQUENCE{envvar_prefix}STEP_RECOVERY{envvar_prefix}LOGGER_NAMEssh_config~/.ssh/configSSH configuration file to use with SSH remoteInvocation of steps.ssh_hostSSH host configuration name prime host.ssh_portSSH port to use for SSH connectivityLOGGINGdictionary of logging configurations.DATABASESdictionary of database configurations.Configuration file exampleEVENTOR:
   debug: False
   task_construct: process
   envvar_prefix: EVENTOR_
   max_concurrent: -1
   stop_on_exception: True
   sleep_between_loops: 0.25
   sequence_arg_name: None
   day_to_keep_db: 5
   remote_method: ssh
   pass_logger_to_task: False
   shared_db: False

    DATABASES:

        sqfile1:
            dialect: sqlite
            database: /tmp/runly.db

        pgdb1:
            dialect:  postgresql
            drivername :  psycopg2
            username: pgusername
            password: pgpassword
            host:     ubuntu-guest-02
            port:     5433
            database: pyground
            schema: play

    LOGGING:
        logging_level: 10
        logdir: /var/log/eventor
        level_formats:
            10: ('[ %(asctime)-15s ][ %(host)s ][ %(processName)-11s ][ %(levelname)-7s ]'
                 '[ %(message)s ][ %(module)s.%(funcName)s(%(lineno)d) ]')
            default: ('[ %(asctime)-15s ][ %(host)s ][ %(processName)-11s ]'
                      '[ %(levelname)-7s ][ %(message)s ]')
        consolidate: False
        console: True
        file_prefix: ''
        file_suffix: ''
        file_mode: 'a'
        maxBytes: 0
        backupCount: 0
        encoding: 'utf8'
        delay: False
        when: 'h'
        interval: 1
        utc: False
        atTime: 86400Database NotesIt is possible to create configuration for sqlite memory with cache=shared. However, SQLAlchemy will not work with it well in threaded and multiprocessing environment.  Hence, at this point,Eventordoes not support it.Eventoradd_eventmethodadd_event(name,expr=None)Argsname: string unique id for eventexpr: logical expression ‘sqlalchemy’ style to automatically raise this expression.syntax:expr : (expr, expr, ...)
     | or_(expr, expr, ...)
     | eventif expression is of the first style, logicalandwill apply.the second expression will apply logicalor.the basic atom in expression isevenwhich is the product ofadd_event.ReturnsEvent object to use in otheradd_eventexpressions,add_assocmethods, or withadd_steptriggers.Eventoradd_stepmethodadd_step(name,func,args=(),kwargs={},triggers={},acquires=[],releases=None,recovery={},config={})Argsname: string unique id for stepfunc: callable object that would be call at time if step executionargs: tuple of values that will be passed tofuncat callingkwargs: keywords arguments that will be passed tofuncat callingtriggers: mapping of step statuses to set of events to be triggered as in the following table:statusdescriptionSTEP_READYset when task is ready to run (triggered)STEP_ACTIVEset when task is runningSTEP_SUCCESSset when task is successfulSTEP_FAILUREset when task failsSTEP_COMPLETEstands for success or failure of taskacquires: list of tuples of resource pool and amount of resources to acquire before starting.releases: list of tuples of resources pool and amount of resources to release once completed. If None, defaults toacquires.  If set to empty list, none of the acquired resources would be released.recovery: mapping of state status to how step should be handled in recovery:statusdefaultdescriptionSTEP_READYStepReplay.rerunif in recovery and previous status is ready, rerunSTEP_ACTIVEStepReplay.rerunif in recovery and previous status is active, rerunSTEP_FAILUREStepReplay.rerunif in recovery and previous status is failure, rerunSTEP_SUCCESSStepReplay.skipif in recovery and previous status is success, skipconfig: keywords mapping overrides for step configuration.namedefaultdescriptionstop_on_exceptionTruestop flow if step ends with ExceptionReturnsStep object to use in add_assoc method.Eventoradd_assocmethodadd_assoc(event,*assocs,delay=0)Argsevent: event objects as provided by add_event.assocs: list of associations objects.  List is composed from either events (as returned by add_event) or steps (as returned by add_step)delay: seconds to wait, once event is triggered, before engaging its associationsReturnsN/AEventortrigger_eventmethodtrigger_event(event,sequence=None)Argsevent: event objects as provided by add_event.sequence: unique association of triggered event.  Event can be triggered only once per sequence.  All derivative triggers will carry the same sequence.ReturnsN/AEventorrunmethodrun(max_loops=-1)when callingrun, information is built and loops evaluating events and task starts are executed.
In each loop events are raised and tasks are performed.  max_loops parameters allows control of how many
loops to execute.In simple example,ev.run()engage Eventor’srun()method.Argsmax_loops: max_loops: number of loops to run.  If positive, limits number of loops.defaults to negative, which would run loops until there are no events to raise and
no task to run.ReturnsIf there was a failure that was not followed by event triggered, result will be False.Eventorclosemethodclose()when callingclose, Eventor object will close its open artifacts.  This is similar to close method on multiprocessing Pool.In simple example,ev.close()engage Eventor’sclose()method.ArgsN/A.ReturnsN/A.RecoveryWhen running in recovery, unless indicated otherwise, latest run (initial or recovery) would be used.Note that when running a program with the intent to use its recovery capabilities, in-memory storecannotbe use.
Instead, physical storage must be used.Here is an example for recovery program and run.Recovery Example1importeventorasevr2importlogging3importmath4fromacrilibimportLoggerAddHostFilter56appname=os.path.basename(__file__)7logger=logging.getLogger(appname)89defsquare(x):10y=x*x11logger.info("Square of%sis%s"%(x,y))12returny131415defsquare_root(x):16y=math.sqrt(x)17logger.info("Square root of%sis%s"%(x,y))18returny192021defdivide(x,y):22z=x/y23logger.info("dividing%sby%sis%s"%(x,y,z))24returnz2526defbuild_flow(run_mode=evr.RUN_RESTART,param=9,run_id=None):27ev=evr.Eventor(name=appname,run_mode=run_mode,run_id=run_id,28config={'LOGGING':29{'logging_level':logging.INFO}},))3031ev1s=ev.add_event('run_step1')32ev1d=ev.add_event('done_step1')33ev2s=ev.add_event('run_step2')34ev2d=ev.add_event('done_step2')35ev3s=ev.add_event('run_step3',expr=(ev1d,ev2d))3637s1=ev.add_step('s1',func=square,kwargs={'x':3},38triggers={evr.STEP_SUCCESS:(ev1d,ev2s,)},)39s2=ev.add_step('s2',square_root,kwargs={'x':param},40triggers={evr.STEP_SUCCESS:(ev2d,),},41recovery={evr.STEP_FAILURE:evr.STEP_RERUN,42evr.STEP_SUCCESS:evr.STEP_SKIP})43s3=ev.add_step('s3',divide,kwargs={'x':9,'y':3},)4445ev.add_assoc(ev1s,s1)46ev.add_assoc(ev2s,s2)47ev.add_assoc(ev3s,s3)48ev.trigger_event(ev1s,3)49returnev505152defconstruct_and_run():53# start regularly; it would fail in step 254ev=build_eventor(param=-9)55run_id=ev.run_id56ev.run()57ev.close()5859# rerun in recovery60ev=build_eventor(evr.RUN_RECOVER,param=9,run_id=run_id)61ev.run()62ev.close()636465if__name__=='__main__':66construct_and_run()Example Output1[ 2016-12-07 08:37:53,541 ][ INFO ][ Eventor store file: /eventor/example/runly03.run.db ]2[ 2016-12-07 08:37:53,586 ][ INFO ][ [ Step s1/3 ] Trying to run ]3[ 2016-12-07 08:37:53,588 ][ INFO ][ Square of 3 is 9 ]4[ 2016-12-07 08:37:53,588 ][ INFO ][ [ Step s1/3 ] Completed, status: TaskStatus.success ]5[ 2016-12-07 08:37:55,644 ][ INFO ][ [ Step s2/3 ] Trying to run ]6[ 2016-12-07 08:37:55,647 ][ INFO ][ [ Step s2/3 ] Completed, status: TaskStatus.failure ]7[ 2016-12-07 08:37:56,663 ][ ERROR ][ Exception in run_action:8<Task(id='2', step_id='s2', sequence='3', recovery='0', pid='8112', status='TaskStatus.failure', created='2016-12-07 14:37:55.625870', updated='2016-12-07 14:37:55.633819')> ]9[ 2016-12-07 08:37:56,663 ][ ERROR ][ ValueError('math domain error',) ]10[ 2016-12-07 08:37:56,663 ][ ERROR ][ File "/sand/eventor/eventor/main.py", line 62, in task_wrapper11result=step(seq_path=task.sequence)12File "/sand/eventor/eventor/step.py", line 82, in __call__13result=func(*func_args, **func_kwargs)14File "/eventor/example/runly03.py", line 66, in square_root15y=math.sqrt(x) ]16[ 2016-12-07 08:37:56,663 ][ INFO ][ Stopping running processes ]17[ 2016-12-07 08:37:56,667 ][ INFO ][ Processing finished with: failure ]18[ 2016-12-07 08:37:56,670 ][ INFO ][ Eventor store file: /eventor/example/runly03.run.db ]19[ 2016-12-07 08:37:57,736 ][ INFO ][ [ Step s2/3 ] Trying to run ]20[ 2016-12-07 08:37:57,739 ][ INFO ][ Square root of 9 is 3.0 ]21[ 2016-12-07 08:37:57,739 ][ INFO ][ [ Step s2/3 ] Completed, status: TaskStatus.success ]22[ 2016-12-07 08:38:00,798 ][ INFO ][ [ Step s3/3 ] Trying to run ]23[ 2016-12-07 08:38:00,800 ][ INFO ][ dividing 9 by 3 is 3.0 ]24[ 2016-12-07 08:38:00,800 ][ INFO ][ [ Step s3/3 ] Completed, status: TaskStatus.success ]25[ 2016-12-07 08:38:01,824 ][ INFO ][ Processing finished with: success ]Example HighlightsThe functionbuild_flow(code line 24) build an Eventor flow using three functions defined in advance.
Since no specific store is provided in Eventor instantiation, a default runner store is assigned (code line 25).
In this build, steps2(lines 30-35) is being set with recovery directives.The first build and run is done in lines 47-48.  In this run, a parameter that would cause the second
step to fail is being passed.  As a result, flow fails.  Output lines 1-17 is associated with the first run.The second build and run is then initiated.  In this run, parameter is set to a value that would pass
steps2and run mode is set to recovery (code lines 51-52). Eventor skips successful steps and start
executing from failed steps onwards.  Output lines 18-25 reflects successful second run.Delayed AssociationsThere are situations in which it is desire to hold off activating a task.  This behavior is captured in Eventor as a delayed association.Associations can be made delayed.  Assuming source event is associated to target event with time delay.  When source event is triggered, Eventor will wait time delay seconds before triggering target event.In such situations, it sometimes desire to run Eventor engine in specific period on a time line instead of continuously.  For example, if Eventor is synchronizing activities that has 6 hours association delay.  Instead of running Eventor continuously, it can be set to run every 5 minutes, and save computing resources on the side.Withdelayed associations, Eventor can run incontinuerun mode (RunMode.continue_).  When running incontinue, Eventor will pick up from where it left last run.The following example presentdelayed associationwithcontinuerun mode.Delay Example1import eventor as evr2import logging3import os4import time56appname = os.path.basename(__file__)7logger = logging.getLogger(appname)89def prog(progname):10logger.info("doing what %s is doing" % progname)11logger.info("EVENTOR_STEP_SEQUENCE: %s" % os.getenv("EVENTOR_STEP_SEQUENCE"))12return progname131415def build_flow(run_mode):16ev = evr.Eventor(name=appname, run_mode=run_mode,)1718ev1s = ev.add_event('run_step1')19ev2s = ev.add_event('run_step2')20ev3s = ev.add_event('run_step3')2122s1 = ev.add_step('s1', func=prog, kwargs={'progname': 'prog1'}, triggers={evr.STEP_SUCCESS: (ev2s,)})23s2 = ev.add_step('s2', func=prog, kwargs={'progname': 'prog2'}, triggers={evr.STEP_SUCCESS: (ev3s,)})24s3 = ev.add_step('s3', func=prog, kwargs={'progname': 'prog3'},)2526ev.add_assoc(ev1s, s1, delay=0)27ev.add_assoc(ev2s, s2, delay=10)28ev.add_assoc(ev3s, s3, delay=10)2930ev.trigger_event(ev1s, 1)31return ev323334def construct_and_run():35ev = build_flow(run_mode=evr.RUN_RESTART)36ev.run()37ev.close()3839if __name__ == '__main__':40construct_and_run()Example Output1[ 2017-08-16,16:31:29.277048 ][ Task-s1(1)  ][ INFO    ][ [ Step s1/1 ] Trying to run ]2[ 2017-08-16,16:31:29.277903 ][ Task-s1(1)  ][ INFO    ][ doing what prog1 is doing ]3[ 2017-08-16,16:31:29.278114 ][ Task-s1(1)  ][ INFO    ][ EVENTOR_STEP_SEQUENCE: 1 ]4[ 2017-08-16,16:31:29.278360 ][ Task-s1(1)  ][ INFO    ][ [ Step s1/1 ] Completed, status: TaskStatus.success ]5[ 2017-08-16,16:31:41.028196 ][ Task-s2(1)  ][ INFO    ][ [ Step s2/1 ] Trying to run ]6[ 2017-08-16,16:31:41.029191 ][ Task-s2(1)  ][ INFO    ][ doing what prog2 is doing ]7[ 2017-08-16,16:31:41.029429 ][ Task-s2(1)  ][ INFO    ][ EVENTOR_STEP_SEQUENCE: 1 ]8[ 2017-08-16,16:31:41.029697 ][ Task-s2(1)  ][ INFO    ][ [ Step s2/1 ] Completed, status: TaskStatus.success ]9[ 2017-08-16,16:32:02.931265 ][ Task-s3(1)  ][ INFO    ][ [ Step s3/1 ] Trying to run ]10[ 2017-08-16,16:32:02.932407 ][ Task-s3(1)  ][ INFO    ][ doing what prog3 is doing ]11[ 2017-08-16,16:32:02.932661 ][ Task-s3(1)  ][ INFO    ][ EVENTOR_STEP_SEQUENCE: 1 ]12[ 2017-08-16,16:32:02.932940 ][ Task-s3(1)  ][ INFO    ][ [ Step s3/1 ] Completed, status: TaskStatus.success ]13[ 2017-08-16,16:32:03.014584 ][ MainProcess ][ INFO    ][ Processing finished with: success; outstanding tasks: 0 ]Example HighlightsThe example program builds and runs Eventor sequence 4 times.  The build involves three tasks that would run sequentially.  They are associated to each other with delay of 10 seconds each (lines 26 and 28.)The first time, sequence is build withrestartrun mode (line 35).  In this case, the sequence is initiated.  The next four runs are incontinuerun mode (line 48).  Each of those run continue its preceding run.  To have it show the point, a varying delay is introduced between runs (lines 46-47).Each run limits the number of loop to a single loop (lines 40 and 50).  A single loop entails Eventor executing triggers and tasks until there is none to execute.  It may be though that there are still outstanding delayed association to act upon.This behavior is different than continuous run (using max_loops=-1), which is the default.  In such run, Eventor will continue to loop until there are no triggers, tasks, and delayed association to process.Eventor runs can be observed in example output lines 1-5, 6, 7-11, 12, and 13-17 each.  Note that the second and forth runs had not trigger to execute on.  The associated tasks’ delays was not yet matured.Resourcesadd_stepallows association of step with resources.  If acquires argument is provided, before step starts,Eventorwill attempt to reserve resources.  Step will be executed only when resources are secured.Whenreleaseargument is provided, resources resources listed as its value will be released when step is done.  If
release is None, whatever resources stated byacquireswould be released.  If the empty list is set as value, no
resource would be released.To use resources, program to use Resource and ResourcePool from acris.virtual_resource_pool.  Example for such definitions are below.Example for resources definitions1importeventorasevr2fromacrisimportvirtual_resource_poolasvrp34classResources1(vrp.Resource):pass5classResources2(vrp.Resource):pass67rp1=vrp.ResourcePool('RP1',resource_cls=Resources1,policy={'resource_limit':2,}).load()8rp2=vrp.ResourcePool('RP2',resource_cls=Resources2,policy={'resource_limit':2,}).load()910ev=evr.Eventor()1112s1=ev.add_step('s0.s00.s1',func=prog,kwargs={'progname':'prog1'},acquires=[(rp2,1),],)Distributed StepsEventor program can work in a clustered environment.  In this arrangement, steps can be defined to run on different nodes in the cluster.  This is possible granted:SSH is defined among cluster nodes.Eventor DB is shared among cluster nodes.Program environment is theseamlessly-the-sameamong cluster nodes.How it worksEventor will be launched from one host,server.  It will then start the same program on every associated host relevant to program,clients.Clientprograms will skipstartingsteps (steps with no )Cluster SSH accessWhen working on distributed environment, Eventor assumes that ssh is set properly among participating hosts.To allow ssh run command with .profile (or .bash_profile) are not automatically executed, add the following before RSA key in .ssh/authorizedkeyscommand"if [[\"x${SSH_ORIGINAL_COMMAND}x\"!=\"xx\"]]; then source ~/.profile; eval\"${SSH_ORIGINAL_COMMAND}\"; else /bin/bash --login; fi;"<key>DatabaseEventor program would be launched on all cluster nodes relevant to the program.TODOsThe following is some of the major tasks intended to be completed into this product.asynchronous tasks: embed mechanism to launch asynchronous tasks.remote callback mechanisms: allow remote asynchronous tasks communicate with Eventor (TCP/IP, HTTP, etc.)virtual resources shared across distributed environment.improve SSHPipe to better indicate SSH channel was established.Change log5.0added database configuration allowing the use of SqlAlchemy database engines.added shared_db to indicate db is shared among multiple programs and runs.added run_id as unique identifier for program run (not to be confused with recovery).improved documentation to reflect the need for mp.freeze_support() and mp.set_start_method(‘spawn’).added dependency on namedlist, and PyYAML, packages.bug fix in delay.5.1SSH remote invocation of steps.use of socket based logging.centralized logging of remote agents in prime server.Added {envvar_prefix}LOGGER_NAME to allow step logger to be set appropriately.Additional InformationEventor github project (Eventor github project) has additional examples with more complicated flows.SSH Pipeblog clarifying the mechanism using by Eventor for remote steps and centralized logging.