Package: impulsare-job
impulsare/jobOverviewA jobs manager, specific toimpulsare. It reads, write and delete jobs from an sqlite db.Seetests/static/for examples of configuration.Installation / UsageTo install use pip:$pipinstall--upgradeimpulsare-jobConfigurationYou need to create a configuration file that contains:job:db:/tmp/test.db# requiredArchitectureWriterimpulsare/job implements a writer to :Create / Update jobs (save())Delete jobs (delete())Add / Remove Hooks (hooks_writer.add_hook()andhooks_writer.del_hook())Add / Remove Fields (fields_writer.add_field()andfields_writer.del_field())Add / Remove Rules related to Fields (fields_writer.add_rule()andfields_writer.del_rule())ReaderAnd a Reader to :Get a JobGet related hooksGet related fields + their rulesProperties of a job{'name':str,# required'active':bool,# default : True'description':str,'priority':int,# default : 1'input':str,# required'input_parameters':dict,'output':str,# required'output_parameters':dict,'mode':str# c (create), u (update), cu (create/update), d (delete). Default: c}ExamplesCreate a simple Job (no hooks / rules)fromimpulsare_jobimportWriterwriter=Writer('/etc/impulsare/config.yml')writer.set_prop('name','My Job')writer.set_prop('input','csv')writer.set_prop('input_parameters',{'delimiter':'csv'})writer.set_prop('output','sql')writer.set_prop('output_parameters',{'db':'test'})job=writer.save()Update a Jobfromimpulsare_jobimportWriter# Lets assume the job id = 1writer=Writer('/etc/impulsare/config.yml','My Job')job=writer.get_job()print(job.name)# Output: 'My Job'# Set the job to Inactivewriter.set_prop('active',False)writer.save()Verify if a hook exists, else add it# .... continuation of code aboveifnotwriter.hooks_writer.hook_exists('test'):writer.hooks_writer.add_hook(name='upload_file',method='upload_file',when='after_process')Allowed properties for hooks:{'name':str,# required'method':str,# required'when':str,# required'description':str,'active':bool,# Default : True'priority':int# Default: 1}Other methods:hooks_writer.get_hookshooks_writer.del_hookThere is no methodupdate, to update a hook, delete it then recreate it.Verify if a field exists, else update it and add a transformation ruleWarning: a field is identified by itsoutputvalue that must be unique
(we can’t send two values for the same field while we can use the same input field
for various output).Field# .... continuation of code aboveifwriter.fields_writer.field_exists('firstname'):writer.fields_writer.del_field('firstname')writer.fields_writer.add_field(input='first_name',output='firstname')Allowed properties for fields:{'input':str,# required'output':str,# required}Other methods:fields_writer.get_fieldfields_writer.get_fieldsThere is no methodupdate, to update a field, delete it then recreate it.Add a rulewriter.fields_writer.add_rule(output_field='firstname',name='uppercase',method='uppercase')Allowed properties for rules:{'name':str,# required'method':str,# required'description':str,'active':bool,# Default : True'params':list,'blocking':bool,# Default : False'priority':int# Default: 1}Other methods:fields_writer.del_rulefields_writer.get_rulesfields_writer.rule_existsThere is no methodupdate, to update a rule, delete it then recreate it.Retrieve a Job, its hooks and fieldsfromimpulsare_jobimportReaderReader=Reader('/etc/impulsare/config.yml','My Job')job=Reader.get_job()hooks=Reader.get_hooks()fields=Reader.get_fields()# Get rules for first field : rules = fields[0].rulesDevelopment & Tests$pipinstall-rrequirements.txt$pipinstall-rrequirements-dev.txt$py.testTODODon’t check if table exists on each model but do it on app installationTo get the DB :fromimpulsare_jobimportmodelsdb=models.get_db('/etc/impulsare/config.yml')db.create_tables([models.Job,models.Hook,models.Rule])Refactor writerTo have a class for hooks, and another for rules.