Package: pypyrus-runner
runnerThe aim of this project is to create simple and effective instrument for automatic job scheduling and management in Python.Mainrunnerfeatures:Minimal and user friendly command line interface.Cross platform - application can be used in any OS with Python 3.Opportunity to handle all jobs in one place instead of using many (Oracle Enterprise Scheduler, Linux Crontab and etc.).Generate built-in scheduler.Generate built-in job.Edit generated built-in jobs.Delete generated built-in jobs.Configuration with ini files.Automatic job scheduling and execution at certain time, during certain period or each time cycle.Manual job execution for now or for certain date and time.Scheduling with simple tsv file.Run jobs using virtual Python environment.Run jobs implemented in non Python technologies like C++, Java, Bash, Perl, etc.Built-in logging.Built-in notifications and alarms.Mainrunnertargets:Add some CLI commands.Attach database torunneras option for schedule and new data streams storage.Add simple web GUI.Improve notifications' mechanism and how they looks.Optimizations.Getting StartedRequirementsOperation systems: Windows, Linux, Mac OS.Python version: 3.7.1.Python modules dependencies (use latest releases):logbook,tables,notifierInstallationTo install just download the latestrunnerrelease and copy /src/runner folder to your/python/folder/Lib/site-packages/.How to UseTo start working withrunneryou need just one simple file containingrunner.Managerinstance.Let's go to the root of your disk *C:* (or any other location, it does not matter) and create the folderrunner.
Move to that folder and create new file namedmanager.pywith next few string of code.> C:\runner\manager.py

import runner

manager = runner.Manager()Nowmanager.pyis your mainrunnerinterface and the way to deploy the application inC:\runnerfolder.Execute themanager.py. You will receive the main help note that can also be called with themanager.py help.When you executemanager.pyfirst time it generates main configuration fileconfig.iniin the same folder.$ python manager.py
INFO: File C:\runner\config.ini created.

List of available commands:

create scheduler    Generate a scheduler with all elements.
create job          Generate a job with all elements.
list jobs           Show all jobs registered in the schedule.
edit schedule       Open the schedule file in the editor.

edit job            Open the job script in the editor.
run job             Execute the job by id with or without run time.
run jobs            Execute the jobs listed in the file.
delete job          Delete the job by id.

edit config         Open one of the configuration files in the editor.

help                Show this message.

For more details type [command] help.You could get more specific help for each command:$ python manager.py run job help

Execute the job by id with or without run time.
Parameters:
id         integer                  Id of the job you want to run.
trigger    yyyy-mm-dd/hh24:mi:ss    Date with or without time in ISO format
           yyyy-mm-dd               for what you want to run the job.We recommend to repeat that action for each command for more understanding.Create SchedulerType commandcreate schedulerto generate the scheduler.
You will be asked about necessary attributes and then scheduler will be generated in the folder.$ python manager.py create scheduler

Please follow the steps to create the scheduler.
Enter the name or leave empty to use default:

Enter the description or leave empty to use default:

********************************************************************************
INFO: Creating scheduler scheduler...
INFO: File C:\runner\scheduler.py created.
INFO: File C:\runner\config.ini updated.
INFO: File C:\runner\schedule.tsv created.
INFO: Folder C:\runner\jobs created.About files:C:\runner\scheudler.py- scheduler instance and starter.C:\runner\config.ini- main configuration file.C:\runner\schedule.tsv- job schedule.C:\runner\jobs- folder for future jobs.Start SchedulerTo run the scheduler just execute filescheduler.py.$ python scheduler.pyNow the scheduler is working.
Logs can be found in the folderC:\runner\logs.To stop the scheduler process just type the CTRL + C.Create JobScheduler is ready and jobs can be created.Type commandcreate jobto generate the job.
You will be asked about necessary attributes and then job will be generated.$ python manager.py create job

Follow the instructions to create the job.
Inputs with * are mandatory.
Enter the name or leave empty to use default:
test
Enter the short description:
Test Job
Enter the environment or leave empty to use Python:

Enter the month day (1-31):

Enter the week day (1-7):

Enter the hour (0-23):
8
Enter the minute (0-59):
0
Enter the second (0-59):
0
Activate job (Y/N)? *:
N
********************************************************************************
INFO: Creating job...
INFO: Job ID: 0
INFO: Folder C:\runner\jobs\0 created.
INFO: File C:\runner\jobs\0\job.py created.
INFO: File C:\runner\jobs\0\config.ini created.
INFO: File C:\runner\jobs\0\script.py created.
INFO: Job test successfully added to schedule!About files:C:\runner\jobs\0- this is a job folder which corresponds to job id.C:\runner\jobs\0\job.py- job instance and starter.C:\runner\jobs\0\config.ini- job configuration file.C:\runner\jobs\0\script.py- script that must be executed by job.Inschedule.tsvnew record for created job will be provided.Note that we used statusN.
Jobs with such status will be not executed until status changed to Y.
It is done to prevent accidental job run before it is not finalized.SchedulingSchedule is one of the most important part of the application.All jobs scheduled inschedule.tsv.
The file is scanned byrunnerand its formatting is quite critical.Main formatting requirements are:Each line is a table row.Column separator is only a tabulation.Last line must be always empty.Whenschedule.tsvis modifiedrunnercatches the changes, makes necessary updates and also prints message to the log about that.To see all scheduled jobs use commandlist jobs:$ python manager.py list jobs

-----------------------------------------------------------------------------------------------------------------------------
|ID |NAME |DESCRIPTION |ENVIRONMENT |FILE                    |MONTH_DAY |WEEK_DAY |HOUR |MINUTE |SECOND |PARAMETERS |STATUS |
-----------------------------------------------------------------------------------------------------------------------------
|0  |test |Test Job    |python      |C:\runner\jobs\0\job.py |*         |*        |8    |0      |0      |           |N      |
-----------------------------------------------------------------------------------------------------------------------------To see only active jobs addactive.$ python manager.py list jobs activeAlso you could either open file directly or use commandmanager.py edit scheduleto open file using chosen editor in the main config.Schedule DescriptionField nameDescriptionIDUnique id that job receives automatically during creation.NAMEName of job.DESCRIPTIONDescription of job.ENVIRONMENTName of environment in which job must be executed. All used inrunnerenvironments must be listed in the main config.FILEEntry file. If you use built-in jobs then it is ajob.py. In other case it is up to you how job should be started.MONTH_DAYDay of month when job must be executed.WEEK_DAYDay of week when job must be executed.HOURHour of day when job must be executed.MINUTEMinute of our when job must be executed.SECONDSecond of minute when job must be executed.PARAMETERSAdditional arguments that will be passed to FILE in the time of execution.STATUSStatus of job. Y is an active, N is an inactive.Job frequency inrunneris rather flexible.
Be attentive because some combinations can forcerunnerto execute job every second!You could schedule jobs by manipulating with time fields next few ways:Execute job at certain date and time. Use simple integers.Execute job every time cycle. Use integers with symbol/to point that it is a cycle value.Execute job only during certain period. Use integers with,and-to point that it is a period value.For more understanding we list the most popular combinations:MONTH_DAYWEEK_DAYHOURMINUTESECONDWhen to Execute1*800Each first day of month at 8 am.*1800Each first day of week at 8 am.**800Each day at 8 am.**/100Each hour.***/50Each five minutes.*1-5800Each day during weekdays at 8 am.1,11,21*800Each 1, 11 and 21 day of month at 8 am.More About SchedulingAutomatic scheduling inrunneris based on moments.
Moment - is a dynamic scheduler attribute that describes current timestamp as a count of seconds past from era begin.You can log each scheduler moment if you need to.
For this set theshowtimeoption in theLOGsection of the main config toTrue.
Then in log each moment will be recorded as an empty message:2018-12-31 23:59:59|INFO|Moment includes an active and a passive phases.Active phase stands for all necessary actions for scheduling and internalrunnerneeds.
After active phase finished passive phase begins during whichrunnersleeps till the next step.If active phase longs more than 1 second in case of some trouble then scheduler internal time will differ from real one.
When this happenedrunnerwill automatically synchronize moment with real time.
That will be registered in the log that may help in troubleshooting:2018-12-31 23:59:59|WARNING|TIME IS BROKEN!
2018-12-31 23:59:59|INFO|SYNCHRONIZING THE TIME...
2018-12-31 23:59:59|INFO|SUCCESSIf you want to log each active phase time consumption then set theshowdelayoption of theLOGsection in the main config toTrue:TaskingA task is a set of actions that a job must perform.
All tasks must be described in a special job script file.
For our test job this is the filejobs\0\script.py.
That file already containsjobinstance so it can be used in the script too.Open script file or use special commandedit jobthat will open it using chosen editor in main config.$ python manager.py edit job 0Modify file as in the example below:> C:\runner\jobs\0\job.py

from job import job

# Write the code to be executed by job down below.
print('Wake up, Neo...')Now go toschedule.tsvand changeSTATUSfor modified job fromNtoY.Job is ready and at next 08:00:00 it will be executed.Run JobJob can be executed automatically by scheduler or manually by user.For automatic execution inschedule.tsvmust be determined:ENVIRONMENTthat is a name of environment described in the main config.MONTH_DAY,WEEK_DAY,HOUR,MINUTE,SECOND- time of execution.User execution can be done by two ways.The first one is to use special commandrun jobwith mandatory argumentsidand optional argumenttrigger.Argumentidis an integer unique identification number of job that can be found inschedule.tsv.Argumenttriggeris a date or datetime in ISO timestamp format for which job must be executed.In the example below we execute the job with id 0 for 1th January 2019, 8 AM.$ python manager.py run job 0 2019-01-01/08:00:00
INFO: REQUEST TO RUN THE JOB.
INFO: ID: 0.
INFO: NAME: test.
INFO: DESCRIPTION: Test Job.
INFO: TRIGGER: 2019-01-01/08:00:00.
ARE YOU SURE Y/N?
Y
Wake up, Neo...
INFO: DONE.Second one is to executejob.pyfile of the job you want to run.
Following command will execute the job with id 0 for current moment:$ python jobs\0\job.py
Wake up, Neo...Optionally you could use argument -t/--trigger to pass certain date and time.
So execution of job with id 0 for 1th January 2019, 8 AM will looks like:$ python jobs\0\job.py -r 2019-01-01/08:00:00
Wake up, Neo...Also if you have many jobs that must be executed it is possible to use special file.
Let's create fileC:\runner\run_jobs.txtwith listed ids and triggers:0 2018-12-31/08:00:00
0 2019-01-01/08:00:00Note that between id and trigger there is a space.Execute the command:$ python manager.py run jobs C:\runner\run_jobs.txt
INFO: REQUEST TO RUN THE JOB.
INFO: ID: 0.
INFO: NAME: test.
INFO: DESCRIPTION: Test Job.
INFO: TRIGGER: 2018-12-31/08:00:00.
ARE YOU SURE Y/N?
Y
Wake up, Neo...
INFO: DONE.
********************************************************************************
INFO: REQUEST TO RUN THE JOB.
INFO: ID: 0.
INFO: NAME: test.
INFO: DESCRIPTION: Test Job.
INFO: TRIGGER: 2019-01-01/08:00:00.
ARE YOU SURE Y/N?
Y
Wake up, Neo...
INFO: DONE.LoggingLogging inrunnerimplemented onlogbookand available through thelogobject.
So you could even uselogin yourscript.py.> C:\runner\jobs\0\script.py

from job import job

# Write the code to be executed by job down below.
print('Wake up, Neo...')
job.log.info('Hello there.')Let's imagine that today is 31th December 2018, 23:59:59.
Run the job and in log file you will see:> C:\runner\jobs\0\logs\test_20181231235959.log

********************************************************************************
*                                                                              *
*          APP: test                                                           *
*         DESC: Test Job                                                       *
*      VERSION: None                                                           *
*    TIMESTAMP: 2018-12-31 23:59:59                                            *
*         FILE: C:\runner\jobs\0\logs\test_20181231235959.log                  *
*      MACHINE: AMD64                                                          *
*    PROCESSOR: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel             *
*     HOSTNAME: Falcon                                                         *
*         USER: Kenobi                                                         *
*    SYSTEM_NM: Windows                                                        *
*   SYSTEM_RLS: 10                                                             *
*   SYSTEM_VER: 10.0.14393                                                     *
*       PYTHON: 3.7.1                                                          *
*           ID: 0                                                              *
*      TRIGGER: 2018-12-31 23:59:59                                            *
*      CONFIGS: C:\runner\jobs\0\config.ini                                    *
*      PERSONS: None                                                           *
*                                                                              *
********************************************************************************
2018-12-31 23:59:59|INFO|JOB STARTED.
2018-12-31 23:59:59|INFO|Hello there.
2018-12-31 23:59:59|INFO|JOB FINISHED.
2018-12-31 23:59:59|INFO|TIME SPENT: 0 seconds.Visitlogbookpage to see more features of it.Notifications and AlarmsSometimes in work you may need to get feedback from the job.
It can be easily done with a help ofemailobject which allows to write messages on the email.To use notifier you need SMTP server.
Connection must be defined in config:ip,port,user,password,email addressoptions of the[EMAIL]section in the job config (or other external config).To send simple text message on the email:> C:\runner\jobs\0\script.py

subject = 'Help me'
text = 'You are my only hope'
recipient = 'ObiWanKenobi@email.com'
job.email.notify(subject, text, recipient)Another case of using notifier is to inform that something went wrong in the job execution.
We use alarms for this purposes inrunner.
Usealarm()to send standard alarm notification on emails listed in thepersonsoption of the[GENERAL]section in the job config.Let's change our job with id 0 little bit and execute it.> C:\runner\jobs\0\config.ini
[JOB]
persons = user@email.com> C:\runner\jobs\0\script.py
try:
  1/0
except ZeroDivisionError:
  job.log.error()
  job.alarm()As a result you will receive a message onuser@email.com:Error occurred during execution.
Job id: 0.
Job name: test.
Job description: Test Job.
Job trigger: 2018-12-31 23:59:59.
Please check the C:\runner\jobs\0\logs\test_20181231235959.log for more details.InC:\runner\jobs\0\logs\test_20181231235959.logyou will find the error details:2018-12-31 23:59:59|ERROR|FAIL! ERROR: ZeroDivisionError. FILE: C:\runner\jobs\0\script.py. LINE: 7. REASON: division by zero.Visitnotifierpage to see more features of it.ConfigurationYou could configure both scheduler and job using configuration files or writing parameters directly to instances constructors.
Remember that direct write to instances has a higher priority than configuration files.Also you are able to use additional configuration files by passing them through theconfigparameter ofSchedulerorJobinstances:scheduler = runner.Scheduler(config = 'C:/configs/myconfig.ini')
# or
job = runner.Job(config = ['C:/configs/myconfig.ini', 'C:/users/Kenobi/myconfig.ini'])But always consider that once you mention these additional configuration files then all parameters from them will be copied torunnerconfiguration files.To edit main configuration file use commandedit config:
To edit certain job configuration file use commandedit config job 0.Options which most likely will be used are already placed to the configuration files.
Below the options descriptions.OptionSectionValue ExampleDescriptioneditorMANAGERnotepad, nanoText editor used to edit files.nameSCHEDULER, JOBscheduler, job_000Name of scheduler or job.descSCHEDULER, JOBScheduler, Job 0Description of scheduler or job.scheduleSCHEDULERC:\runner\schedule.tsv, /runner/schedule.tsvPath to schedule file.consoleLOGTrue, FalseOutput log to console instead of file.limit_by_dayLOGTrue, FalseDo we need to close/open log at the start of new day?limit_by_sizeLOGTrue, FalseDo we need to close/open log when maximum size is reached?max_sizeLOG10485760Maximum size of log.showtimeLOGTrue, FalseDo we need to see each scheduler moment in log?showdelayLOGTrue, FalseDo we need to see scheduler active phases time consumptions in log?pythonENVIRONMENTpython, python3Path or command to Python executable.cppENVIRONMENTcppPath or command to C++ executable.javaENVIRONMENTjavaPath or command to Java executable.personsJOBObiWanKenobi@email.comEmail address (one or more) who receives job notifications and alarms.ipEMAIL127.0.0.1Ip address of host with SMTP server.portEMAIL587Port of STMP server.need_loginEMAILTrue, FalseDo we need to login to SMTP server?userEMAILKenobiUsername to login.passwordEMAIL4mayBeWithYouPassword to login.addressEMAILLukeSkywalker@email.comEmail address that sends job notifications and alarms.need_debugEMAILTrue, FalseDo we need to see details of connection process?need_tlsEMAILTrue, FalseDo we need to cover connection to SMTP with TLS protocol?More About EnvironmentsInrunneryou could add any environments for job execution.
Let's example on Python virtual environmentvenv:$ python -m venv C:\runner\venvThat will deploy Python virtual environment right in the folder with our runner application.Open main config and add to theENVIRONMENTsection a new option with the path to the deployedvenvmain executable:[ENVIRONMENT]
<...>
venv = C:\runner\venv\scripts\python.exeThen change in the schedule theENVIRONMENTfor the test job frompythonpicked during the creation tovenv.Now your job will be handled by virtual Python copy, sopip installany modules you need and feel your self free.For information about constructors attributes refer to the technical documentation.For details of external module objects go to appropriate page with this module documentation.TestsYou could find tests for Windows and Linux in /test.IssuesTo report about found bugs and problems refer toissuesUninstallTo uninstall just delete folder your/python/folder/Lib/site-packages/runner.