Package: taskwatcher
taskwatcherDESCRIPTION'taskwatcher'is a set of libraries/commands to turn scripts/programs intotasks:launch.py: Starts a new task under taskwatcher monitoringcontrol.py: General control commandsparse.py: Feedback file parser, returns a json string from the feedback fileAnhistoryof the terminated tasks is kept.Astatusof the currently running tasks is available.Tasks aremonitoredby checking if pid is still in process list.
Optionally, if feedback is provided by the launched program, program is expected to provideheartbeatby updating the feedback file.An optionaltask timeoutmay terminated the task if no hearbeat was received during the allowed 'timeout' period.There are conditions applies so a task can start, multiple scenario :Task does not provide feedack, can run multiple time:no control need, a call to lauch would be enough, task would be automatically reserved and launchedTask does not provide feeback but only one of this kind should work at a given timer:a reservation with a task name and option --unique set should be done befor calling launch. A taskid would be returned, use this task id in call to lauch.Task provides feeback.Reservation is required prior calling launch.If task should only run once, use --unique option during reservation.Prior to any launch call to start a new task, a taskid (number) should have been reserved with a call control.py --reserve.A feedback file is a simple text file with key/value pairs. It can be parsed with parse.py to return a json formated string.LimitationsCurrently only supports mono-process program (track a single pid)DesignChoice was made to avoid complex sockets operations.
Simple design with feedback files also allows simple state recovery, simplicity and versatility of the feedback parameters.
A basic file-based sqlite database is used to store running tasks data and keep track of historical tasks, instead of implementing a daemon.ConfigurationCurrently no configuration file is implemented. All options should be provided using cli options.No centralized daemon:a launcher is started with the program to rungetting feeback from the running task can be done by calling control.py with command 'feedback'history and current status of running tasks are located in sqlite dbExpectation from the launched commandA command may be ran without specific requirement, however to benefit from additional features, the run command may provide feedback via a text file.The suggestion is to add to programs launched with taskwatcher a command line option (--feedback) to generate thefeedbackfile.The feedback file is expected to be named based on the taskid : feedback_TASKID.log
Example :Call without taskwatcher :checkitbaby.py --playbook myPlaybook --playlist myPlaylist --run 1 --dryrunSame call with taskwatcher :Get a taskid :control.py --reserve==> Got 1Launch task using taskid 1 :launch.py --taskid 1 --name 'Runner' --info1 'Info1' --info2 'Info2' --info3 'Info3' --feedpath /fortipoc/playbooks/myPlaybook/run/1 --db /fortipoc/taskwatch/sqlite.db --timeout 30 -- checkitbaby.py --playbook myPlaybook --playlist myPlaylist --run 1 --dryrun --feedback feedback_1.logNotes :info1, info2, info3 are optional, it can be usefull to store side information in the task managerthe command to run is located after the --called program is informed with --feedback feedback_1.log that it should write a feedback file named feedback_1.logRunning task statuswithout feedback :running: pid of the task is seenwith feedback :running: pid of the task is seen,last feedback file update was done in less than 25% of timeout timersilent: pid of the task is seen,last feedback file update was done in less than 50% of timeout timerstalled: pid of the task is seen,last feedback file update was done in less than 75% of timeout timerFeedback file syntaxThe feedback file is named 'feedback.log', it should be generated by the launched program.
Any kind of usefull information could be delivered as long as :it is key/value pairline starts with a keywork emcompassed with [] to specify the information keyword.keyword should have no spaces and should not start with a digitvalue provided should immediately follow the [keyword] without spacesline starting with # are considered comments/debug and will be ignoredempty lignes will be ignoreda keyword without any value on a line clears the keyword and value information from the feedback[] (without keyword) clears all key/value pairs collected up to this point.
Note : this could be usefull to end the task with a clear followed by a report.Feedback file processinglaunch.py does not parse feedback file. It only checks the file update from the file update time for the task timeout fonction.control.py processed feedback file to provide output.The last read value for a keyword updates any precedent values.use parse.py to parse and retrieve json from the feedback file.Example of a feedback.log[info]Starting test
[playbook]myPlaybook
[info]
[heartbeat]
[testcase_id]001
[testcase_name]Initialization
[progress]12
[progress]15
[testcase_id]002
[testcase_name]Setting up topology
[progress]10If processed until this point, the above output would provide information like:{"playbook":"myPlaybook","testcase_id":"002","testcase_name":"Setting up topology","progress":"10"}Notes :'info' is not provided because it was cleared by the '[info]' line in line 3'heartbeat' has no information, it would only reset the 'timeout' counterlaunch.pyRoles : 
   - launches command
   - monitor the command checking its pid in process list
   - monitor activity of the feedback file (by its update timing information)
   - kills command if not updating feedback file within the timeout 
   - update the running task db about process state, duration and timer status
   - manage task termination : archive task on database, delete feedback file

Pre-requisite :
A unique taskid should have been reserved from a call to control.py to avoid duplicates.
If no reservation was made, the task won't start

Usage : launch.py --db <database> --taskid <taskid> --command '<process or script with all its options>'

Parameters :
--db       <database>     : sqlite database file
--taskid   <taskid>       : task identifier (could be a number or a generated random string (8 chars max)    

Optional parameters :  
--name     <name>         : Name for the task. Use command name if not provided
--info1 INFO1             : any information (optional)
--info2 INFO2             : any information (optional)
--info3 INFO3             : any information (optional)

--feedpath <path>         : Feedback path where feedback.log is expected
--timeout  <seconds>      : a value in second after which the command is considered timeout
				            and should be kill (any update in feedback.log resets the timer)control.pyRoles :
   - Provide list of all running tasks with their latests status
   - Provide history of terminated tasks
   - Provide all latests feedback information from the task
   - Manage history
   - Kill task (and cleanup feedback file if necessary)
   - Initialise (or re-initialize db)
   - Set a reservation for a task id

Usage : control.py --db <database> <command>

Options :
--db <database>      : sqlite database file

List of available commands :

--initialize         : Create or recreates a task database (all info is lost)

--update             : Update database time informations (task duration)

--list               : Provides a table displaying the list of the currently running tasks with : 
                       [ taskid, name, info1, info2, info3, pid, status, starttime, duration(s), feedback(yes/no), timer(s), timeout(s) ]

--reserve            : Returns a unique taskid, reserved for future call of the launcher

--feedback <taskid>  : Returns a json formatted output of the feedback values for the given task
                     : Only available if the command provides feedback (feedback=yes in list)

--history            : Dump all historical tasks completed
--clear              : Clear all tasks history

--kill <taskid>      : Request to terminate a specific task
--killall <taskname> : Request to terminate all tasks named <taskname>parse.pyusage: parse.py [-h] [--debug] --feedback filename

Task controller

optional arguments:
  -h, --help           show this help message and exit
  --debug, -d          turn on debug
  --feedback filename  selects feedback file to process


Example: 
 ./parse.py --feedback tests/textfile_progress.txt{\"progress\":\"100\"}sqlite databaseAn sqlite database is used for 3 purposes :keep track of the current running tasks :Launcher.py updates tasks status but does not process feedbackkeep track of the latest feedback from the launched commandcontrol.py called with --feedback parses command feedback file, stores information and returns a json objectkeep an history of the previously completed taskscontrol.py called with --historyTable format* Table tasks:
  Keeps track of running tasks status
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------
  |       id(#1)        |  name | info1 | info2 | info3 |   pid   |   status   |   feedback    |  reservetime |   starttime   | duration |  lastupdate  | timeout |
  | INTEGER PRIMARY KEY |  TEXT | TEXT  | TEXT  | TEXT  | INTEGER |  TEXT(#2)  |  INTEGER(#3)  |  INTEGER(#4) |   INTEGER(#4) | INTEGER  |  INTEGER(#4) | INTEGER |
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------

  Note : 
    #1 : should be automatic (use null during insert)
    #2 : RESERVED|RUNNING|SILENT|STALLED
    #3 : 0 if no feedback provided ; 1 if feedback provided
    #4 : unix date format

  taskid reservation consists of inserting a new task with all field empty, except status=RESERVED and reservetime set

* Table feedbacks:
  Keeps track of data feebacks from the run command, stored as json key/value pairs
  ---------------------------------------
  |    id   | feedback  |   lastupdate  |
  | INTEGER |  BLOB(#1) |   INTEGER(#2) |
  ---------------------------------------

  Note :
  #1 : json format expected
  #2 : unix date format


* Table history:
  Keeps track of the completed tasks
  Final state of json feedback is stored (this allows to store json reports before the command terminates)
  --------------------------------------------------------------------------------------------------------------------------------------------------------
  |       id(#0)        |  taskid(#1)  | taskname |  info1 | info2 | info3 | termsignal | termerror |   starttime   |   endtime   | duration | feedback  | 
  | INTEGER PRIMARY KEY |   INTEGER    |   TEXT   |  TEXT  | TEXT  | TEXT  |  TEXT(#1)  |  TEXT(#2) |   INTEGER(#3) | INTEGER(#3) | INTEGER  |  BLOB(#4) |
  --------------------------------------------------------------------------------------------------------------------------------------------------------

  Notes :
  #1 : keeps track of the type of termination signal
  #2 : keeps track of the terminaison error message if any
  #3 : unix date format
  #4 : json format expected