Package: urlmonitor
UrlMonitorThis program is intended to be run periodically from cron or a systemd.timerservice on GNU/Linux, a Periodic Task on Windows, or whatever you are using.When run, the program will check the URLs from its list, and, if they have changed since the last time they were checked, will run a series of actions as a result.The list of URLs and actions to be run if changed are specified in a YAML file (by default calledurllist.yml). This file consist on a LIST of OBJECTS, each containing the fields name with a descriptive string, url with the url to be checked, and actions, which is a LIST of OBJECTS, where the key is the action name and the value are the arguments, which can be a STRING or a LIST of strings.An example of urllist.yml file:----name:Check for new versions of my favourite softwareurl:https://www.coolproject.org/download/filesactions:-email_notify:-joe@domain.com-jill@somewhere.else.com-name:Get latest version of the softwareurl:https://www.coolproject.org/latestactions:-re_match:-'[Ss]ource[Tt]arball.*<ahref=\"([^"]+)'-run_script:download_and_build https://www.coolproject.org/{re_match[1]}As you can see on the last line, the arguments to the actions are expanded with the variables from previous actions, the url, the contents, the HTTP status code obtained while fetching the URL, etc.InstallationJust run the providedsetup.pyprogram:$ python3 setup.py installRunningYou can call the program with--helpfor the program options::$ urlmonitor --help
usage: urlmonitor [-h] [--config FILE] [--persist-file FILE]
                  [ymlfile [ymlfile ...]]

positional arguments:
  ymlfile               Url check specification.

optional arguments:
  -h, --help            show this help message and exit
  --config FILE, -c FILE
                        Configuration file.
  --persist-file FILE, -p FILE
                        Persistence file to store URL status. Default
                        ./persistence.sqliteTheymlfile(s) contain the URLs to check, as discussed above.The--persist-fileis a database that will be created in order to track whether the URLs have changed and when.The--configfile is a YAML file that specifies where the options are located. It is discussed on the next section.ConfigurationUrlMonitor is configured using a YAML file containing an OBJECT with the different sections where you can specify:Global variables (set_vars): available to all the actions.Actions (actions): Each additional action individually, specifying its name, the Python module where it is located, and perhaps a set of configuration parameters.Action directories (action_dir): Specify a directory containing Python files. Each Python file contains an action named like the file.Action configurations (action_config): Contains an OBJECT whose entries are the names of the actions and the values are the configuration parameters for that action. Any action can be configured in this way, whether it has been specified usingactionor anaction_dir. Some predefined actions also need to be configured this way, for example theemail_notifypredefined action needs the parametersmtp_serverto be set.Logging configuration (logging_config): Contains the settings needed to calllogging.config.dictConfig()as per thePython standard library documentation. The logger name that will be used isURLMONITOR. If this section is not present, sensible defaults will be used.An example of the configuration follows::---# define global variables for all the actionsset_vars:var1:5.4var2:-list-of-"strings"var3:keys:ofa:dict# load all the actions in a directory#   all the .py files in the directory will be loaded#   if the module is action_name.py, the action name will be 'action_name'#   and will be initialised with the objects in its entry in an#   an 'actions_config' section#action_dir:/path/to/dir1action_dir:# specify a list of paths-/path/to/dir2-/another/path/dir3# actions and configuration objects:# configuration objects are mappings that the actions need to initialise# themselves, like the smtp host in a 'mail_send' action or a database name# and connection parameters to look up/store stuff etc.## The actions will be called (or not) at runtime with the url that's changed,# the contents of that url, and the variables with their values at that point.actions_config:# configuration objects for actions defined on action_dir# as opposed to actions defined in actions:action_in_dir1:update_url:https://www.someserver.comsomething_or_another:34some_action_in_dir3:parameter:valueanother_parameter:another_valueactions:# always a list containing mappings of actions and configuration values# should always have a name and module-name:add_to_databasemodule:/path/to/module.py# ad_hoc configuration parameters, defined and expected by# each particular actiondbhost:host1.mynetworkdbname:documentsdbuser:albertdbpassword:hello0011-name: another_action:module:/the/path/to/another_action.pyfoo:barlogging_config:# the following is passed directly to logging.config.dictConfig()loggers:URLMONITOR:level:DEBUGhandlers:-consolehandlers:console:class:logging.StreamHandlerstream:ext://sys.stdoutlevel:DEBUGPredefined ActionsThe following actions are available out of the box:set_varsSet global variables. The variables defined using this action are global and can be used by all the URL objects and their actions.Example:set_vars:value_1:34list_var:-item 1-item 2-another itemlist_varsThis action lists all the variables and their values to the log. The log level must be debug.no_actionThis is a dummy action. It does nothing.re_matchSearch the content of the URL using one or more regular expressions passed as arguments. The regular expressions are applied usingre.MULTILINE(seePython'sremodule documentation) i.e. '^' matches the beginning of every line and '$' matches every end of line.As a result, it creates two variables:re_match: A list of all the group matches in the first regular expression.re_match_all: A list containing the list of all group matches of all the regular expressions in the argument list, in the same order as they appear.Example:-name:Get amount and codeurl:http://www.interesting.com/amounts_codes/actions:-re_match:-'Theamountis:\s*([0-9]+)'-'Code\s*(\w+)'-set_vars:amount:{re_match[1]}code:{re_match_all[1][1]}email_notifySend an email with a notification of the URL change to the recipient email addresses specified in the argument list.Example:-name:Check for new versions of my favourite softwareurl:https://www.coolproject.org/download/filesactions:-email_notify:-joe@domain.com-jill@somewhere.else.comBefore this action can be used it must be configured in the yaml configuration file with the following parameters:NameValueDefault valuesmtp_serverserver namefrom_addressemail addressurlmonitor@hostnamesmtp_portport number25 for SMTP, 587 for SSLsmtp_encryptionSSL or STARTTLSempty, no encryptionsmtp_userusernameempty, no smtp authenticationsmtp_passwordpassword base64 encodedConfiguration example:-actions_config:email_notify:smtp_server:mailhost.server.comfrom_address:urlbot@mydomain.comsmtp_encryption:STARTTLSsmtp_port:465smtp_user:philsmtp_password:aGVsbG8wMDExMjI=run_scriptThis action runs a program whose command line is its argument list.Example:-name:Download the new fileurl:https://www.filesite.net/targetfiles.htmlactions:-re_match:'<ahref=\"(.*)\">Thefile'-run_script:wget {re_match[1]}This action sets the following variables:VariableValuereturn_codeThe exit code of the programerrorThe error encountered, if anystdoutWhatever the program has written to its standard outputstderrWhatever the program has written to its error outputWriting Custom ActionsYou can write a custom action easily. You just need to write a Python module that exports a callable calledaction_objectwith the following signature:defaction_object(name,arglst,url,content,variables,log):passWhere the arguments are:ArgumentDescriptionnameThe action namearglstThe list of arguments.urlThe url that changedcontentThe content (possibly HTML) of the urlvariablesA mapping containing the variables and their valueslogA logging objectIf the arguments inarglstwill be expanded by the calling routine and should contain the final value.If your action requires configuration parameters, you can create a class that has a__call__method and instantiate intoaction_object. At configuration time, the program will call theinitialisemethod with a dictionary containing the configuration parameters as argument.In order to simplify the process, the moduleactionbaseprovides the classActionfrom which your action can inherit. This already provides aninitialisemethod which will install the configuration parameters, which you can access as instance attributes ofaction_object. In addition, you should provide a list of expected parameters and a default value for those that can have one in the class variablescheck_cfg_varsanddefault_vars. An exception is raised if a parameter that has been specified incheck_cfg_varsis not found and there is no default value.As an example, you can check the code for theemail_notifyaction.fromurlmonitor.actionbaseimportActionclass_EmailAction(Action):check_cfg_vars=["smtp_server","from_address","smtp_encryption","smtp_user","smtp_password","smtp_port"]default_vars={"from_address":"urlmonitor@{}".format(_NODE),"smtp_encryption":None,"smtp_user":None,"smtp_password":"","smtp_port":0,}...def__call__(self,name,arglst,url,content,variables,log):...action_object=_EmailAction()LicenseThis software is released under theMIT License