Package: jw.lognotify
This is yet another real-time log scanner done for the ever reoccurring reason of failure to find an existing suitable
tool.Featuressimple, yet flexible configuration in theYAMLformatactions programmable in Python, Tcl or Bashlog rotation detectionvariable leading log context for every eventburst mode: collect events within a certain time frame and report them togetherConfigurationLognotify reads a configuration file specified on the command line with the--configor-coption. The
configuration specifies what content to search for and what to do if some is found in the log.OverviewConsider this example:when:
    - error
    - problem
    - critical
    - fatal
    - bad
    -
        - not
        - ^perl
        - ^/pascal/i
do:
    python: |
        if logfile:
            print(logfile, messages[-1][-1])
    burst: 2
---
when:
    - strange
do: |
    echo $logfile: ${message[-1]}A configuration can have one ore more sections, denoted by a---delimiting line. Each section specifies what to
look for and what to do if something interesting is found.SectionsEach section consists of awhenclause and adoclause. For every line from a logfile, all sections are checked for
matching search expressions in thewhenclause.  When a match is found for a section, the correspondingdoclause
is executed.WhenclauseThewhenclause contains an itemized list of expressions to searche for in every incoming line from the log.
Variations in syntax specify how to search for the item. The expressions are tried in order. As soon as a match is
found, thedoclause is executed and processing of this particular section is terminated.Thewhenlist forms anOR-expression. Within the list, sublists may specifyAND-expressions. Thus- expr1
-
    - expr2a
    - expr2bmeansexpr1 OR (expr2a AND expr2b)Groups can be infinitely nested. The general rule is that every group within anORgroup is anANDgroup and vice
versa.Search expressionsA search expression can have one of the following forms:wordSearch forwordirrespective of case and match only at word boundaries. Thuserrormatches in the following
lines:Error: invalid syntaxAn error occurred.No error-checking enabled.butnotin:No errorchecking.maxerrorIf you want to match irrespective of word boundaries, you have to use regular expressions (see below)./word/[flags]Search for aregular expression. Some flags for altering the
operation are available:imatch case insensitive. SeeIGNORECASE.mmatch in multi-line mode. Probably not very useful. SeeMULTILINE.lmatch according to the current locale. SeeLOCALE.smake ‘.’ match any character, including newline. SeeDOTALL.umatch according to the Unicode character properties table. SeeUNICODE.xparse verbose regex with comments and white space. SeeVERBOSE.All these expressions can be prefixed with a caret (^) to mean “do not match word”:^word^/word/NoteSince the whole configuration is expressed in YAML, strings containing certain characters must be quoted in order
not to interfere with the YAML syntax. These characters are:[ ] { } ! " ' : ? % @ , - # ~ | > * &. Also,
certain words have special meaning in YAML and must therefore also be quoted:yes,no,on,off,true,false, andnull.PitfallsThe search algorithm gives rise to surprises in certain constellations. One common error is to request something
like this:-
    - not
    - ^this
-
    - not
    - ^thatwhere^thisand^thatcancel each other out. If a line contains ‘not’ it will always match, no matter whetherthisorthatoccurs in the line. The proper way would be-
    - not
    - ^this
    - ^thatThe most common pattern is to search for any line containingword1,word2orword3but notexcept1orexcept2.
You might be inclined to write this as- word1
- word2
- word3
-
    - ^except1
    - ^except2But this would not work. The way to do it goes along the follong lines: written as a logical expression, it would be(word1 OR word2 OR word3) AND (NOT except1 OR NOT except2)which translates to(word1 OR word2 OR word3) AND NOT except1 AND NOT except2which, expressed as list operations, translates toAND(OR(word1, word2, word3), NOT(except1), NOT(except2))We have therefore anANDlist on top. However, in lognotify we start out in anORlist. We therefore have to put ourANDlist as the single element into the topORlist. The final result would be# OR list
-
    # AND list
    -
        # OR list
        - word1
        - word2
        - word3
    - ^except1
    - ^except2DoclauseThedoclause specifies what action to take when one of the expressions in thewhenclause matches. To run commands
on the selected logfile lines,Python,bashorTclcan be used. Some
variables are injected, depending on the language used. Scripts receive one or more events at a time depending on
whether context and/or burst mode was requested. If neither context not burst mode is requested, one single line is
reported at a time.NoteUse the pipe character at the end of a line prior to the code block to cause YAML to process the following indented
block without interpretation, leaving line endings intact (see the examples below).ContextContext is a number of lines running up to the actual log event line. It can be requested with the--config/-Cflag. Context lines are marked with aTruevalue inPythonorTclor a value of 1 inbashorshto
distinguish them from log lines. However, if a context line is also a regular log line (appearing because it is part of
a burst) it is not marked as such.Burst modeIn burst mode, log lines arriving within a certain time frame are kept together and appear in the same call. Burst mode
can be requested either as aburstspecifier in adoclause or with a--burstor--force-burstcommand line
flag. Good values for burst time frames are between 2 and 5 seconds. The--force-burstflag overrides values
specified indoclauses while--burstdoes not.PythonPythoncode can be one block or be split into an initialization section and a runtime section. The former is executed
once at startup and is intended to contain stuff likeimportstatements, function definitions and the like. The
latter is run for every event.InPython, the following variables are available:logfileA string containing the path of the logfile where the event was coming frommessagesA list of tuples. For each event the tuple contains:a bool which is True if the entry is a context linethe sequence numbera float with a timestampa string with the message textExample (assuming Python3 syntax):do:
    python: |
        for msg in messages:
            print('%s: %s' % (logfile, msg[3]))Example with initialization (assuming Python3 syntax):python:
    - |
        # Setup UDP socket
        import sys
        import socket
        sock = socket.socket(type=socket.SOCK_DGRAM)
        sock.connect(('127.0.0.1', 7777))
    - |
        # Write stuff to UDP socket
        for msg in messages:
            sock.send('json:{}\n'.format(msg[3].replace(r'\n', ' ')).encode('u8'))TclTclcode can be one block or be split into an initialization section and a runtime section. The former is executed
once at startup and is intended to contain stuff likeprocstatements. The latter is run for every
event.InTcl, the following variables are available:logfileThe path of the logfile where the event was coming frommessagesA list of lists. For each event the inner list contains:a bool which is True if the entry is a context linethe sequence numberan int with a timestampa string with the message textExample:do:
    tcl: |
        foreach m $messages {
            puts "$logfile: [clock format [expr int([lindex $m 2])]] [lindex $m 3]"
        }Example with initialization:do:
    tcl:
        - |
            proc output {m} {
                puts $m
            }
        - |
            foreach m $messages {
                output "$logfile: [clock format [expr int([lindex $m 2])]] [lindex $m 3]"
            }Bash and shInbashandsh, the following variables are available:logfileThe path of the logfile where the event was coming fromiscontextAn array with an int for every line where 1 means it is a context line or 0 otherwiseseqnoAn array containing the sequence number for every linetimeAn array containing the timestamp in ISO format for every linemessageAn array containing the text for every lineExample:do:
    bash: |
        echo $logfile: ${time[-1]} ${message[-1]}But sincebashis the default language, it can be written as:do: |
    echo $logfile: ${message[-1]}Thedoclause can be omitted altogether in which case a default ofdo:
    python: |
        for msg in messages:
            print('%s: %s' % (logfile, msg[3]))is assumed.RunningCommand synopsis:lognotify[-h] –configCONFIG[–full][–burstBURST| –force-burstFORCE_BURST]
[–contextCONTEXT] [–debug] [–version]
logfile [logfile …]Positional arguments:logfileA log file to scanOptional arguments:-h,--helpshow this help message and exit--configCONFIG,-cCONFIGspecify config file--full,-fscan files from beginning--burstBURST,-bBURSTreport bursts of BURST seconds together--force-burstFORCE_BURST,-BFORCE_BURSTforce reporting bursts of BURST seconds together--contextCONTEXT,-CCONTEXTspecify context size--debug,-dPrint some debug information to stderr--version,-vdisplay version and exitAt least one path to an existing, readable log file is expected.The--fullor-foption requests reading files from the start. Without the flag, reading begins at the current
end of file. Sequence numbering always begins from the point where reading begins.The--debugor-doption sends information to the standard error file. Repeating the flag increases the
amount of information.Useful scriptsThis section is a collection of useful scripts.Send desktop notificationTo be used as root (change'username'accordingly):from subprocess import check_call

check_call(
    [
        'su', 'username', '-c',
        'DISPLAY=:0 notify-send "%s" "%s"' % (logfile, '\n'.join('> '[m[0]] + m[3] for m in messages))
    ]
)Send desktop notification to remote machineTo be used as root (change'hostname'andusernameaccordingly):from subprocess import check_call
from platform import node

check_call(
    [
        'ssh', 'hostname',
        r'su username -c "DISPLAY=:0 notify-send \"%s: %s\" \"%s\""' % (
            node(),
            logfile,
            '\n'.join('! '[m[0]] + m[3] for m in messages)
        )
    ]
)Send e-mailChange'mail-user','mail-user-password',source-emailanddestination-emailaccordingly:do:
    python: |
        from smtplib import SMTP
        import sys

        client = SMTP('localhost')
        try:
            client.starttls()
        except:
            pass
        client.login('mail-user', 'mail-user-password')
        client.sendmail(
            'source-email',
            'destination-email',
            'From: source-email\n'
            'To: destination-email\n'
            'Subject: Message in %s\n\n'
            '%s\n' % (logfile, '\n'.join('> '[m[0]] + m[3] for m in messages))
        )