Package: logsweet
logsweet========A suite with a variety of tools for handling log messages.The name *logsweet* is a word play by combining *sweet logging* and a*suite of logging tools*.Requirements-------------  Python ≥3.4Installation------------Python Package ``logsweet``~~~~~~~~~~~~~~~~~~~~~~~~~~~::pip install logsweetor::python ./setup.py installDocker Image ``mastersign/logsweet``~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~There is a public Docker image for *logsweet* available... code:: shdocker run --rm -it mastersign/logsweet [command] [OPTIONS] [ARGUMENTS]Remember to mount log files as volumes into the container, when usingthe ``watch`` or ``mock`` command.Usage-----Logsweet supports the following commands:-  ``watch`` watches for new lines in text files (e.g. log files) andbroadcasts them via a ∅MQ PUB socket and/or sends them via a ∅MQ PUSHsocket.-  ``listen`` listens to new lines broadcasted via ∅MQ PUB socket and/orcollects new lines from ∅MQ PUSH sockets and prints them on theconsole.-  ``proxy`` can connect to watchers at the backend and to listeners atthe frontend. It supports both transmission modes (PUB/SUB andPUSH/PULL) at both backend and frontend. It can be used as a singlepoint of knowledge or to build groups of log streams.-  ``mock`` to write random log messages to text files for testingpurposes.One or more known services~~~~~~~~~~~~~~~~~~~~~~~~~~At the service hosts ``service-1`` and ``service-2`` run:.. code:: shlogsweet watch -b *:9000 "/var/log/myservice/*.log"To watch the two services:.. code:: shlogsweet listen -c service-1:9000 -c service-2:9000Use a proxy for dynamic services~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~First start the proxy on host ``log-proxy`` to wait for backendconnections from services on port 9001 and frontend connections fromlisteners on port 9002:.. code:: shlogsweet proxy -bb *:9001 -fb *:9002Run one or more listeners to print the messages:.. code:: shlogsweet listen -c log-proxy:9002Start one or more watchers to send log messages to the proxy:.. code:: shlogsweet watch -c log-proxy:9001 "/var/log/myservice/*.log"Use two proxies for high availability~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~First start two proxies on host ``log-proxy-1`` and ``log-proxy-2``:.. code:: shlogsweet proxy -bb *:9001 -fb *:9002Then start one or more listeners to print the messages:.. code:: shlogsweet listen -c log-proxy-1:9002 -c log-proxy-2:9002Start one or more watchers to send log messages to the proxies:.. code:: shlogsweet watch -c log-proxy-1:9002 -c log-proxy-2:9002 "/var/log/myservice/*.log"If multiple proxies are alive, the load is balanced evenly over allproxies. If a proxy becomes unavailable, its load is automatically takenby the remaining proxies.Write random log messages for testing~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~You can write random log messages to one or more log files in order totest *logsweet* and its usage scenarios:.. code:: shlogsweet mock -i 0.2 /var/log/test/1.log /var/log/test/2.logUse the ``-i`` or ``--interval`` option to control the interval for newlog messages in seconds.Configuration-------------The commands ``watch`` and ``listen`` can process a YAML configurationfile. The configuration controls *filtering*, *coloring* output andtriggering *actions*.For the command ``watch`` to print the text lines, the switch ``--echo``must be used.A configuration file is specified with the ``-cfg <yaml file>`` option.Here is an example:.. code:: yamlversion: '0.1'# The include statement is an array of regular expressions or a single one.# A log messages is dropped if it does not match any of the expressions.include:- 'error|Error|ERROR'- 'warning|Warning|WARNING'# The exclude statement is an array of regular expressions or a single one.# A log message is dropped if it matches any of the expressions.exclude: 'ignore\s+this'# The colors statement is an array of colorization rules.# The rules are tried in the given order until one matches.colors:# A colorization rule must have a pattern, which is a regular expression,# optionally with named groups.- pattern: '\[ERROR\]'# There are multiple color statements with the syntax of.# <foreground color> [on <background color>]# Every color is a color name from the X11 rgb.txt.# Spcifically the names supported by the Python package colorful# are supported.# The color names are Camel Case with a lower first letter.line: red  # line specifies a color for the whole linematch: black on lightGrey # match specifies a color for the whole match- pattern: '^.*?(?P<level>\[[A-Z]+\]).*(?P<user>admin|user)'line: lightGreymatch: whitelevel: yellowuser: cyan# The actions statement is an array of action rules.# The rules are all tried, regardless of matches.# To execute actions from a watch or listen command,# the switch -x or --exec-action must be used.actions:# An action rule must have a pattern, which is a regular expression,# optionally with named groups.- pattern: '\[ERROR\] (?P<err>.*)$'# If the rule has an url property, it is considered an HTTP GET action.# The URL is treaded as a Python string template,# that means it can contain variables from the named regex groups.url: 'http://127.0.0.1:23456/notify-error/${err}'# A timeout in seconds can be specified,# to limit the amount of time used to execute the HTTP request.timeout: 2.5*******Changes*******All notable changes to this project will be documented in this file.Until this project reaches the maturity of version 1.0.0, the versioningfollows the following rules:- The MAJOR number is 0.- The MINOR number increases, if backward compatibility was brokenor an important set of features has been added.- The PATCH number increases, if backward compatible featureshave been added or bugs got fixed.After this project reaches the version number 1.0.0 it will follow`Semantic Versioning <http://semver.org/>`_...This document follows the guidelines in http://keepachangelog.md.Use the following change groups:Added, Changed, Deprecated, Removed, Fixed, SecurityAdd a link to the GitHub diff like.. _`<this version>`: https://github.com/mastersign/logsweet/compare/v<last-version>...v<this-version>`<this version>`_ - <date>==========================Unreleased==========Added------ proxy- binding and connecting on all sides- Docker release- YAML configuration for filtering, coloring and HTTP actions- improved CLI0.0.0 - 2019-02-21===================Added------ basic project structure- text file watcher with ZeroMQ PUB- ZeroMQ SUB with printing- mockup log files- basic CLI