Package: haproxy-log-analysis
HAProxy log analyzerThis Python package is aHAProxylog parser.
It analyzes HAProxy log files in multiple ways (see commands section below).NoteCurrently only theHTTP log formatis supported.Tests and coverageNo project is trustworthy if does not have tests and a decent coverage!DocumentationSee thedocumentation and APIatReadTheDocs.Command-line interfaceThe current--helplooks like this:usage: haproxy_log_analysis [-h] [-l LOG] [-s START] [-d DELTA] [-c COMMAND]
                            [-f FILTER] [-n] [--list-commands]
                            [--list-filters] [--json]

Analyze HAProxy log files and outputs statistics about it

optional arguments:
  -h, --help            show this help message and exit
  -l LOG, --log LOG     HAProxy log file to analyze
  -s START, --start START
                        Process log entries starting at this time, in HAProxy
                        date format (e.g. 11/Dec/2013 or
                        11/Dec/2013:19:31:41). At least provide the
                        day/month/year. Values not specified will use their
                        base value (e.g. 00 for hour). Use in conjunction with
                        -d to limit the number of entries to process.
  -d DELTA, --delta DELTA
                        Limit the number of entries to process. Express the
                        time delta as a number and a time unit, e.g.: 1s, 10m,
                        3h or 4d (for 1 second, 10 minutes, 3 hours or 4
                        days). Use in conjunction with -s to only analyze
                        certain time delta. If no start time is given, the
                        time on the first line will be used instead.
  -c COMMAND, --command COMMAND
                        List of commands, comma separated, to run on the log
                        file. See --list-commands to get a full list of them.
  -f FILTER, --filter FILTER
                        List of filters to apply on the log file. Passed as
                        comma separated and parameters within square brackets,
                        e.g ip[192.168.1.1],ssl,path[/some/path]. See --list-
                        filters to get a full list of them.
  -n, --negate-filter   Make filters passed with -f work the other way around,
                        i.e. if the ``ssl`` filter is passed instead of
                        showing only ssl requests it will show non-ssl
                        traffic. If the ``ip`` filter is used, then all but
                        that ip passed to the filter will be used.
  --list-commands       Lists all commands available.
  --list-filters        Lists all filters available.
  --json                Output results in json.
  --invalid             Print the lines that could not be parsed. Be aware
                        that mixing it with the print command will mix their
                        output.CommandsCommands are small purpose specific programs in themselves that report specific statistics about the log file being analyzed.
See them all with--list-commandsor online athttps://haproxy-log-analyzer.readthedocs.io/modules.html#module-haproxy.commands.average_response_timeaverage_waiting_timeconnection_typecounterhttp_methodsip_counterprintqueue_peaksrequest_path_counterrequests_per_hourrequests_per_minuteserver_loadslow_requestsslow_requests_counterstatus_codes_countertop_ipstop_request_pathsFiltersFilters, contrary to commands,
are a way to reduce the amount of log lines to be processed.NoteThe-ncommand line argument allows to reverse filters output.This helps when looking for specific traces, like a certain IP, a path…See them all with--list-filtersor online athttps://haproxy-log-analyzer.readthedocs.io/modules.html#module-haproxy.filters.backendfrontendhttp_methodipip_rangepathresponse_sizeserverslow_requestssslstatus_codestatus_code_familywait_on_queuesInstallationAfter installation you will have a console scripthaproxy_log_analysis:$ pip install haproxy_log_analysisTODOadd more commands:(help appreciated)reports on servers connection timereports on termination statereports around connections (active, frontend, backend, server)your ideas herethink of a way to show the commands output in a meaningful waybe able to specify an output format. For any command that makes sense (slow
requests for example) output the given fields for each log line (i.e.
acceptance date, path, downstream server, load at that time…)your ideasCHANGES5.1.0 (2022-12-03)Only get the first IP fromX-Forwarded-Forheader.
[gforcada]Improve tests robustness.
[gforcada]Fixtop_ipsandtop_request_pathscommands output.
They were showing all output, rather than only the top 10.
[gforcada]Movetestsfolder to the top-level.
[gforcada]5.0.0 (2022-11-27)Drop testing on travis-ci.
[gforcada]Use GitHub Actions.
[gforcada]Format the code withpyupgrade,blackandisort.
[gforcada]Usepip-toolsto keep dependencies locked.
[gforcada]Bump python versions supported to 3.7-3.11 and pypy.
[gforcada]Drop python 3.6 (EOL).
[gforcada]4.1.0 (2020-01-06)New command:requests_per_hour.
Just like therequests_per_minutebut with hour granularity.
Idea and first implementation done byvalleedelisle.
[gforcada]Fix parsing truncated requests.
Idea and first implementation byvixns.
[gforcada]4.0.0 (2020-01-06)BREAKING CHANGES:Complete rewrite to use almost no memory usage even on huge files.
[gforcada]Add parallelization to make parsing faster by parsing multiple lines in parallel.
[gforcada]Rename commandcounter_slow_requeststoslow_requests_counter,
so it is aligned with all other_countercommands.
[gforcada]Changed thecounter_invalidcommand to a new command line switch--invalid.
[gforcada]Regular changes:Drop Python 2 support, and test on Python 3.8.
[gforcada]Remove the pickling support.
[gforcada]Add–jsonoutput command line option.
[valleedelisle]3.0.0 (2019-06-10)Fix spelling.
[EdwardBetts]Make ip_counter use client_ip per default.
[vixns]Overhaul testing environment. Test on python 3.7 as well. Use black to format.
[gforcada]2.1 (2017-07-06)Enforce QA checks (flake8) on code.
All code has been updated to follow it.
[gforcada]Support Python 3.6.
[gforcada]Support different syslog timestamps (at least NixOS).
[gforcada]2.0.2 (2016-11-17)Improve performance forcmd_print.
[kevinjqiu]2.0.1 (2016-10-29)Allow hostnames to have a dot in it.
[gforcada]2.0 (2016-07-06)Handle unparsable HTTP requests.
[gforcada]Only test on python 2.7 and 3.5
[gforcada]2.0b0 (2016-04-18)Check the divisor before doing a division to not getZeroDivisionErrorexceptions.
[gforcada]2.0a0 (2016-03-29)Major refactoring:# Rename modules and classes:haproxy_logline -> linehaproxy_logfile -> logfileHaproxyLogLine -> LineHaproxyLogFile -> Log# Parse the log file on Log() creation (i.e. in its __init__)[gforcada]1.3 (2016-03-29)New filter:filter_wait_on_queues.
Get all requests that waited at maximum X amount of milliseconds on HAProxy queues.
[gforcada]Code/docs cleanups and add code analysis.
[gforcada]Avoid using eval.
[gforcada]1.2.1 (2016-02-23)Support -1 as a status_code
[Christopher Baines]1.2 (2015-12-07)Allow a hostname on the syslog part (not only IPs)
[danny crasto]1.1 (2015-04-19)Make syslog optional.
Fixes issuehttps://github.com/gforcada/haproxy_log_analysis/issues/10.
[gforcada]1.0 (2015-03-24)Fix issue #9.
log line on the syslog part was too strict,
it was expecting the hostname to be a string and was
failing if it was an IP.
[gforcada]0.0.3.post2 (2015-01-05)Finally really fixed issue #7.namespace_packageswas not meant to be on setup.py at all.
Silly copy&paste mistake.
[gforcada]0.0.3.post (2015-01-04)Fix release on PyPI.
Solves GitHub issue #7.https://github.com/gforcada/haproxy_log_analysis/issues/7[gforcada]0.0.3 (2014-07-09)Fix release on PyPI (again).
[gforcada]0.0.2 (2014-07-09)Fix release on PyPI.
[gforcada]0.0.1 (2014-07-09)Pickle :class::.HaproxyLogFiledata for faster performance.
[gforcada]Add a way to negate the filters, so that instead of being able to filter by
IP, it can output all but that IP information.
[gforcada]Add lots of filters: ip, path, ssl, backend, frontend, server, status_code
and so on. See--list-filtersfor a complete list of them.
[gforcada]Add :method::.HaproxyLogFile.parse_datamethod to get data from data stream.
It allows you use it as a library.
[bogdangi]Add--list-filtersargument on the command line interface.
[gforcada]Add--filterargument on the command line interface, inspired by
Bogdan’s early design.
[bogdangi] [gforcada]Create a new module :module::haproxy.filtersthat holds all available filters.
[gforcada]Improve :method::.HaproxyLogFile.cmd_queue_peaksoutput to not only show
peaks but also when requests started to queue and when they finished and
the amount of requests that had been queued.
[gforcada]Show help when no argument is given.
[gforcada]Polish documentation and docstrings here and there.
[gforcada]Add a--list-commandsargument on the command line interface.
[gforcada]Generate an API doc forHaproxyLogLineandHaproxyLogFile.
[bogdangi]Create aconsole_scripthaproxy_log_analysisfor ease of use.
[bogdangi]Add Sphinx documentation system, still empty.
[gforcada]Keep valid log lines sorted so that the exact order of connections is kept.
[gforcada]Add quite a few commands, seeREADME.rstfor a complete list of them.
[gforcada]Run commands passed as arguments (with -c flag).
[gforcada]Add a requirements.txt file to keep track of dependencies and pin them.
[gforcada]Addtravisandcoverallssupport. See its badges onREADME.rst.
[gforcada]Add argument parsing and custom validation logic for all arguments.
[gforcada]Add regular expressions for haproxy log lines (HTTP format) and to
parse HTTP requests path.
Added tests to ensure they work as expected.
[gforcada]Create distribution.
[gforcada]