Package: tailparse
tailparseMeant to mimic, sort of, how the OGlogparser for MS
Serverworked.Very much a work-in-progress atm. Goal is to make this a standalone
python executable for deployment on servers. Will probably have to
change the name to make space on package repos.Installationtailparsehas no dependencies to install. It only works with
python 3 and has only been tested in python 3.10. Presumably it would
work with any python 3.python3 setup.py install --userThis will install it in your home directory
(i.e.home/<username>/.local/bin/) and doesn't require sudo
privileges.Usagetailparse -q <sqlite-query> your.logsHelp:$ tailparse --help
usage: tailparse [-h] [-p] [-i INPUT_FORMAT] [-q QUERY] [-r MAX_ROWS] [-s SAVE_DB] [-f FILE] [logs]

Process logs as if they were SQL.

positional arguments:
  logs                  The path to the input log we're processing. If not present, will use stdin.

options:
  -h, --help            show this help message and exit
  -p, --print           Print out the requisite columns
  -i INPUT_FORMAT, --input-format INPUT_FORMAT
                        The format of the log we're processing. Defaults to 'nginx'. Options include ['nginx']
  -q QUERY, --query QUERY
                        The query to execute. Don't include any 'FROM' statement as this is added automatically. If not
                        included, make sure to include a -f/--file arugmenet
  -r MAX_ROWS, --max-rows MAX_ROWS
                        Number of max rows to print. Defaults to 20. Put 0 to print all.
  -s SAVE_DB, --save-db SAVE_DB
                        Whether to save the resulting SQLite data file. Defaults to not saving it and using ':memory:'
                        instead. If the database exists, then the log file will not be used to populate it and instead it
                        will be read from. This can be helpful if you're running a lot of queries as the log file doesn't
                        need to be re-parsed everytime.
  -f FILE, --file FILE  Execute multiple queries contained with a file. Can be used in place of -q/--queryCaching the database on-diskIt can help to write the SQL database to disk instead of reading from
memory. This can be done with the-sor--save-dbarguments:$ time tailparse -s tmp.sqlite3.db -q "SELECT COUNT(*) FROM logs" sample.logs
COUNT(*)
100090

real    0m1.026s
user    0m0.835s
sys     0m0.117s
$ time tailparse -s tmp.sqlite3.db -q "SELECT COUNT(*) FROM logs" sample.logs
COUNT(*)
100090

real    0m0.648s
user    0m0.533s
sys     0m0.095s

# and without any caching
$ time tailparse -q "SELECT COUNT(*) FROM logs" sample.logs
COUNT(*)
100090

real    0m0.910s
user    0m0.866s
sys     0m0.045sNote that if you do this,tailparsewill not attempt to rewrite the databaseRunning TestsTo run tests, at the top level run the following:./test.shTo get coverage reports, which requires thecoveragemodule, you can run:./test.sh -hWhile testing relies entirely onunittest, which is built-in,coverageandmypyare used to do coverage reports and type
checking, respectively../test.shwill check for these installs at
runtime.ContributingFeel free to make a PR!Note that this library is trying very hard to avoid any dependencies
and stay core-python-only. If there is a strong reason to include one,
please include a reason why.blackis used for all code formatting. Testing will be required for
any contributions.Todoswriteproper
testssplit thelogparser.pyfile up into separate chunkstailparse.execute.execute_query: write integration tests
against this and check for the output string --all for 'nginx'includesample.logviashuf -n N input > outputSELECT * FROM logs LIMIT 1SELECT * FROM logs LIMIT 2SELECT * FROM logs: stops at 20 by defaultSELECT * FROM logs, w/max_rows=0: does not stop at 20SELECT *: complains before it executesw/query_file: write to/tmpand clean up afterw/save_db: write to/tmpand clean up afterw/print_columns: asTrue, should print just columnstailparse.print_output.print_output: write unit tests
against this and check for output stringwrite testing shell script -- should fail ifmypyfails or
ifcoverageandmypyaren't installedupdate the README to use current version screenshotsadd--versionargumentwrite proper contribution guides, especially for new log formatswrite properMakefile to make it easy for people to see whats
going onremove pandas dependency and only use pure pythonsupport writing to disk for the sqlite3 database, not just to
memorysupport other formats, not just NginxapachemorganAbility to process multiple SQL commands in a text file,
separated by lineinfer the table (e.g.FROM logs) so I don't have to specify it
everytimeLicenseGPL-v3