Package: logreduce
Based on success logs, logreduce highlights useful text in failed logs.
The goal is to save time in finding a failure’s root cause.On average, learning run at 2000 lines per second, and
testing run at 1300 lines per seconds.How it workslogreduce uses amodelto learn successful logs and detect novelties in
failed logs:Random words are manually removed using regular expressionThen lines are converted to a matrix of token occurrences
(usingHashingVectorizer),An unsupervised learner implements neighbor searches
(usingNearestNeighbors).CaveatsThis method doesn’t work when debug content is only included in failed logs.
To successfully detect anomalies, failed and success logs needs to be similar,
otherwise the extra informations in failed logs will be considered anomalous.For example this happens with testr where success logs only contains ‘SUCCESS’.InstallFedora:sudo dnf install -y python3-scikit-learn
git clone https://softwarefactory-project.io/r/logreduce
pushd logreduce
python3 setup.py develop --user
popdopenSUSE:sudo zypper install python3-scikit-learn
git clone https://softwarefactory-project.io/r/logreduce
pushd logreduce
python3 setup.py develop --user
popdPip:pip install --user logreduceCommand Line Interface UsageLogreduce needs abaselinefor success log training, and atargetfor the log to reduce.Logreduce prints anomalies on the console, the log files are not modified:"%(distance)f | %(log_path)s:%(line_number)d: %(log_line)s"Local file usageCompare two files or directories without building a model:$logreducedifftestr-nodepool-01/output.goodtestr-nodepool-01/output.fail0.232 | testr-nodepool-01/output.fail:0677:  File "voluptuous/schema_builder.py", line 370, in validate_mapping
0.462 | testr-nodepool-01/output.fail:0678:    raise er.MultipleInvalid(errors)
0.650 | testr-nodepool-01/output.fail:0679:  voluptuous.error.MultipleInvalid: required key not provided @ data['providers'][2]['cloud']Compare two files or directories:$logreducedirpreprod-logs//var/log/Or build a model first and run it separately:$logreducedir-trainsosreport.clfold-sosreport/good-sosreport/$logreducedir-runsosreport.clfnew-sosreport/Zuul job usageLogreduce can query Zuul build database to train a model.Extract novelty from a job logs:$logreducejobhttp://logs.openstack.org/...#Reducecomparaisontoasingleproject(e.g.fortoxjobs)$logreducejob--projectopenstack/novahttp://logs.openstack.org/...#Compareusingmanybaselines$logreducejob--count10http://logs.openstack.org/...#Includejobartifacts$logreducejob--include-pathlogs/http:/logs.openstack.org/...Or build a model first and run it separately:$logreducejob-train--jobjob_namejob_name.clf$logreducejob-runjob_name.clfhttp://logs.openstack.org/.../Journald usageLogreduce can look for anomaly in journald, comparing the last day/week/month
to the previous one:Extract novelty from last day journal:$logreducejournal--rangedayBuild a model using journal of last month and look for novelty in last week:$logreducejournal-train--rangemonthgood-journal.clf$logreducejournal-run--rangeweekgood-journal.clfFilters configurationSome content yields false positives that can be ignored through filters.
Using the–configcommand line attribute, filters can be set for
exclude_files, exclude_paths and exclude_lines. Here is an example
filters configuration file:filters:exclude_files:-"deployment-hieradata.j2.yaml"-"tempest.html"exclude_paths:-"group_vars/Compute"-"group_vars/Controller"-"group_vars/Undercloud"exclude_lines:# neutron dhcp interface-"^tap[^]*$"# IPA cookies-"^.*[Cc]ookie.*ipa_session="Python Module APILogreduce can be used as a python module for custom use-case.First you need to create a classifier object:fromlogreduceimportClassifier,Tokenizer,render_htmlclf=Classifier(# A function to normalize filename, for example to remove dates or idfilename_to_modelname=lambdafn:fn,# A function to ignore some file, for example configuration filekeep_file=lambda_:True,# A function to process lineprocess_line=Tokenizer.process)Then you train the object on baseline:clf.train(["./success-logs/"])And you test target and create a report:result=clf.process(["./failed-logs/"])withopen("report.html","w")asof:of.write(render_html(result))logreduce-testsThis package contains tests data for different type of log such as testr
or syslog. Each tests includes a pre-computed list of the anomalies in log
failures.This package also includes a command line utility to run logreduce against all
tests data and print a summary of its performance.Test formatEach tests case is composed of:A.goodfile (or directory) that holds the baselineA.failfile (or directory)Ainfo.yamlfile that describe expected output:threshold:float# set the distance threshold for the testanomalies:-optional:bool# to define minor anomalies not considered false positivelines:|# the expected lines to be highlightedTraceback...RuntimeError...EvaluateTo run the evaluation, first install logreduce-tests:git clone https://softwarefactory-project.io/r/logreduce-tests
pushd logreduce-tests
python3 setup.py develop --userlogreduce-tests expect tests directories as argument:$logreduce-teststests/testr-zuul-[0-9]*[testr-zuul-01]: 100.00% accuracy,  5.00% false-positive
[testr-zuul-02]:  80.00% accuracy,  0.00% false-positive
...
Summary:  90.00% accuracy,  2.50% false-positiveAdd –debug to display false positive and missing chunks.TODOsAdd terminal colors outputAdd progress barBetter differentiate training debug from testing debugAdd a starting log line and report writtenAdd tarball traversal in utils.files_iteratorAdd logstash filter moduleImprove tokenization testsRoadmapDiscard files that are 100% anomalousReport mean diviation instead of absolute distancesInvestigate second stage modelContributeContribution are most welcome, usegit-reviewto propose a change.
Setup your ssh keys after sign inhttps://softwarefactory-project.io/auth/loginCode style is managed withblack,
runblack logreducebefore commit to format the source file.