Package: logai
LogAI: A Library for Log Analytics and IntelligenceTable of ContentsIntroductionInstallationGetting StartedExplore LogAI GUI PortalRun Simple Time-series Anomaly Detection ApplicationBuild Customized LogAI ApplicationsDeep-learning Anomaly Detection BenchmarkingDocumentationTechnical Report and Citing LogAIContactLicenseIntroductionLogAI is a one-stop open source library for log analytics and intelligence. LogAI supports various log analytics and log intelligence tasks such as log summarization, log clustering, log anomaly detection and more. It adopts the OpenTelemetry data model, to enable compatibility with different log management platforms. LogAI provides a unified model interface and integrates popular time-series, statistical learning and deep
learning models. Alongside this, LogAI also provides an out-of-the-box GUI toolkit for users to conduct interactive log
analysis. With LogAI, we can also easily benchmark popular ML and deep-learning algorithms for log anomaly detection
without putting in redundant effort to process the logs. We have opensourced LogAI to facilitate a wide range of
applications benefiting both academic research and industrial prototyping.The following table compares LogAI with several existing log analysis Tools, include both commercial log management platforms
like NewRelic and DataDog, or popular log analysis open source tools on Github.CoverageLogAINewRelic Log MonitoringDataDog Log Explorerlogparserloglizerdeep-loglizerlog3COpenTelemetry log data model:white_check_mark::white_check_mark::white_check_mark:Unified data loader and preprocessing:white_check_mark::white_check_mark::white_check_mark::white_check_mark::white_check_mark::white_check_mark:Auto log parsing:white_check_mark::white_check_mark::white_check_mark:Log clustering:white_check_mark::white_check_mark::white_check_mark::white_check_mark:Log anomaly detection - time-series:white_check_mark::white_check_mark::white_check_mark:Log anomaly detection - traditional ML:white_check_mark::white_check_mark:Log anomaly detection - deep Learning:white_check_mark::white_check_mark::white_check_mark:Huggingface integration:white_check_mark:GUI for result visualization:white_check_mark::white_check_mark::white_check_mark:InstallationQuick InstallYou can install LogAI core library usingpip install:# Check out LogAI code repo from Githubgitclonehttps://github.com/salesforce/logai.gitcdlogai# [Optional] Create virtual environmentpython3-mvenvvenvsourcevenv/bin/activate# Install LogAIpipinstalllogaiInstall Optional DependenciesLogAI core library is light-weight with limited dependent packages installed. Users can install optional dependencies
to enable extended functionalities of LogAI.Deep Learning Log Analysis. To conduct deep learning model related tasks and run benchmarking,
please install extra requirements bypip install "logai[deep-learning]".Enable LogAI GUI Portal*. To use LogAI GUI portal,
please install extra requirements bypip install "logai[gui]".LogAI Development. To contribute to LogAI development, build and test code changes,
please install extra requirements bypip install "logai[dev]".Complete Installation. you can install the full list of dependencies bypip install "logai[all]".Known Issues:warning: You may seeResource punkt not foundwhile using LogAI. You can downloadpunktpackage from NLTK to solve the problem.python-mnltk.downloaderpunktGetting StartedBelow we briefly introduce several ways to explore and use LogAI, including exploring LogAI GUI
portal, benchmarking deep-learning based log anomaly detection using LogAI, and building your
own log analysis application with LogAI.Explore LogAI GUI PortalYou can also start a local LogAI service and use the GUI portal to explore LogAI.# Check out LogAI code repo from Githubgitclonehttps://github.com/salesforce/logai.gitcdlogai# [Optional] Create virtual environmentpython3-mvenvvenv# create virtual environmentsourcevenv/bin/activate# activate virtual env# install LogAI and GUI dependenciespipinstall".[dev]"pipinstall".[gui]"# Start LogAI serviceexportPYTHONPATH='.'# make sure to add current root to PYTHONPATHpython3gui/application.py# Run local plotly dash server.Then open the LogAI portal viahttp://localhost:8050/orhttp://127.0.0.1:8050/in your browser:The control panel is on the left side of the page. There are three applications you can choose from:
Log Summarization, Log Clustering and Anomaly Detection.Control PanelFile Setting. You can select the log type and log files to be processed. Now LogAI supports three
public datasets: HDFS, BGL and HealthApp. For each log type we included several sample log data. After
log file selected, you can choose the attributes you want to be involved in log processing. The selected
attributes will be treated as structured log attributes.Algorithm Setting. For different applications, the algorithms options may be different.
For example, auto-log parsing algorithms are utilized for log summarization, but log clustering uses auto-parsing algorithms,
vectorization algorithms, categorical encoding and clustering algorithms. You can select an
algorithm and change the parameters in each algorithm section. After algorithm configuration, simply click "run" to
run the application.Log SummarizationLog summarization App summarize and group the raw logs by log patterns and attributes. You can click on
each log pattern and see what the pattern looks like and the dynamic values in each position. You can also
see the chart of occurrance trend of this pattern on the right side.Log ClusteringLog clustering App groups raw logs into clusters by calculating the semantic representation of each logline.
Then using clustering algorithms to generate log clusters. In this example, we choose k-mean wherek==8to
generate 8 clusters. The result is shown as a pie chart and you can click each portion of the pie chart to check
the raw logs in this cluster.Anomaly DetectionLog anomaly detection App conduct log anomaly detection tasks. Similar to log clustering, log anomaly detection
also needs to extract information from raw logs and generate representation of loglines. Depend on the type of anomaly detection,
The representation can be different.Time-series anomaly detection. If we use time-series algorithm like ETS, the raw logs will be converted
into log counter vectors by given time interval. Then ETS is performed on the generated log counter vectors and detect
anomalous timestamps on the counter vector time-series.Semantic anomaly detection. If we use unsupervised outlier detection algorithms such as One-class SVM, the raw logs will
be converted into semantic vectors and feed the One-class SVM model. Then the model will detect anomalous loglines.LogAI GUI portal is just an example to demo LogAI capabilities. We know this may not be the best way to visualize the
results and there might be bugs in how the results are displayed. We will keep working with the open source community
to improve usability of the portal. Any feedbacks and contributions are welcome :blush:.Run Simple Time-series Anomaly Detection ApplicationYou can also use LogAI in more programtic ways. LogAI supports configuration files in.jsonor.yaml.
Below is a samplelog_anomaly_detection_config.jsonconfiguration for anomaly detection application.
Make sure to setfilepathto the target log dataset file path.{"open_set_data_loader_config":{"dataset_name":"HDFS","filepath":""},"preprocessor_config":{"custom_delimiters_regex":[]},"log_parser_config":{"parsing_algorithm":"drain","parsing_algo_params":{"sim_th":0.5,"depth":5}},"feature_extractor_config":{"group_by_category":["Level"],"group_by_time":"1s"},"log_vectorizer_config":{"algo_name":"word2vec"},"categorical_encoder_config":{"name":"label_encoder"},"anomaly_detection_config":{"algo_name":"one_class_svm"}}Then to run log anomaly detection. You can simply create below python script:importjsonfromlogai.applications.application_interfacesimportWorkFlowConfigfromlogai.applications.log_anomaly_detectionimportLogAnomalyDetection# path to json configuration filejson_config="./log_anomaly_detection_config.json"# Create log anomaly detection application workflow configurationconfig=json.loads(json_config)workflow_config=WorkFlowConfig.from_dict(config)# Create LogAnomalyDetection Application for given workflow_configapp=LogAnomalyDetection(workflow_config)# Execute Appapp.execute()Then you can check anomaly detection results by callingapp.anomaly_results.For full context of this example please checkTutorial: Use Log Anomaly Detection Application.Build Customized LogAI ApplicationsYou can build your own customized log analysis applications using LogAI. Here we show two examples:Tutorial: Log Clustering Using LogAITutorial: Log Anomaly Detection Using LogAIDeep-learning Anomaly Detection BenchmarkingLogAI can be used to benchmark deep-learning anomaly detection results.
Atutorialis provided for
Anomaly Detection Benchmarking using LSTM anomaly detector for HDFS Dataset. More examples of deep-learning anomaly
detection benchmarking on different datasets and algorithms can be found inDeep Anomaly Detection Benchmarking Examples.DocumentationFor more detail about LogAI library and advanced use cases, please visitLogAI Documentation.Technical Report and Citing LogAIYou can find more details about LogAI in thetechnical report.
If you're using LogAI in your research or applications, please cite using this BibTeX:@misc{https://doi.org/10.48550/arxiv.2301.13415,
    title = {LogAI: A Library for Log Analytics and Intelligence},
    author = {Cheng, Qian and Saha, Amrita and Yang, Wenzhuo and Liu, Chenghao and Sahoo, Doyen and Hoi, Steven},
    publisher = {arXiv},
    year = {2023},
    doi = {10.48550/ARXIV.2301.13415},
    url = {https://arxiv.org/abs/2301.13415},
    copyright = {arXiv.org perpetual, non-exclusive license}
}ContactIf you have any questions, comments or suggestions,
please do not hesitate to contact us atlogai@salesforce.com.LicenseBSD 3-Clause License