Artifact for "Demonstration-Free: Towards More Practical Log Parsing with Large Language Models'"
Creators: name: Yi, Xiao
name: Le, Van-Hoang
name: Zhang, Hongyu
Publisher: name: Zenodo
Year: 2024

description: This is the artifact for reuse and reproduction of experimental results presented in our ASE'24 paper titled "Demonstration-Free: Towards More Practical Log Parsing with Large Language Models".

SETUP

Install LogBatcher

Install LogBatcher from PyPI: pip install logbatcher

OR, Install LogBatcher from source: pip install -e .

Set your API Key in config.json

To run with docker

Download the pre-installed docker image logbatcher.tar , and run the following command:

docker load -i logbatcher.tar
docker images
docker build -t logbatcher .
docker run -it logbatcher



USAGE




Data format


LogBatcher mainly takes a raw log file (in plain text format) as input and outputs the parsed log file (in CSV format). A raw log file is a log file with each line representing a complete log.

Following the data format from LOGPAI, the data can also be a structured log file. A structured log file is a CSV file that includes at least the LineID and Content columns for parsing, with optional EventID and EventTemplate columns for evaluation.



Usage example


We provide a usage example for more convenient reuse, which is presented as follows. The usage example can be found in the file demo.py. The example provides a test on a specific dataset Apache from LOGPAI. If you want to evaluate LogBatcher on your own dataset, please replace the arguments file_name and dataset_format with your own raw log file path to load log data and the corresponding dataset format to extract the contents. Run python demo.py and find the results in the outputs/parser/test folder.



python demo.py
Parsing 2000 logs in dataset Apache...
100%|██████████████████████████████████| 2000/2000 [00:04<00:00, 420.55log/s]
parsing time: 4.756490230560303
idetified templates: 6

To evaluate the output of the usage example:

cd evaluation && python logbatcher_eval.py --config test --dataset Apache

Calculating Edit Distance....
100%|███████████████████████████████████████████████████████████| 2000/2000 [00:00<00:00, 4029110.47it/s]
Normalized_Edit_distance (NED): 1.0000, ED: 0.0000,
Grouping Accuracy calculation done. [Time taken: 0.002]
Start compute grouping accuracy
100%|███████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 2084.64it/s]
Grouping_Accuracy (GA): 1.0000, FGA: 1.0000,
Grouping Accuracy calculation done. [Time taken: 0.006]
Parsing_Accuracy (PA): 1.0000
Parsing Accuracy calculation done. [Time taken: 0.001]
100%|███████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 10677.06it/s]
PTA: 1.0000, RTA: 1.0000 FTA: 1.0000
Identify : 6, Groundtruth : 6
Template-level accuracy calculation done. [Time taken: 0.003]

Benchmarking

To benchmark on all datasets in loghub-2k or loghub-2.0, you can run the following commands:

python benchmark.py --data_type [DATATYPE] --model [MODEL] --batch_size [BATCHSIZE] --chunk_size [CHUNKSIZE] --sampling_method [SAMPLINGMETHOD]

The description of the arguments can be found in benchmark.py or below:



--data_type
  Datasets type, Options: ['2k', 'full'], default: '2k'.
--model
  the Large Lauguage model used in LogBatcher, default: 'gpt-3.5-turbo-0125'.
--batch_size
  size of a batch query, default: 10.
--chunk_size
  size of a log chunk, default: 2000.
--clustering_method
  clustering method used in the partitioning stage, Options: ['dbscan', 'meanshift', 'hierarchical'], default: 'dbscan'.
--sampling_method
  sampling method used in the batching stage, Options: ['dpp', 'similar', 'random'], default: 'dpp'.

To evaluate, run the following command



cd evaluation && python logbatcher_eval.py --config logbatcher_2k


The description of the arguments:



--config
  The folder name of the outputs, Options: ['test', 'logbatcher_2k', 'logbatcher_full']
--data_type
  Datasets type, Options: ['2k', 'full'], default: '2k'
--dataset
  To evaluate on a single dataset, default: 'null'.

CODE

The code to reproduce the experimental results in the paper is available at https://github.com/LogIntelligence/LogBatcher