Automatic Workload Characterization Using System Log Analysis

In a previous related work, the authors presented a framework for the dynamic derivation of analytical performance models in autonomic systems. The framework automates the process of deriving and parameterizing Queuing Network (QN) performance models even when detailed knowledge of the system characteristics and user behavior are not readily available. In this paper, we begin to explore and implement the various components of the framework, where we focus on automated workload characterization using system log analysis. In particular, we show an automated technique for generating a Customer Behavior Model Graph (CBMG) by reverse engineering high-level application workflows at the user interface level given the availability of system logs. We ran a number of experiments on the Apache OFBiz ERP system and used Logstash to parse the embedded Apache Tomcat access logs. The results show that our approach for deriving CBMGs is accurate and can be used to estimate the various system workloads. 1 Motivation and Introduction In a related paper, the authors presented a framework for the dynamic derivation of analytical performance models in autonomic systems [2]. The framework automates the process of deriving and parameterizing Queuing Network (QN) performance models even when detailed knowledge of the system characteristics and user behavior are not readily available. More details in section 4. In this paper, we begin to explore and implement the various components of the framework, where we focus on automated workload characterization using system log analysis, which is the primary function of the Workload Model Analyzer component. In particular, we show an automated technique for generating a Customer Behavior Model Graph (CBMG) by reverse engineering high-level application scenarios at the user interface level. Large application systems are characterized by workloads that consist of sessions, where users interact with the system through a sequence of requests. The idea of this paper is to analyze the logs of multi-tier systems so that sessions (sequence of interactions with the system) can be derived and modeled. One of the techniques used to model a repeated sequence of user interactions with application systems is the Customer Behavior Model Graph (CBMG) [8, 9]. CBMG is a state transition diagram used to describe the behavior of a group of users exhibiting similar behavioral patterns while interacting with the application system. The CBMG is also referred to as a User Interaction Model, which describes the nature of interaction between the user and the system. The model describes the frequency of interaction and the sequence of requests submitted to the system [10] In most cases, the sequence of interactions is dictated by the application itself as it guides the users from one step to the next. The key to accurate characterization of the workload model is to extract the most commonly used patterns, and use those to create an approximate workload model. Outliers can be characterized separately and their impact on the system performance can be analyzed as well. However, the focus of this paper is to find the repeated workflows in user sessions, and we view the outliers as ”noise” that, although has an impact on the overall system performance model, it does not represent a significant repeated behavior and, therefore, can be abstracted out of the resulting workload model. In the related paper, the authors focused on gray box modeling, where they treated the system as a “gray box,” where some information about the internal structure of the system is known, such as the number of tiers in a multitiered system, along with input and output parameters. In this paper, however, we assume the availability of system access logs and monitoring logs and we show how to produce a set of CBMGs from these logs. The CBMG is used to perform clustering at the user session level. This clustering is then used to produce the average number of visits to each application module, and consequently, the service demands per application module. We use the Elasticsearch set of tools generally referred to as the ”ELK Stack” (Elasticsearch, Logstash and Kibana). Logstash uses filters to parse log files and store their data in Elasticsearch, which can then be used to search the fields retrieved from the log files for further analysis. When deriving an analytical performance model for a computer system, it is essential to understand the system architecture (i.e., its hardware and software components and their interconnections), and the behavior of the computer system and its users during operations. For example, how users operate the software (e.g., think time and workload intensity) and how the system responds to normal operations versus operating exceptions. All these aspects contribute to the specifications and parameterization of the performance model. In this paper, we focus on characterizing the behavior of the system users, the transaction requests they submit and the handling of these requests by the software and hardware components. Input and output server logs, such as database logs, web server logs and various other softwarespecific logs can be used to characterize the workload in terms of arrival rate, throughput, processing time and the probability of transaction rejection. In particular, we automatically derive the system’s CBMGs, which model the flow of transactions from one server to another at the application module level. The rest of the paper is organized as follows. Section 2 addresses the semantic analysis of system logs and the techniques used to automate this analysis process in order to accurately characterize the workload. Section 3 describes the tools that we used to parse and analyze system logs, produce the CBMGs and the system’s workload model. Section 4 describes the CBMG derivation algorithm and our implementation approach. Section 5 shows the results of our experiments using OFBiz, JMeter and the ELK stack, and a comparison between the actual OFBiz scenario and the CBMG produced by our approach. In section 6 we present a few observations and lessons learned from our experiments. Section 7 discusses some related work. Finally, section 8 includes concluding remarks and some of our current efforts related to the automatic derivation of system performance models.