# Data Preprocessing Method and API for Mining Processes from Cloud-Based Application Event Logs
## 2 Related Work and Challenges  
Important motivations for applying process mining techniques on a dataset include discovering and generating understandable process models (or maps). When analyzing processes with many different cases, much noise, and a high diversity of behaviors, which are typical of cloud-based applications, the generated models are often confusing and difficult to understand. [PERSON] and [PERSON] [8] call such process models spaghetti models and explored the iterative use of pattern abstraction to produce simpler and hierarchical models that eliminate irrelevant paths and reduce complexity. [PERSON] et al. [9] introduced another filtering method that identifies noise and infrequent behavior in processes as outliers. This filtering approach is based on identifying whether an event is likely to happen depending on a sequence of previous events, which was defined as the conditional probability between sequence of events in processes. A recent survey on user behavior analysis from clickstream data highlights different approaches based on clustering, classification, and association, but no PM approaches specifically [10]. Our own systematic literature review [11] however assessed 27 peer-reviewed papers related to the application of PM technologies to cloud processes as well as clickstream datasets, including applications to customer journey process modeling [12]. This review shows that the area of cross-organizational process mining (initially explored by [PERSON] [13; 14]), common in a cloud-based context, is still challenging. There are also common PM-related challenges arising from the evolution or drifting over time of the systems and processes being monitored, as well as of the concepts they manipulate [15]. The review also highlights specific challenges faced when preprocessing event logs collected from cloud-based applications, including clickstream data. In particular [11]:  
1. Cloud-based logs are generated from different monitoring systems, often from different organizations, leading to ID/case alignment issues especially in the presence of privacy constraints. 2. Cloud-based logs often contain hundreds of attributes that are sparsely populated, which require analysts to select/merge attributes at preprocessing time. 3. Cloud-based processes typically evolve more rapidly than conventional processes, especially in a DevOps context with daily deployments. Different types of events with attributes can then start/stop appearing in the logs at a very high rate. 4. For real-time processing, the granularity of cloud-based logs is often much finer (hundreds of clicks/events per second) than in conventional processes, which adds to the processing complexity. 5. Given that users are usually free to click anywhere on a Web application, and that the proportion of incomplete process instances is high in cloud-based applications, the readability of mined process models is low (and spaghetti-like). 6. Dealing with noisy cloud-based logs, where too many events are tracked, adds to the complexity of models. 7. Many cloud-based events have generic names and must be split according to some of their attributes into different events. 8. The monitoring and trace management tools in cloud environments frequently evolve, which in turn results in changes in the nature of the collected logs. 9. Events whose ordering or even presence is not important (for instance, _click-refresh-click-refresh_) often pollute cloud-based logs and should often be aggregated (e.g., to _click-click-refresh_ or even just _click-refresh_) in order to simplify the resulting processes. In the process mining literature, simplification of event logs is often done with a combination of abstraction [16], event/trace filtering (e.g., based on frequencies or attributes) [17], and trace clustering (e.g., K-means clustering) [18]. Although filtering techniques alone work well with structured processes, they have problems discovering less structured ones, especially in a cloud-based context [16].