# A Cloud-Based Framework for Large-Scale Log Mining through Apache Spark and Elasticsearch
## 2 Related Work  
As an important complementary component of the data discovery engine, log mining discovers implicit knowledge from user behavior data by learning meaningful patterns, trends, and linkages, e.g., Google explores user logs to catch what the world is searching and publishes reports of trends in  
Figure 1: The components of log mining in the Mining and Utilizing Dataset Relevancy from Oceanographic Dataset (MUDROD) search engine. a year ([[https://trends.google.com/trends/](https://trends.google.com/trends/)]([https://trends.google.com/trends/](https://trends.google.com/trends/))), whereby it was found that the two most popular phrases searched in the Unites States were \"world cup\" and \"Hurricane Florence\" in 2018. [PERSON] et al. proposed the RankSVM algorithm to improve the ranking performance of a search engine by tracking user behavior on the ranking list. The assumption is that if a user clicked a link with a lower rank, the link is more relevant to the user's search than other links with a higher rank [6]. In the GLScience domain, researchers also pay close attention to log mining. Logs were collected and analyzed to assist search engines to understand queries [7], rank data intelligently [8], analyze a traveler's moving patterns for route recommendation [9], discover semantic relationships among geospatial vocabularies for domain ontology population [5], discover interesting spatiotemporal theme patterns [10], etc. Sessionization is the process of splitting a collection of logs into multiple sessions which record a user's behavior on a website during a time period, serving as the first and fundamental step of log mining. However, it is quite a challenge to reconstruct sessions when the user identifier is missing in the raw logs. Multiple methods were proposed to group logs into sessions relying on time interval, referrer, or both [4; 11; 12]. In previous research, we proposed a method of reconstructing sessions from multiple logs to build a semantic knowledge base [4; 5]. The knowledge extracted from logs is valuable, but the speed of the log-mining process needs improvement. For instance, for the small vertical search portal PO.DAAC, the log-mining framework built on Elasticsearch took more the one hour to finish monthly log analysis. Thus, we propose a cloud-based framework to speed up log mining with Apache Spark and Elasticsearch. Logs are continually produced in different formats and rates, and can uncover useful information through proper and effective analysis. Cloud computing and parallel computation frameworks were introduced to support log analysis, as they can manage and analyze data of large size with a high production rate [13; 14] The distributed computing paradigm can provide on-demand storage and computing resources for log mining [15; 16]. The parallel computation frameworks, e.g., Apache Spark and Apache MapReduce, improve data analysis performance through data parallelism. [PERSON] et al. proposed a unified cloud platform with batch analysis and in-memory computing capacity by combining Hadoop and Spark [16]. [PERSON] et al. applied Hadoop for large-scale log analysis to efficiently detect abnormal traffic from high-volume data [17]. Given the significance of log analysis, several open-source and commercial solutions were designed for log collection, storage, search, analysis, and visualization. The ELK stack, i.e., Elasticsearch, Logstash, and Kibana, automatically collects, indexes, aggregates, and visualizes logs [18]. [PERSON] et al. efficiently geo-identified the website user traffic through logs using ELK stack [19]. [PERSON] et al. used the Elasticsearch ecosystem to monitor the Infrastructure as a Service (IaaS) and scientific application on the cloud, keeping track of usage and dynamic allocation of resources [20]. Both Spark and Elasticsearch have advantages and disadvantages for log mining. Spark is capable of handling big data, but it is not designed for log analysis.