# A Cloud-Based Framework for Large-Scale Log Mining through Apache Spark and Elasticsearch
## 1 Introduction  
Spatial data portals (SDPs) serve the Earth science community with massive geospatial data [1]. However, as the volume and variety of the data are increasing faster than ever, they pose a great challenge for SDPs to provide reliable and quality service [2]. An emerging trend in data discovery is mining user behaviors from logs for the latent linkages between users and data [3]. Taking the National Aeronautics and Space Administration (NASA) Physical Oceanography Distributed Active Archive Center (PO.DAAC) as an example, a solution was proposed to improve oceanography data discovery and access by mining user behavior data, called the Mining and Utilizing Dataset Relevancy from Oceanographic Dataset (MUDROD) [2]. The MUDROD engine manages logs and converts them into a series of sessions and domain-knowledge-like oceanographic vocabulary linkages based on Elasticsearch [4, 5]. Elasticsearch is a component of the ELK stack and provides a solution to automatically index, search, analyze, and visualize logs with Logstash and Kibana.  
When learning useful knowledge from web usage logs, as more logs are collected, the more comprehensive a knowledge base will be populated. The faster the log analysis performs, the sooner the knowledge base keeps up with the trends in the data portal. However, the log-mining process is compute-intensive and time-intensive because it associates with complicated data preprocessing and mining algorithms such as user identification, crawler detection, session identification, and knowledge extraction (Figure 1). Session identification, e.g., requires the mining framework to differentiate a user's log from another user's log efficiently [4]. PO.DAAC provides nearly 10 years of historical logs as starting seeds for the MUDROD search engine. It takes more than five days (\(\sim\)120 h) to process all logs. In addition, the changing log volume requires on-demand computing resources. History logs are gathered once, and online logs are collected for analysis periodically. Enough computing resources are an indispensable prerequisite for large historical log analysis, while, for those logs collected on the fly, fewer computing resources are needed. Therefore, a scalable framework is desirable to optimize the log-mining process with a dynamic number of computing nodes.  
Elasticsearch can search data and ingest logs with simple operations, e.g., filtering Uniform Resource Locator (URL) with the aid of Logstash. Kibana enables users to analyze and visualize logs. As an evolving open-source project, Elasticsearch provides rich resources and Application Programming Interfaces (APIs) for customizable search functionalities. We can implement log analysis functionalities using existing resources provides by the ELK stack efficiently. However, the ELK stack has an orientation toward efficient search rather than high-performance analysis. To improve performance of the log-mining framework built on Elasticsearch, a straightforward way is to adopt parallel processing frameworks. Big data frameworks, e.g., Apache Spark and Hadoop, are used to handle big data in recent years. Spark can accelerate the data analysis process with data parallelism and provide in-memory computing capability. To efficiently mine knowledge from web usage logs and make historical logs searchable and visualizable, we proposed a cloud-based framework for large-scale log mining using Spark and Elasticsearch. The framework can efficiently reconstruct sessions from offline logs and learn knowledge from user behavior data by making full use of advanced features provided by Spark and Elasticsearch, e.g., data parallelism and full-text indexing features. In addition, logs and extracted sessions are searchable on the platform. An information flow graph describes how data are flowing in the framework, and a log partitioner is designed to further speed up the log-mining process. We utilize PO.DAAC web usage log mining to demonstrate and evaluate the proposed framework and methods.