# RCM: A Remote Cache Management Framework for Spark
## 1 Introduction  
With the further development of the Internet, information has become the inevitable choice to cater to the change of the times. Nowadays, massive data generated by users have been stored on the Internet. In order to cope with these massive data, the complexity of big data platforms is increasing [1]. Modern big data platform systems are divided into a computing layer and database storage layer. The computing layer, such as Mapreduce, Spark and Flink, is responsible for establishing multi-nodes service and splitting a job into multiple tasks for parallel computing [2]. Facing diversified data, the database storage layer often uses different database software as storage components to build heterogeneous storage systems. For example, putting a document into the MongoDB database, putting structured data into the MySql database, putting large data blocks into the HDFS database, and putting small data into the HBase database [3]. As shown in Figure 1, users establish tasks by accessing the computing layer. When a task accesses data, it creates a link to the database through the routing network. Because the link to databases has a large cost, the important result of querying will be stored in the application cache layer. When users query the same data again, the task will directly come from the application cache layer. By saving the result in a cache, a large number of repeated links to databases are avoided [4]. The application cache layer is a storage area built by memory and IO speed, which is faster than the database link. Common cache architectures include local cache and remote cache. The local cache stores results in the node, which can respond to queries very quickly. The downside of this is that the local cache cannot share data between different nodes,and it has limited memory resources. This architecture is difficult to scale out. In order to overcome these disadvantages, the remote cache stores the data in the memory of remote servers. By managing clusters of remote cache servers, this cache layer is easy to scale out. At the same time, it is easier to share the architecture of a remote cache than that of a local cache. Remote cache softwares, such as Redis, Apache Ignite and Memcached, provide abundant interfaces, which are used by many applications [5]. However, these remote cache solutions lack a remote cache management strategy for big data platforms. [PERSON] et al. [6] investigated and mentioned that Spark would replace old data in a Least Recently Used (LRU) method to release more resources. However, there is no mechanism specifically for caching RDD in Spark, and the important costs are not been taken into consideration with LRU. [PERSON] et al. [6] designed a strategy to dynamically tune cache resources. [PERSON] et al. [7] proposed a strategy of expelling caches based on the requirements of a task. [PERSON] [8] designed a hierarchical storage strategy for hot data and cold data. These works confirm that a cache management optimization strategy is an effective way to reduce job execution time. Based on the above conclusion, we conducted in-depth research on cache weight generation algorithms, cache replacement algorithms and cache placement algorithms. Some challenges are summarized: Firstly, cache weight is usually measured by the frequency of accessing. Frequently accessed data are preferentially retained in memory [9]. However, for the platform of big data, other costs, such as the response time of the query database and the data size, also influence the cache hit rate by different degrees. Secondly, the phenomenon of time loss is more obvious under modern big data platforms. In a period of time, the hit rate of the hot spot is high. As time went on, the hit rate of these data reduces. The characteristics of time loss need to be formalized. As a result, we need a brand-new algorithm to generate weight. Thirdly, the cache placement algorithm chooses the best server to store different cache data. This algorithm is similar to the virtual machine placement algorithm.