# Optimization of the Join between Large Tables in the Spark Distributed Framework
## 1 Introduction  
With the rapid development of the Internet in recent years, the era of big data has arrived. After years of development, a large number of new high-performance technologies have emerged in the field of big data, such as Apache Spark [1; 2; 3] and Apache Flink [4], which are stronger than MapReduce [5; 6] in terms of query and computation performance and which have become powerful tools for big data acquisition, storage, analysis and presentation. Big data analysis technology plays a key role in various industries. [PERSON] et al. [7; 8] studied the importance of big data analysis technology in enterprises.  
Spark is a fast, universal, scalable and highly available big data analysis search engine developed based on Scala. It has upgraded its performance based on the MapReduce model. Developers can deploy Spark on a large number of servers to form clusters that efficiently process data. The core technology of Spark is the use of resilient distributed datasets (RDD) [9]. The data are distributed in the form of RDD on each server for management, to achieve data parallelization and distributed processing. During the data repartitioning process of the Spark task, if data are moved across nodes, Shuffle is generated, as shown in Figure 1. Shuffle is a bridge between Map and Reduce. It corresponds the Map output to the Reduce input and involves serialization and deserialization, cross-node network I/O and disk read/write I/O. If a complex service logic has Shuffle, the next stage can be executed only after the previous stage produces a result. In the mass data Join task of distributed architecture, the data interaction between servers will inevitably generate Shuffle, which means a large number of serialization-deserializations, cross-node network I/Os and disk read and write I/Os.  
Shuffle consists of Shuffle write and Shuffle read phases, as shown in Figure 2. During Shuffle, a large amount of intermediate data is migrated to disks for a long time, and a large amount of network I/O is generated, affecting the overall performance of the Spark job.  
When optimizing the Spark performance, we should not only pay attention to the execution time of tasks but should also pay attention to the network IO and disk read and write. The proper optimization is not only to reduce the uptime but also to reduce the network I/O load, disk usage, and so on. In actual development, the number of Spark tasks ranges from as few as 100 to as many as thousands. In this case, performance optimization of key tasks is extremely important. Proper performance optimization can ensure running efficiency and save resources, and helps to avoid negative effects caused by excessive operation data.  
In order to solve the problems of a long running time, excessive network IO load and high disk occupancy of Spark join tasks between large tables, this paper proposes a lightweight distributed data filtering model using RoaringBitmap [10] to combine broadcast variables and accumulators when the data in two tables are not completely matched. This optimization is theoretically analyzed and experimentally verified. The implementation results show that this method effectively reduces the running time of Spark join and effectively reduces the data transfer and disk read and write in the Shuffle phase. The overall performance of Spark Join tasks is improved.