# Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province

Siwei Fu, Guifen Chen, Shan Zhao, Enze Xiao

► To cite this version:

Siwei Fu, Guifen Chen, Shan Zhao, Enze Xiao. Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province. 11th International Conference on Computer and Computing Technologies in Agriculture (CCTA), Aug 2017, Jilin, China. pp.1-12, 10.1007/978-3-030-06179-1_1 . hal-02111532

HAL Id: hal-02111532

https://inria.hal.science/hal-02111532

Submitted on 26 Apr 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Research and Application of Spark Platform on Big Data
Processing in Intelligent Agriculture of Jilin Province

Siwei Fu, Guifen Chen (✉), Shan Zhao, and Enze Xiao

Jilin Agricultural University, Changchun 130118, China
465136727@qq.com, guifchen@163.com

**Abstract.** Aiming at the demand of real-time massive data processing of Intelligent Agriculture in Jilin Province, this paper studies the big data processing of Intelligent Agriculture in Jilin Province based on Spark platform by acquiring real-time data through monitoring platform. This study first conducted the performance comparison experiment of Hadoop and Spark data processing platform, then used the Spark distributed cluster computing platform, real-time processing the big data of monitoring area. The experimental results show that the Spark platform speeds up 11.4 times faster than the Hadoop platform in the case of 100 million data sizes; and based on the Spark platform for real-time processing of big data intelligent agricultural monitoring network, not only provides memory calculations to reduce IO overhead, but also the results are faster and more accurate. The research results provide strong support for the implementation of precision agriculture technology in intelligent agriculture.

**Keywords:** Spark; big data processing; MapReduce; intelligent agriculture in Jilin province

# 1 Introduction

China is a large agricultural country, facing the ever changing agricultural production technology, Especially the introduction of remote sensing, geographic information technology, precision agriculture and expert system in the field of farmland management. This condition leads to the original variety of agricultural ecosystems turning more complex. Among them, the agricultural data gradually evolved from single quantitative transformation into qualitative transformation. The characteristics of data are massive, dynamic, spatio-temporal, etc, accompanied by structural changes. Thus, if only relying on the existing data mining technology and relational database, it has been unable to meet the current data storage and data analysis needs. It is particular important how to effectively analyze and deal with massive data, and according to the needs of the results back to the user's hands is particular important. One of the starting points in the study of big agricultural data is how to dig out valuable knowledge from complicated agricultural data. Therefore, it is urgent to use a method to classify and extract large amounts of data to find out the correlation and underlying pattern between them[1], establish the platform for analysis and preprocessing data, monitor agricultural production process, and combine with data mining technology to obtain real-time information, formulate appropriate coordination measures, and finally realize the precise operation of the farmland management area. Improving the quality of agricultural products under the premise of sustainable development provides a good development environment and material basis for other areas and even the whole national economic development[2].

Aiming at demand for real-time processing of massive intelligent agricultural data, this study firstly accesses the soil air temperature and humidity data from intelligent monitoring network of the National Spark Program "
Integration and demonstration of corn precise operation technology based on Internet of things " demonstration area at Nong'an county in real time through the intelligent agricultural monitoring platform, proposes the Spark distributed cluster computing platform. Secondly, conducted the performance comparison experiment of Hadoop and Spark data processing platform, then used the Spark distributed cluster computing platform with extended MapReduce computational models, real-time processing the big data of monitoring area. Spark distributing cluster computing platform has efficient supported for multiple computing modes. Spark with the main characteristics of computing in memory can be applied to a wide variety of distributed platform scenarios, including batch processing, iterative algorithms, interactive queries, stream processing, and so on[3,4]. These different calculations are supported by a unified framework. Spark allows us to integrate various processing processes simply and at a low cost, combined Spark streaming, SQL, MLlib, Graph X and other modules and data mining technology to make appropriate analysis of data processing[5], it provides the basis for the analysis of mass data in the future. The research results provide strong support for the processing of agricultural big data and the implementation of precision agriculture technology in intelligent agriculture. Thus, it contributes to the sustainable development of agriculture, and realize the modern development of intelligent agriculture.

# 2 Data Sources and Research Methods

## 2.1 Data Sources

The study area is Nong'an County of Jilin Province, located in the hinterland of Songliao plain, and it is the central part of Jilin Province, attached to the city of Changchun, located 60 kilometers northwest of Changchun City, east longitude 124 degrees 31'~125 degrees 45', north latitude 43 degrees 55'~44 degrees 55', the south is adjacent to the suburbs of Changchun, the east borders on Dehui, northeast across the river and to Fuyu, north of the former Guoer Ross Mongolian Autonomous County, west of Changling, southwest border with Gongzhuling City, County 114.7 kilometers long from north to south, east and west 97.7 km wide [6], the total area is 5400 square kilometers. In 2013, the county planting area of agricultural land reached 366,000 hectares, the coverage rate of fine varieties is above 98%, it is one of the important commodity grain bases in Jilin Province. Nong'an County annual average temperature of 4.7 degrees, frost free period of 145 days, rainfall of 507.7 mm, the effective accumulated temperature of 2800 degrees. Flat terrain, four distinct seasons, is a temperate continental climate. The soil in Nongan county is divided into 10 soil types and 20 sub types, 50 genera and 111 species of soil. The zonal soil is black soil, chernozem[7]. This data comes from the seven real-time data monitoring stations located in Nong'an County Kai'an Town, Hualong Town Chenjiadian Village, real-time monitoring network for intelligent agriculture, monitoring and collection of crop production environment information using wireless sensor networks, thus to realize the real-time processing and analysis of the big data of the intelligent agriculture.

## 2.2 Research Methods

### 2.2.1 Spark cluster computing platform

Spark was originally born at the APM laboratory at the University of Berkeley. It is a fast, general purpose engine that can be used in large-scale data processing[8], today is one of the top open source projects under the Apache Software Foundation. Just as its name Spark, such as lightning fast cluster computing platform, the original design goal of Spark was to make data analysis faster—Not only is it fast, but it also has to be able to write programs quickly
and easily. In order to make the program run faster, Spark provides memory computing, reducing the IO overhead in iterative computation. In order to make the program run easier, Spark is written in a concise, elegant Scala language[9]. Scala provides an interactive programming experience. From enterprise, medical treatment, transportation to retail trade, the big data solutions offered by Spark are pushing ahead with the insights of business that have never been seen before, and thus accelerated decision-making.

The Spark project contains more than one tightly integrated component, the core of Spark is a computing engine that consists of scheduling, distributing, and monitoring applications that are composed of many computing tasks, running on multiple work machines, or a computing cluster. As shown in Figure 1, Spark is a large and unified software stack, including Spark SQL, Spark Streaming, MLlib, GraphX, Spark Core and Independent scheduler, YARN, Mesos Modules[10]. Spark Core implements the basic functions of Spark, including task scheduling, memory management, error recovery, storage system interaction and other modules[11]; Spark SQL is a program package that Spark used to manipulate structured data; Spark Streaming is a component that Spark provides for streaming computations of real-time data[12]; MLlib provides a variety of machine learning algorithms, all of which are designed to be easily scalable on a cluster[13]; GraphX is a library for operating diagrams, which can be computed in parallel[14].

<figure><img src="image_2.png" /><figcaption>Fig.1. Spark software stack</figcaption></figure>

In a distributed environment, the Spark cluster uses a master / slave architecture. In a Spark cluster, one node is responsible for central coordination, and each distributed work node is scheduled. The central coordination node is called the drive node, and the corresponding work node is called the actuator node. Drive nodes can communicate with a large number of actuator nodes, and they also operate as independent Java processes [15]. The drive node, together with all of the actuator nodes, is called a Spark application, as shown in Figure 2:
<figure><img src="image_3.png" /><figcaption>Fig.2. Calculation framework of Spark cluster</figcaption></figure>

The Spark relies on the cluster manager to start the executor node, but in some special cases it also depends on the cluster manager to start the drive node. The cluster manager is a pluggable component in Spark[16]. In this way, in addition to Spark's own standalone cluster manager, Spark can also run on other external cluster managers, such as YARN and Mesos[17].

### 2.2.2 The characteristics and advantages of Spark platform implementation

Although Hadoop has become the defacto standard of big data, there are still many defects in its MapReduce distributed computing model. Spark draws lessons from the advantages of Hadoop and MapReduce, and solves the problems faced by MapReduce. As shown in Figure 3, comparing the execution flow of Hadoop and Spark can be seen. The biggest feature of Spark is the introduction of the concept of an elastic distributed data set (Resilient Distributed Dataset,RDD)[18], this allows Spark to cache data set in each node memory in cluster computing, eliminating the need to load multiple times into memory and disk storage[19], and greatly speed up the processing speed.

<figure><img src="image_4.png" /><figcaption>Fig.3. Comparison of Hadoop and Spark in execution flow</figcaption></figure>
About the advantages of Spark and Hadoop in executing engine, technology stack and so on, As shown in Table 1[20,21,22].

<table><caption>**Table 1.** Comparison table of Hadoop and Spark performance</caption><thead><tr><th>Type</th><th>Spark</th><th>Hadoop</th></tr></thead><tbody><tr><td>Execution engine</td><td>DAG task scheduling execution mechanism</td><td>MapReduce iteration execution mechanism</td></tr><tr><td>Programming language</td><td>Scala, Java, Python</td><td>Java, Python, C/C++, Ruby</td></tr><tr><td>technology stack</td><td>Spark streaming, SQL, MLlib, GraphX</td><td>HDFS, MapReduce, HBase, Hive</td></tr><tr><td>Operation mode</td><td>Independent cluster model</td><td>Distributed memory computing</td></tr><tr><td>Calculation model</td><td>Belong to MapReduce, but not limited to Map and Reduce, multiple data set operation types</td><td>Only Map and Reduce two operations</td></tr><tr><td>IO spending</td><td>Small, memory calculation</td><td>large, disk read write</td></tr><tr><td>Implementation of fault tolerance</td><td>RDD data storage model</td><td>Data replication</td></tr><tr><td>Programming code (code quantity)</td><td>Real-time interactive programming (1)</td><td>Traditional programming (3-6)</td></tr><tr><td>Hardware requirements</td><td>Requirements for memory and CPU</td><td>Cheap, heterogeneous</td></tr><tr><td>Suitable application scenarios</td><td>(1) Data mining with iterative operations<br>(2) Real time and fast calculation<br>(3) Machine learning operation</td><td>Delay is too high, only for offline, batch application scenarios</td></tr></tbody></table>

# 3 Experimental results and analysis

## 3.1 Comparison of efficiency test between Spark and Hadoop

In performance, the efficiency test compares the time differences between Hadoop and Spark processing large amounts of data when performing logical regressions. This experiment from 2015-2017 intelligent monitoring network in Nong'an County 230 million soil, air temperature and humidity data in the selected 5 groups, respectively 10 thousand, 100 thousand, 1 million, 10 million, 100 million data for execution logistic regression time contrast, the efficiency test results are shown in Table 2, Figure 4.

<table><caption>**Table 2.** Comparison of run time between Hadoop and Spark</caption><thead><tr><th>Computing platform</th><th>10 thousand (1W)</th><th>100 thousand (10W)</th><th>1 million (100W)</th><th>10 million (1000W)</th><th>100 million (10000W)</th></tr></thead><tbody><tr><td>Hadoop</td><td>8</td><td>17</td><td>27</td><td>255</td><td>1755</td></tr><tr><td>Spark</td><td>10</td><td>11</td><td>15</td><td>27</td><td>154</td></tr></tbody></table>

The number in Table 2 is the run time, units are seconds. In Figure 4, “W” stands for “ten thousand bars”.
<figure><img src="image_5.png" /><figcaption>Fig.4. Comparison of Hadoop and Spark in efficiency test</figcaption></figure>

## 3.2 Real time processing of big data in Intelligent Agriculture

### (1)Build a stream processing framework for Spark platform

The big data analysis platform is designed to have tremendous capabilities and flexibility to meet all these requirements. The major types of processing used in big data analysis are batch processing, stream processing, and iterative processing. Therefore, we need such a platform to store such huge distributed data and perform all of these types of analysis.

Before implementing the Spark ecosystem, you need to configure the HDFS distributed file system, build Block data blocks, Namenode master nodes and Datanode slave nodes; based on the HDFS distributed processing model, create HBase database to achieve high performance, real-time read and write, column storage, scalable functions; with the Thrift software framework, we build seamless and efficient service with Java, Python, C/C++, Ruby and other programming languages.

For the intelligent agriculture monitoring system in Jilin, which requires millisecond real-time response, the processing data is characterized by online, small, dynamic, relative to off-line data processing, because of the high time requirements, it is more suitable for dealing with small amounts of data and running relatively simple algorithms; for the high fault tolerance and high reliability requirement of the system, Spark uses record update to create a record RDD transform sequence, in order to facilitate the recovery of file partitions.

According to the above requirements analysis, the system uses the Spark Streaming stream computing framework with good fault tolerance and easy combination with machine learning and graph computing in the Jilin intelligent agriculture monitoring platform, real-time reception, calculation and delivery of data streams using Spark Streaming. Based on the MLib machine learning framework, combining the improved particle swarm optimization and the limit learning machine ELM, batch processing the data for the training model; transferred processing data to the distributed file system HDFS and HBase database to store calls; the final results are fed back to the Jilin provincial wisdom agriculture monitoring platform in the form of analytical charts, as shown in Figure 5.
<figure><img src="image_6.png" /><figcaption>Fig. 5. Stream processing framework of intelligent agriculture monitoring platform in Jilin province</figcaption></figure>

## (2) Collecting intelligent monitoring network data

Based on the data comes from seven intelligent monitoring network of the National Spark Program "Integration and demonstration of corn precise operation technology based on Internet of things " demonstration area at Nong'an county, collected from May 1, 2015 to December 31, 2016 crop different dimensions (0-20cm, 20-40cm, 40-60cm, 60-80cm) soil temperature and humidity data(as shown in Table 3), and import the data into the HBase system.

<table><caption>Table 3. Soil moisture and temperature data of intelligent agricultural monitoring network in Jilin province</caption><thead><tr><th rowspan="3">Time</th><th colspan="4">Water content</th><th colspan="4">Soil temperature (°C)</th></tr><tr><th>0~</th><th>20~</th><th>40~</th><th>60~</th><th>0~</th><th>20~</th><th>40~</th><th>60~</th></tr><tr><th>20cm</th><th>40cm</th><th>60cm</th><th>80cm</th><th>20cm</th><th>40cm</th><th>60cm</th><th>80cm</th></tr></thead><tbody><tr><td>2015.5</td><td>8.75%</td><td>15.31%</td><td>19.47%</td><td>17.43%</td><td>14.51</td><td>12.63</td><td>11.18</td><td>10.11</td></tr><tr><td>2015.6</td><td>20.62</td><td>20.62%</td><td>29.38%</td><td>28.85%</td><td>20.46</td><td>18.55</td><td>17.06</td><td>15.94</td></tr><tr><td>2015.7</td><td>12.04</td><td>20.59%</td><td>30.73%</td><td>29.81%</td><td>24.67</td><td>23.07</td><td>21.75</td><td>20.70</td></tr><tr><td>2015.8</td><td>14.16</td><td>20.91%</td><td>31.41%</td><td>30.27%</td><td>23.79</td><td>23.31</td><td>22.79</td><td>22.24</td></tr><tr><td>2015.9</td><td>20.69</td><td>20.69%</td><td>30.09%</td><td>29.43%</td><td>19.02</td><td>19.34</td><td>19.47</td><td>19.41</td></tr><tr><td>2015.1</td><td>12.61</td><td>19.72%</td><td>28.66%</td><td>28.13%</td><td>9.87</td><td>11.36</td><td>12.44</td><td>13.12</td></tr><tr><td>2015.1</td><td>5.69%</td><td>15.78%</td><td>25.63%</td><td>26.05%</td><td>2.22</td><td>4.33</td><td>5.93</td><td>7.04</td></tr><tr><td>2015.1</td><td>3.51%</td><td>9.35%</td><td>11.63%</td><td>21.92%</td><td>-2.51</td><td>-0.30</td><td>1.38</td><td>2.48</td></tr><tr><td>2016.1</td><td>2.70%</td><td>7.47%</td><td>6.24%</td><td>9.91%</td><td>-8.06</td><td>-5.20</td><td>-2.70</td><td>-0.99</td></tr><tr><td>2016.2</td><td>4.70%</td><td>7.76%</td><td>6.15%</td><td>8.50%</td><td>-6.35</td><td>-5.31</td><td>-4.14</td><td>-3.17</td></tr><tr><td>2016.3</td><td>15.33</td><td>13.83%</td><td>11.73%</td><td>11.12%</td><td>-0.75</td><td>-1.04</td><td>-0.98</td><td>-0.80</td></tr><tr><td>2016.4</td><td>15.85</td><td>20.74%</td><td>25.51%</td><td>19.71%</td><td>3.96</td><td>1.39</td><td>0.09</td><td>-0.02</td></tr><tr><td>2016.5</td><td>15.05</td><td>20.98%</td><td>30.96%</td><td>28.80%</td><td>12.75</td><td>10.86</td><td>7.98</td><td>8.36</td></tr><tr><td>2016.6</td><td>14.10</td><td>20.92%</td><td>31.23%</td><td>29.75%</td><td>19.73</td><td>18.10</td><td>16.80</td><td>15.81</td></tr><tr><td>2016.7</td><td>13.05</td><td>20.60%</td><td>31.57%</td><td>30.33%</td><td>24.08</td><td>22.73</td><td>21.58</td><td>20.68</td></tr><tr><td>2016.8</td><td>13.91</td><td>22.07%</td><td>33.36%</td><td>31.36%</td><td>24.73</td><td>24.25</td><td>23.67</td><td>23.08</td></tr><tr><td>2016.9</td><td>18.43</td><td>23.13%</td><td>34.19%</td><td>31.66%</td><td>18.82</td><td>19.29</td><td>19.61</td><td>19.74</td></tr><tr><td>2016.1</td><td>16.76</td><td>21.51%</td><td>31.16%</td><td>29.17%</td><td>9.77</td><td>11.95</td><td>12.65</td><td>13.56</td></tr><tr><td>2016.1</td><td>9.18%</td><td>18.21%</td><td>27.27%</td><td>26.54%</td><td>1.83</td><td>3.79</td><td>5.51</td><td>6.78</td></tr><tr><td>2016.1</td><td>7.35%</td><td>9.55%</td><td>14.36%</td><td>21.40%</td><td>-2.12</td><td>0.18</td><td>1.88</td><td>3.03</td></tr></tbody></table>
(3)Processing of Soil Moisture Monitoring Data

According to Table 3 data,soil moisture data in different depths were
obtained in real time,then to dynamic monitoring and multi angle data processing
analysis.The processing results are shown in Figure 6.

<figure><img src="image_7.png" /><figcaption>**Fig.6.** Analysis of massive soil temperature and humidity data of intelligent
agricultural monitoring network in Jilin province</figcaption></figure>

A.Moisture trend of 0 ~ 80 cm soil in 2015; B.Temperature trend of 0 ~ 80 cm soil in 2015;
C.Moisture trend of 0 ~ 80 cm soil in 2016; D.Temperature trend of 0 ~ 80 cm soil in 2016;
E.Moisture trend comparison of 0 ~ 20 cm soil between 2015 and 2016; F.Temperature trend
comparison of 0 ~ 20 cm soil between 2015 and 2016; G.Moisture trend comparison of 40~60
cm soil between 2015 and 2016; H.Temperature trend comparison of 40~60 cm soil between
2015 and 2016.

3.3 Results and analysis

(1)Comparing the efficiency test of Hadoop and Spark platform

By comparing the efficiency test of Hadoop and Spark platform, the
experimental results are obtained(table 2 and figure 4):When the amount of
monitoring data is 10 thousand,Hadoop running time is lower than Spark
platform,at this point, the Hadoop is processing faster than the Spark.When the
amount of data reaches 100 thousand or more, Spark takes advantage of its fast
memory, computing, and distributed frameworks, In time, it is much better than
Hadoop's multiple iteration algorithm. When the test content reaches 100
million, the Hadoop platform is 11.4 times more expensive than Spark. At the
same time, it can be seen by the contrast chart: Under the Spark distributed
framework, as the amount of data increases, the time difference between the time
spent and the total time is getting smaller and smaller, reflects the stability and
reliability of the Spark platform.

(2)Real-time monitoring and processing of soil moisture data

As can be seen from Figure 6, in 2015 and 2016 in May to December year
on year, the change of soil temperature at different levels is not obvious. But the
soil moisture change is obvious at different levels; soil moisture content was
lowest in late seedling stage (at the end of June), and it was the most in early
mature (September 4th); soil temperature was lowest in the middle of seedling
(June 10th), and highest in late maturing (at the end of September). At the same
time, the trend of moisture change of 20cm and 60cm in 2015 was compared with
that in 2016, affected by climate factors such as rainfall and light in different
years, there were some differences, but the soil temperature did not change
significantly in previous years. The analysis results show that the intelligent
agriculture monitoring network based on Spark platform can deal with soil
moisture data in real-time and effectively. It provides support for precision
agricultural production such as timely sowing of crops and water-saving
irrigation and so on.

4 Conclusion and Prospect

Through accessing the soil air temperature and humidity data from
intelligent monitoring network of the National Spark Program " Integration and
demonstration of corn precise operation technology based on Internet of things "
demonstration area at Nong'an county in real time through the intelligent
agricultural monitoring platform, research on Spark platform for big data
processing of Intelligent Agriculture in Jilin province found that:

(1)Spark and Hadoop platform efficiency comparison test results show that,the Spark platform has the advantage of reducing IO overhead with its memory computing,Hadoop is more suitable for dealing with real time big data of intelligent agriculture monitoring network in Jilin province.

(2)The agricultural big data processing method of intelligent agriculture in
Jilin Province based on Spark platform,using the machine learning algorithm
characterized by dynamic and rapid expansion,combining the Spark streaming
flow calculation framework,able to real-time analyze continuous and rapid
changes in the massive data.Moreover, the calculation results are faster and more
accurate.The implementation of precision agriculture and the wisdom of
agricultural big data processing have a certain role in promoting.

(3)For this study,the problem of the data quantity is small, but the
monitoring network real-time data processing needs large, in the future, we will
further adopt the Spark platform to combine the better clustering algorithm, make
the sensor transmission data to be timely and effective treatment,utilize the
advantages of large data processing,combine soil moisture content with big data
information,constructing perfect intelligent monitoring system of agriculture.

Acknowledgments

This work was funded by the China Spark Program.2015GA660004.
"Integration and demonstration of corn precise operation technology based on
Internet of things".
References

1. Cheng Xueqi, Jin Xiaolong, Wang Yuanzhuo, Guo Jiafeng, Zhang Tieying, Li Guojie, J.:Large Data System and Analysis Technology .Journal of Software(09),1889-1908(2014)

2. Jorge L.Reyes-Ortiz1, Luca Oneto2,Davide Anguita,J.: Big Data Analytics in the Cloud:Spark on Hadoop vs MPI/OpenMP on Beowulf.ELSEVIER(53),121-130(2015)

3. Yang Feng, Wu Huarui, Zhu Huaji, Zhang Haihui,Sun Xiang,J.:Based on Hadoop's massive agricultural data resource management platform.Computer Engineering (12),242-244(2011)

4. Shyam R,Bharathi Ganesh HB, Sachin Kumar S,Prabaharan Poornachandran,Soman K P.J.:Apache Spark a Big Data Analytics Platform for Smart Grid.ELSEVIER(21),171-178(2015)

5. Qi Rongzhi, Wang Zhijian, Huang Yihua, Li Shuiyan,J.:Based on Spark's parallel combination test case set generation method.Journal of Computer Science,1-18(2017)

6. Lu Jian, Chen Guifen, Meng Ying, Chen Hang,J.: Research and System Realization of Farmland Environment Simulation Monitoring Based on 3D GIS. Chinese Journal of Agricultural Science and Technology(03),50-55(2017)

7. Czerwinski D,M.:Digital Filter Implementation in Hadoop Data Mining System.Computer Networks,410-420(2015)

8. He Q, Wang H, Zhuang F, Shang T,Shi Z,J.:Parallel sampling from big data with uncertainty distribution. Fuzzy Sets & Systems,117-133(2015)

9. Yang Zhiwei, Zheng Quan, Wang Song,Yang Jian,Zhou Lele,J.:Adaptive task scheduling strategy under heterogeneous Spark cluster .Computer Engineering(01),31-35(2016)

10. Chen GF,Dong W,Jiang J,Wang GW,J.:Variable-Rata Fertilization Decision-Making System Based on Visualization Toolkit and Spatial Fuzzy Clustering.SENSORLETTERS(01),230-235(2012)

11. Zhang Fan, Yuan Zhaokang, Xiao Fanping,You Kun,Wang Zhangye,J.:Visualization of Large Data Heat Map Based on Spark.Journal of Computer Aided Design and Graphics(11),1881-1886(2016)

12. Chen G, Yang Y, Guo H, Sun X,Chen H,Cai L,M.: Analysis and Research of K-means Algorithm in Soil Fertility Based on Hadoop Platform.Computer and Computing Technologies in Agriculture VIII. Springer International Publishing,100-110(2015)

13. Cai Lixia,D.:Based on large data processing technology Hadoop platform maize precision fertilization intelligent decision system research.Jilin Agricultural University,Changchun(2015)

14. Wen Xin, Luo Kan, Chen Rongguo,J.: Distributed spatial data analysis framework based on Shark / Spark. Earth Information Science(04),401-407(2015)

15. Zhang Xiande,D.:Based on the Spark platform real-time flow calculation recommendation system research and implementation. Jiangsu University,Jiangsu(2016)

16. Wen Qingfu, Wang Jianmin, Zhu Han, Cao Yue,Long Mingsheng,J.: Distributed Hash Learning Method for Approximate Nearest Neighbor Query.Journal of Computers(01),192-206(2017)

17. Li Wanhua, Chen Yuzhong, Guo Kun, Guo Songrong,Liu Zhanghui,J.:Parallel limit learning machine based on improved particle swarm optimization. Pattern Recognition and Artificial Intelligence(09),840-849(2016)

18. Lin Ziyu,M.:Big data technology principle and application.Beijing: People's Posts and Telecommunications Press(2017)

19. Yang Tianqing, Wang Jin, Yang Tao, Zhang Xuejie,J.:A Data Processing Mechanism for High-efficiency Large-scale Graphs in Spark.Application Research of Computers(12),3730-3734(2016)

20. Chen Heng,J.:A Spark-based Distributed Semantic Data Distributed Reasoning Framework.Computer Science(S2),93-96(2016)

21. Zhang Xinxiang, Chen Huahui, Qian Jiangbo, Dong Yihong,J.:HSSM: A method of maximizing hierarchical data for streaming data.Journal of Computer Research and Development(08),1792-1805(2016)

22. Sun Zhongfu, Du Keming, Zheng Feixiang, Yin Shouyi,J.:Research and Application of Large Data in Wisdom Agriculture.China Agricultural Science and Technology Review(06),63-71(2013)