<figure><img src="image_1.png" /></figure>

HAL
open science

# Research of Large-Scale and Complex Agricultural Data Classification Algorithms Based on the Spatial Variability

Hang Chen, Guifen Chen, Lixia Cai, Yuqin Yang

► To cite this version:

Hang Chen, Guifen Chen, Lixia Cai, Yuqin Yang. Research of Large-Scale and Complex Agricultural Data Classification Algorithms Based on the Spatial Variability. 9th International Conference on Computer and Computing Technologies in Agriculture (CCTA), Sep 2015, Beijing, China. pp.45-52, 10.1007/978-3-319-48357-3_5 . hal-01557815

HAL Id: hal-01557815

https://inria.hal.science/hal-01557815

Submitted on 6 Jul 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Research of Large-Scale and Complex Agricultural Data Classification Algorithms Based on the Spatial Variability

Hang Chen<sup>1,2,a</sup>, Guifen Chen<sup>1,b,*</sup>, Lixia Cai<sup>1,c</sup>, YuQin Yang<sup>1,d</sup>

<sup>1</sup>College of Information Technology, Jilin Agricultural University, 130118, China;

<sup>2</sup> Institute of Scientific and Technical Information of Jilin, 130000, China

<sup>a</sup>chenhang0811@163.com, <sup>b</sup>guifchen@163.com, <sup>c</sup>419513823@qq.com,

<sup>d</sup>1172126066@qq.com

**Abstract.** In the actual classification problems, As a result of lack of clear boundary information between classification objects, that could lead to loss of classification accuracy easily. Therefore, this article from the spatial patterns of the sample properties to proceed, fuzzy clustering algorithm is proposed based on the sensitivity of attribute weights, through using the attribute weights to improve the classification capability between confusing samples, That is for researching and analysising soil nutrient spatial data with consecutive years to collect in Nongan town. Then through the analysis of the visualization technology to realize the visualization of the algorithm. Experimental results show that introducing weights portray attribute information could reduce the objective function value, and effectively alleviate the the phenomenon of boundary data that cannot distinguish. Ultimately to improve the classification accuracy. Meanwhile, use of MATLAB to form visualization of three-dimensional image. The results provide a basis for to improve the accuracy of data classification and clustering analysis of large and complex agricultural data.

**Keywords:** large-scale and complex data, spatial variation law, fuzzy clustering, soil nutrients, sensitive attribute weights

## 1 Introduction

The arrival of the era of precision agriculture<sup>[1,2]</sup>, makes a variety of complex link relationship between agricultural data features with apparent spatial variability<sup>[3]</sup> and the correlation. The consequent massive, diverse and dynamic changes, incomplete, uncertain and a series of features, so that each attribute internal link close, but contact between attributes relatively sparse<sup>[4]</sup>. However, Data mining can effectively for data analysis, Wherein the cluster analysis can be used as an independent tool to obtain data distribution situation, so that can observe characteristics of each class, analysis some specific class to move forward a single step, Final extract useful information. But with the rising importance of data structure information and the data on the exponential growth. This shows traditional data mining algorithms have been unable to meet these needs. How to introduce spatial patterns of in large-scale agriculture data<sup>[5]</sup>. And to strengthen the links between attributes for regional management in order to improve the parallel and distributed
implementation strategy of clustering algorithm[6,7]. All of these are gradually attracted researchers' attention[8]. So, On the basis of K-means algorithm, according to the interdependence of spatial unit location. Li Xiang[9], who put forward a new Spatial Contiguous K-Means Cluster algorithm, who removed a lot of debris and isolated cell and taken into account the continuity of the management partition. The actual show that the method is suitable for variable precision agriculture field management operations. Fleming etc. noted that define management zones based on oil properties, terrain and farmers' production experience. Then Appeared feature selection methods which is proposed for large-scale data sets. The purpose is to improve the data processing efficiency and rationality of the decision-making program[10]. While Cui proposed Quick association rules mining algorithm based on a large dense database of vertical data step [11].

Studied the basis of the existing methods, consider the algorithm when dealing with large data sets required scalability and efficiency. Analysis the influence of spatial variability and structural information on the temporal and spatial data, the paper proposed Fuzzy C-Means algorithm that based on attribute weights. In the case of verify its reliability, analysis the algorithm through MATLAB toolbox graphical to further improve the quality of clustering results. The results showed that the introduction of spatial variability and structure information can effectively reduce losses, due to the imbalance caused by the boundary. So, in visual processing, MATLAB played an immediate role.

# 2 Sensitive Attribute Weights Fuzzy C-means Analysis

The traditional clustering algorithms in the classification process vulnerable to the sample spatial variability and structure information effect, the existence of boundary data processing hard to demarcation issues, which lead to low accuracy [12]. So, fuzzy c-means cluster algorithm was introduced to analysis of data space structure information[13].Then constructed fuzzy similar matrix directly after standardization of data. Therefore, this study and master the premise of in soil temporal and spatial variation characteristics and laws, The combination of the spatial variability of attribute weights applied to FCM algorithm. Ultimately improving algorithm's classification capability, reducing the loss of classification precision caused by boundary spatio-temporal data.

## 2.1 Construction of Sensitive Attribute Weights

Firstly, We analyzed spatial patterns of soil nutrient which according to experience of experts in the field and soil fertility characteristics of test area[14]. The results showed that available p variation coefficient was 31.12%, and the number of rapidly available potassium was 21.51%, and available nitrogen was 11.69% [15]. Secondly, the space coefficient of variation was introduced to the algorithm and AHP could solve the weight coefficients. Now, Specific steps are as follows:

(1) Construct pair wise comparative matrix;

(2) Selected any n-dimension normalization original vector $w^{(0)}$;
(3) Calculate $\tilde{\mathbf{w}}^{(j+1)} = \mathbf{A}\mathbf{w}^{(j)}, j = 1,2,\Lambda$;

(4) Normalize the $\tilde{\mathbf{w}}^{(j+1)}$;

(5) For a given precision $\epsilon$ in advance, when $|w_i^{(k+1)} - w_i^{(k)}| < \epsilon$, $j = 1,2,\Lambda, n$, Then $\tilde{\mathbf{w}}^{(j+1)}$ is the requirement feature vector; otherwise return (2);

(6) Compute $\lambda = \frac{1}{n} \sum_{i=1}^{n} \frac{\tilde{w}_i^{(j+1)}}{\tilde{w}_i^{(j)}}$;

(7) Calculation $CI = (\lambda - n)/(n-1)$;

(8) Calculated $CR = CI/RI$;

(9) If the $CR<0.1$, then through the consistency check; Otherwise, reconstructing paired comparative matrix;

(10) when all layers are calculated out, to obtain the total target weight vector $A = (a_1,a_2,\cdots,a_m)$; if not, return back to (1).

## 2.2 Construction of Attribute Weights Fuzzy C-means Algorithm

When dealing with clustering problems with fuzzy concepts [16,17], each sample is not divided into one class strictly, but belongs to a category at a certain membership. Define:

$$J(U,V) = \sum_{k=1}^{n} \sum_{i=1}^{m} u_{ik}^{m} d_{ik}^{2} (U = (u_{ik})_{c \times n}, d_{ik} = \|x_k - v_i\|),$$ which based on the membership degree

matrix [18]. Objective function refers to the sum of weighted square distance between the samples and the center of the cluster. The optimization class refers to make the objective function to take the minimum class. If all points of a class are closed to the center of the class, the value of the goal function is very small.

## 2.3 MATLAB Modeling Tool

MATLAB is an interactive programming language based on matrix manipulation. The main functions of MATLAB including data analysis, numerical calculation and engineering drawing and so on [19]. It can also be marked and printed the graphics. This paper adopts MATLAB to process and analyze the data, compares this kind of algorithm with the traditional fuzzy c-means, compares the accuracy of the algorithm through the objective function value, and realize the 3d visualization process of the data.

# 3 Experimental Results and Discussion

## 3.1 Data Sources

The use of 3S(these are GPS, GIS, RS) and sensor technology acquisition Nongan town soil nutrient information, then based on the geolocation of arable land, then
positioning collecting position of the point. Select the main factor affecting soil fertility[20] (nitrogen, phosphorus, potassium) as the sample data for research, the sampling distribution is as follows:

<figure><img src="image_3.png" /><figcaption>Fig.1 Soil Sampling Point in Nongan town</figcaption></figure>

The plum blossom in figure 1 sampling methods is the five sampling method, that is the grid on the four horns and on the center of the grid as the soil samples mixed grid soil sample. Collecting the data content to farmers in 2010 in the town of partial data, for example, as be showed in table 1.

<table><caption>Table 1. the sample data</caption><thead><tr><th>Town name</th><th>Alkaline hydrolysis nitrogen (mg/kg)</th><th>Available phosphorus (mg/kg)</th><th>Available potassium (mg/kg)</th><th>latitude</th><th>longitude</th></tr></thead><tbody><tr><td>Nongan town</td><td>118</td><td>30.8</td><td>208</td><td>44.5</td><td>125.2</td></tr><tr><td>Nongan town</td><td>118</td><td>36.3</td><td>198</td><td>1535</td><td>3695</td></tr><tr><td>Nongan town</td><td>118</td><td>42.8</td><td>227</td><td>44.5</td><td>125.2</td></tr><tr><td>Nongan town</td><td>103</td><td>32.1</td><td>198</td><td>0502</td><td>354</td></tr><tr><td>Nongan town</td><td>132</td><td>15.8</td><td>227</td><td>44.5</td><td>125.2</td></tr><tr><td>Nongan town</td><td>165</td><td>10.8</td><td>237</td><td>0702</td><td>542</td></tr><tr><td>Nongan town</td><td>147</td><td>30.8</td><td>217</td><td>44.5</td><td>125.2</td></tr><tr><td>Nongan town</td><td>143</td><td>21.3</td><td>198</td><td>1245</td><td>5438</td></tr><tr><td>Nongan town</td><td>147</td><td>30.8</td><td>217</td><td>44.5</td><td>125.2</td></tr><tr><td>Nongan town</td><td>143</td><td>21.3</td><td>198</td><td>057</td><td>5578</td></tr></tbody></table>
## 3.2 Data Processing

Processing soil nutrient data in Nongan town from 2008 to 2012, Take on [0,1] evenly distributed random number to determine the initial membership degree matrix. Which determined cluster center by iteratively. Among them, step 1 iteration of the cluster centers is:

$$v^{(i)} = \frac{\sum_{k=1}^{n} (u_{ik}^{(i-1)})^m x_k}{\sum_{k=1}^{n} (u_{ik}^{(i-1)})^m}, i = 1, 2, \dots, c$$

where c is the number of classes, m > 1.

## 3.3 Application and Analysis of Algorithms

Preprocessing the data after combined with the analysis of the soil nutrient spatial variation. Algorithm through continuous iterative to adjust the size of the objective function value in order to achieve the classification of soil fertility. To objects in the cluster based on the continuity of time and space, after processing the sample data, We use sensitive attribute weights fuzzy C-means algorithm to analysis the data from 2008 to 2012. Experiments show that when taking membership degree exponent 8, clustering result is obvious. In the case of the same power exponent value, compared with the traditional fuzzy C-means clustering algorithm, after repeated experiments and found that the accuracy and operational efficiency of the improved algorithm are both higher traditional algorithm. Wherein the results of 2011 as shown in Table 2.

<table><caption>**Table 2. The result**</caption><thead><tr><td>Algorithm</td><td>Mean objective function</td><td>Average running time(s)</td></tr></thead><tbody><tr><td>Fuzzy C-Means</td><td>15.118689</td><td>0.217</td></tr><tr><td>Attribute weights Fuzzy C-Means</td><td>11.989009</td><td>0.180</td></tr></tbody></table>

From Table 2 we can see that under the same conditions, the objective function value is smaller, and the accuracy of the relative increase 21.7%, also it has a higher operating efficiency. That because the sample edge has no clear demarcation point, and Fuzzy C-Means could improve this problem when dealing with data. On this basis, introducing of spatial variation regularity. Without prejudice to the classification results, the better management area is divided. Combined with the results of the above analysis, using MATLAB visualization toolbox for data processing. The results obtained in Fig. 2 and Table 3.
<figure><img src="image_4.png" /><figcaption>Fig.2 The three dimensional clustering figure</figcaption></figure>

<table><caption>Table 3. The clustering results</caption><thead><tr><th>Clustered Data</th><th>2008</th><th>2009</th><th>2010</th><th>2011</th><th>2012</th></tr></thead><tbody><tr><td>Cluster 0</td><td>3</td><td>93</td><td>44</td><td>74</td><td>82</td></tr><tr><td>Cluster1</td><td>123</td><td>99</td><td>114</td><td>108</td><td>108</td></tr><tr><td>Cluster2</td><td>174</td><td>108</td><td>142</td><td>118</td><td>110</td></tr></tbody></table>

The table 3 and figure 2 show that after years of continuous precise fertilization, the similar degree of data is improved in gradually, the discrepancy between categories gradually become smaller, the soil fertility difference is leveling off. All above shows that after precise fertilization, the plot of soil in Alkeline-N, Olsen-P and Olsen-K three nutrients data integrated similarity increased year by year; On the other hand also proved that the attribute weights are C clustering algorithm is suitable for evaluation of soil fertility.

## 4. Conclusion

The attribute weights fuzzy c-means algorithm is used to analysis and evaluation for Nongan county Nongan town soil nutrient data for five consecutive years (2008 - 2012). The test results show that after five consecutive years of precise fertilization, soil fertility condition had the obvious change. The attribute weights c-means clustering algorithm is an effective methods of research and evaluation of the soil fertility, in line with the farmers, the change trend of soil fertility.

Firstly, the algorithm consider the spatial variability of soil fertility, combined with AHP to determine the sensitive attribute weights. the original scattered data not only retain the traditional algorithms consider the problem of difficult to deal with the boundary points by using the concept of fuzzy sets, and to overcome the imbalance between the various properties and is sensitive to "noise" and outlier data shortcomings.

Secondly, the paper used the sensitive attribute weights fuzzy c-means algorithm to do the clustering analysis for Nongan soil data in 2011 which included the soil alkaline hydrolysis nitrogen, available phosphorus and available potassium three nutrients data. The results show that the algorithm was 21.7% higher than that of traditional algorithm of relative accuracy and efficiency increased by 17%, the improved algorithm clustering effect is better.
Thirdly, using the algorithm to analysis soil nutrient data which precision
fertilization consecutive for five years, the results show that the whole plot soil in
alkali solution nitrogen, available phosphorus and available potassium in three kinds
of nutrient data integrated similarity increased with each passing year. The results of
the experiment are consistent with the actual situation of soil fertility, which provides
a new reference for the analysis of the status of soil fertility in the future.

Fuzzy clustering is a rather ambiguous concept, the two clustering algorithms
should be repeated iterations based on the exponent value of the objective function
membership degree, so as to determine the relatively close to the true clustering value.
We know MATLAB can handle large-scale data, and the formation of the visual
clustering results. Covered in this article the agricultural data mostly a single soil
nutrient data. Face of the growing complexity of massive agricultural data, the
original matrix processing mode is not enough. In the future, attention should be
application testing large data sets, in order to confirm the validity of the algorithm of
massive data clustering.

Acknowledgment

The paper was supported by the national “863” project (2006AA10A309), the
National spark plan project(2015GA660004), National Spark Plan (2008GA661003)
and Shi Hang projects of Jilin province (2011- Z20).

References

1. Shuhao Zhang. Research of precision agriculture automatic variable fertilization theory and technology based on GPS, GIS.[D]. Jilin: Jilin University, 2003: 1-3.

2. Liping Chen. Theoretical and Experimental Studies on Variable-rate Fertilization in Precision Farming [D]. Beijing: China Agricultural University, 2003: 98-104.

3. Jiangqiu Xiang, Fu Qiang, Wang Zilong. Applied research progress of spatial variability theoretical in soil properties analyis[J]. Soil and Water Conservation study.2008,15(1):250-253.

4. Zhao Enlai,Hao Wenning,Liu Hang,Rong Yu,Zhu yaohua.Compression Algorithm of Beidou Position Redundant Data Based on Time Series Clustering.Computer Engineering 2012,2:40-42.

5. Guifen Chen. Research and Application of Spatial Data Mining Technology for Precision Agriculture [D].ChangChun: Jilin University,2009.

6.Gao Xiaomei Feng Yun, Feng Xingjie. Incremental - K Medoids clustering algorithm [J]. Computer engineering, 2005, 7 (31): 181-183.

7. Leung K W T,Ng W,Lee D L. Personalized Concept-based Clustering of Search Engine Queries[J]. IEEE Transactions on Knowledge and Data Engineering, 2008, 20(11): 1505-1518.

8. DENG Min, LIU QiLiang, WANG Jia Qiu,SHI Yan.A general method of spatio-temporal clustering analysis. SCIENCE CHINA PRESS,, 42,(1),111-124.(2012).
9. Li Xiang, Pan Yu Chun, Zhao Chunjiang.Precision agriculture management zones based on spatial continuity of the clustering algorithm[J].Agricultural Engineering Journal.2005,21(8):78~82.

10. Zhang Li,Chengong He. A large data set suitable for feature selection method [J]. Computer Science,2007,33(4):184~186.

11. Cui Jian,Li Qiang.Quick association rules mining algorithm based on a large dense database of vertical data step[J].Computer Science.2012,39(1):134~137,151.

12. Jing Hu, Daoliang Li, Qingling Duan, Yueqi Han, Guifen Chen. A Fuzzy C-Means Clustering Based Algorithm to Automatically Segment Fish Disease Visual Symptoms. SENSOR LETTERS, (10):1-8, (2012).

13. Jing Hu, Daoliang Li, Qingling Duan, Yueqi Han, Guifen Chen. A Fuzzy C-Means Clustering Based Algorithm to Automatically Segment Fish Disease Visual Symptoms. SENSORLETTERS,2012,(10):1-8.

14. Zhao Yueling, HAN Haiyan, CAO Liying, CHEN Guifen.Study on Soil Nutrients Spatial Variability in YuShu City.Computer and Computing Technologies in Agriculture, (2),:1-7,(2012).

15. GuiFen Chen, LiIng Tsao, Guowei Wang. Application of weighted spatial fuzzy clustering algorithm in soil fertility evaluation [J]. Chinese Agricultural Science.2009,42 (10) :3559-3563.

16. BACKER E, JA IN A K. A clustering performance measure based on fuzzy set decomposition [J]. IEEE Trans on Pattern Analysis and Machine Intelligence, 1981, 3 (1): 66277.

17. Yan Li, Zhou Shi, Cifang Wu, Feng Li, Jieliang Cheng. Definition of Management Zones Based on Fuzzy Clustering Analysis in Coastal Saline Land. Scientia Agricultura Sinica, 2007, 40(1): 114-122.

18. Yu Helong, Liu Dayou, Chen Guifen.Determination of the Soil Nutrient Management Zones Based on Weighted Fuzzy Clustering.Transactions of the Chinese Society for Agricultural Machinery,2009,(40):177-182.

19. Bai Xiaoming, Xiong Shijun.The implementation and application of mathematical experiment system based on MATLAB [D].Nanchang University .2012.

20. Umeda M, Kaho T, Michihisa IIDA, Choung Keun LEE. Effect of variable rate fertilizing for paddy field. 2001 ASAE annual international meeting, 2001, Paper Number. 01(Part. II).