Journal of Theoretical and Applied Information Technology
15th November 2019. Vol.97. No 21© 2005 – ongoing JATIT & LLS

ISSN: 1992-8645

www.jatit.org

E-ISSN: 1817-3195

EFFECT OF CLUSTERING DATA IN IMPROVING MACHINELEARNING MODEL ACCURACY
SAMIH M. MOSTAFA1, HIROFUMI AMANO2
1
Mathematics Department, Faculty of Science, South Valley University, Qena, Egypt
2

Research Institute for Information Technology, Kyushu University, Japan
E-mail: 1 samih_montser@sci.svu.edu.eg, 2 amano@cc.kyushu-u.ac.jp
ABSTRACTSupervised machine learning algorithms consider the relationship between dependent and independent
variables rather than the relationship between the instances. Machine learning algorithms try to learn therelationship between the input and output from the historical data in order to attain precise predictions
about unseen future. Conventional foretelling algorithms are usually based on a model learned and trained
from historical data. The instances in the historical data may vary in its characteristics. The variation may
be a result of difference in case's pertinence degree to some cases compared to others. However, theproblem with such machine learning algorithms is their dealing with the whole data without considering
this variation. This paper presents a novel technique to the trained model to improve the prediction
accuracy. The proposed method clusters the data using K-means clustering algorithm, and then applies theprediction algorithm to every cluster. The value of K which gives the highest accuracy is selected. The
authors performed comparative study of the proposed technique and popular prediction methods namely
Linear Regression, Ridge, Lasso, and Elastic. On analysing on five datasets with different sizes anddifferent number of clusters, it was observed that the accuracy of the proposed technique is better from thepoint of view of Root Mean Square Error (RMSE), and coefficient of determination 𝑅 .
Keywords: Prediction accuracy, K-means, clustering, regression, machine learning algorithms.
1.

INTRODUCTION

Machine learning (ML) is considered as theimportant subfield of artificial intelligence and isbeing adopted for numerous of various
applications [1, 2]. ML addresses the study andconstruction of models capable of learning fromthe data. Understanding what and how the ML
algorithm is learning is an issue for the developersof the ML applications [3]. ML can be classified
into unsupervised and supervised. Unsupervised
learning groups the data into categories dependingon the basis of the similarities between data in eachgroup. On the other hand, supervised learningmeans that the machine learns with the assistanceof the labeled training data. Estimating unknown(independent, dependent) mapping of a systemusing a specific number of (independent,
dependent) samples is called learning [4, 5]. This
process of estimating needs data collected (i.e.,
training data), and an algorithm that deals with thisdata and learns from it. Generally speaking, thelearning algorithm learns pattern in the data on
hand and create a set of rules to map input/output
2973

relation. Data is categorized into labeled (withoutcome) or unlabeled (without outcome).
Outcome variable(s) may be continuous or distinct,
regression is a way of predicting for continuous
outcome, and classification is a way of predicting
for distinct outcome (i.e., the response to bepredicted is the probability or the true of anevent/class), the number of classes can be two or
more. On the other hand, clustering is applied tounlabeled data using the similarities between
observations to group them into clusters [6–8]. The
statistical method depends on the characteristics ofthe data (e.g., similarities between instances in theclustering technique), in other words, the moresimilarities the better statistical method accuracy.
Regression is one of the most common statisticalprocesses for estimating dependent/independent
relationship when the dependent to be predicted isa continuous value. The regression line is a refinedoutline of averages and is drawn in such a way as
to reduce the error of the fitted values in relation to
the actual values. Equation of the simple linear
regression can be defined by the following form:
d

C

C I

Journal of Theoretical and Applied Information Technology
15th November 2019. Vol.97. No 21© 2005 – ongoing JATIT & LLS

ISSN: 1992-8645

www.jatit.org

where C is the intercept and the C is the slope ofthe regression line. In addition to explaining therelationship between dependent d and independent
I variables, the model also predicts the value of
dependent variable from the independent variablevalues from the equation:
d

C

C I

the hat symbol refers to the predicted value ofthe unknown coefficient/variable. In simple
regression, for a given dependent variable there is
one independent variable. However, in real cases
there is more than one independent variable, so
existing of multiple independent variables is called
multivariate regression. The mathematical notation
can take the form:
d

C I

C I

C I

⋯

C I

where m is the number of independent variables.

E-ISSN: 1817-3195

supported with an illustrative example; dataset
specification, the prediction algorithms used in thecomparison, the parameters used in thecomparison, results and discussion are discussed in
section 3.
2.

PROPOSED METHOD
In the proposed method, the authors avail
from the similarity attribute of the data byclustering it into groups (clusters) before applying
the statistical method. In the proposed method, thenumber of clusters is determined by the elbow
method (heuristic method of validation andinterpretation of symmetry within cluster analysis).
The data is clustered using K-means algorithm inwhich the number of clusters resulted from theelbow method is used for the clustering. The
selected prediction algorithm is applied for every
cluster. For deeper clarification, the next
subsection discusses an illustrative example.
Algorithm:

1.1. Goodness of the Model
The performance of the model can beevaluated using following metrics:• R-squared: how well the model fits the data.
•Root Mean Square Error (RMSE): how closethe estimated values are to the actual values.
1.2. Paper Contributions & Novelty
This paper gives a brief discussion ofmachine learning types, proposes a method for
improving the prediction accuracy, and compares
between the proposed method and the common
methods. In contrast to other techniques, e.g.,
Clustering Lasso (CL) which selects groups ofvariables that have the same mechanism ofpredicting the dependent variable [9], the noveltyof the proposed work boils down in benefitingfrom similarities between instances and applying
the selected prediction algorithm for each cluster.
The proposed work does not depend on
collinearity among variables or the number ofvariables compared to instances.

Proposed method

1.

Input data

2.

Select the prediction algorithm

3.

Find the value of K using elbow method

4.

Cluster the data using K-means

5.

Apply the selected prediction algorithm ineach cluster

In the proposed method, clustering prepressing
step is applied to the data before applying theprediction algorithm for the purpose of improving
the accuracy of model generated by the userselected prediction algorithm.
2.1 Illustrative Example
To clarify the proposed method, an
artificial data with 600 observations has been
generated, and linear regression algorithm isapplied to the data (Figure 1). K-means algorithm
is applied to clustering the data, the value of K isdetermined using elbow method (Figure 2), thedata is clustered into three clusters (Figure 3), thenlinear regression algorithm is applied to the data ineach cluster (Figure 4).

1.3. Organization
The rest of this paper is organized asfollows: Section 2 presents the proposed method

2974

Journal of Theoretical and Applied Information Technology
15th November 2019. Vol.97. No 21© 2005 – ongoing JATIT & LLS

ISSN: 1992-8645

www.jatit.org

Figure 1. Regression line for all data

Figure 2. Determining the value of k using elbow method.

Figure 3. Three clusters of the data

Figure 4. Three regression lines for the clusters

2975

E-ISSN: 1817-3195

Journal of Theoretical and Applied Information Technology
15th November 2019. Vol.97. No 21© 2005 – ongoing JATIT & LLS

ISSN: 1992-8645

www.jatit.org

E-ISSN: 1817-3195

The comparison is done between fourcommon prediction algorithms namely multible
linear regression (MLR), Ridge, Lasso, andElasticNet and the proposed method. Table 2 gives
short descriptions of the prediction algorithms
used.

3. EXPERIMENTAL IMPLEMENTATION
3.1. Datasets
Six datasets that are commonly used indatabases repository are used in the comparative
study (Table 1).
3.2. Prediction Algorithms

Table 1. Datasets specifications
Dataset name#Instances#Features

References

Diabetes

442

11

[10]

Graduate Admissions

500

8

[11]

California

20640

9

[12]

Diamonds

53940

10

[13]

Boston

506

12

[14]

Iris

150

6

[15]

Table 2. Packages and functions used
Prediction method
MLR

Short description
minimizes the residual sum of squares between the observed responses in the dataset,and the responses predicted by the linear approximation.

References
[16]

Ridge

solves a regression model where the loss function is the linear least squares function,and imposes a penalty on the size of the coefficients.

[17]

Lasso

estimates sparse coefficients. Coefficients that add lightweight value to the model will
be zero[18]

ElasticNet

allows for learning a sparse model where few of the weights are non-zero like Lasso,
while still maintaining the regularization properties of Ridge.

[19]

3.3. Performance Measure
The comparisons were done from thepoint of view of the following parameters: Root mean squared error (RMSE): indicates
how close the forecasted values are to the actual
values; hence the lower value of RMSE, thegood of the model performance [20]. The
mathematical notation can be written as:
𝑅𝑀𝑆𝐸

∑

𝑦
𝑛

𝑅 𝑦, 𝑦

1
𝑦

∑
∑
1
𝑛

𝑦
𝑦

𝑦
𝑦

;

𝑦

3.4. Experimental Results and Discussion
The experiments are conducted on acomputer equipped with 16 GB of RAM, Intel corei5-2400 (3.10 GHz), 1 TB of HDD, Gnu/Linux
Fedora 28 of OS, and Python (version 3.7) of
programming language. Table 3 summarizes theobservations by comparing the improvements ofthe proposed approach versus common fouralgorithms from the point of view of R2 score and
RMSE. The notable observations are:
 Clustered MLR is better than MLR in R2
score and RMSE for all datasets.
 Clustered Ridge is better than Ridge in R2
score and RMSE for all datasets.
 Clustered Lasso is better than Lasso for Iris,
Diamond, and Diabetes, and behaves𝑦where 𝑦 and 𝑦 are the actual value and forecasted
value of the i-th observation respectively, and n is
the number cases.
 Coefficient of determination (R2 score): it is ameasure of how perfectly the evaluatedregression line of the model adapts the data
distribution [21]. It can be written as:2976

Journal of Theoretical and Applied Information Technology
15th November 2019. Vol.97. No 21© 2005 – ongoing JATIT & LLS

ISSN: 1992-8645



www.jatit.org

somewhat similar to Lasso for Admission,
Boston, and California.
Clustered ElasticNet is better than ElasticNet
for Iris, Diamond, and Diabetes, and behaves
somewhat similar to ElasticNet for
Admission, Boston, and California

E-ISSN: 1817-3195

Table 4 shows the comparisons between the fourcommon methods and the proposed method fromthe point of view of R2 score and RMSE.

Table 3. Proposed approach versus MLR, Ridge, Lasso, and ElasticNet
MLR
dataset2

R scoreAdmission

Ridge
RMSE

2

R scorebetterbetter

BostonbetterCalifornia

Lasso
RMSE

2

R scorebetterbetterbetterbetterbetterbetterDiabetes

better

Diamond
Iris

ElasticNet
RMSE

2

R score

RMSE

worst

worst

worst

same

betterbetterbettersame

same

betterbetter

worst

worst

worst

worst

betterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterbetterto improve the prediction accuracy. Four common
algorithms are compared with the proposed
method, the results showed that the proposed
method achieves significant improvement from thepoint of view of RMSE, and coefficient ofdetermination 𝑅 . In the future research avenues,
the proposed approach will be analysed in moredataset, other standard error metrics will beconsidered (e.g., P-value and T-value).

4. Conclusion and Future Works
This paper introduces a method that aims
to improve the prediction accuracy of the model byclustering the data and applying the selected
algorithm, which is a user choice, for each cluster.
Unlike the traditional supervised algorithms whichfind the relationship between the dependent andindependent variables, the proposed approach
benefits from the similarities between the instances

2977

Journal of Theoretical and Applied Information Technology
15th November 2019. Vol.97. No 21© 2005 – ongoing JATIT & LLS

ISSN: 1992-8645

www.jatit.org

E-ISSN: 1817-3195

Table 4. Comparisons between common algorithms and the proposed
Clustered MLR

datasetClusterID012
Admissi
on
(500
obs.)

3
4
5

#
obs.
per
clus
ter

80
85
84
83
82
86

Average1
Boston
(506
obs.)

2

185
137
184

Average

RMSE

R2 score

RMSE

R2 score

RMSE

0.9005870.7647280.900393

0.042692
0.06693
40.03267
70.0358890.040976
0.0631270.047049204

0.903082
0.73140.772022
0.8762850.9171420.8538270.842293064
0.036151576

0.0421530.0715180.0494360.0352460.0403530.07100
40.051618251
0.088516127

0.8966870.772732

0.043522
0.06578
50.0312180.60816
0.7988810.740846565

3.809681
4.940371
4.0671294.272393928

0.7599970.3291450.7768420.621994708
0.19108178

0.6659580.6429950.7535320.64994
4
0.678107166

0.6016860.730971
0.448398
0.5818090.590716165

0.6277220.24455
4
0.092901

38.0004934.339150.168727508

36.16982113

0.87173
0.914561
0.88446
0.87274
3286

0.81549
9

Improvement

0
California(20640
obs.)

1
2
3

720
8
113
39
234
185
9

Average
Improve
ment0Diabete
s
(442
obs.)

1

180
262

Average
Improvement

0
Diamond(53940obs.)

1
2
Average

329
63
565
4
153
23

Clustered Ridge

R2 score

Improve
ment

0

MLR

0.8786490.4046590.5922870.625198361

302.700
3
1986.2131007.742
1098.885094

0.60065
0.7059910.6304020.641191351
0.057573789-0.58888-1.30065
0.9447616
1.178592683
0.330429-0.51074
0.291297
0.03699
4739

2978

Ridge

R2 score

RMSE

0.0423190.071091

0.902320.734592
0.7716180.8758020.9154820.8538660.842280112
0.037388136

0.049480.0353140.04075
50.07099
4
0.051658971
0.091137201

0.909090.8694950.906431
0.8881930.873771396

0.03620.042882
0.062099
0.046950917

4.3450846.4642684.2841745.03117545
0.150815953

0.8046510.5997210.8070250.737132497

3.92008
4
4.9932853.98392
54.299097932

0.7552140.307886
0.77468
40.612594663
0.203295656

4.3881646.5658944.304843
5.086300222
0.154769136

0.6351890.773106
0.4897380.5978290.623965652
0.053287367

0.66600
1
0.6429930.6016480.730973
0.4739080.5824240.597238209

0.6276890.6006630.7059470.63035
50.641163447
0.046101354

0.6352170.773095
0.48977
50.5978670.623988367
0.042869642

55.11027
54.687390.179101-1.15089

0.10635

39.6125
34.08364

64.120453.70962

54.89883005

0.142725619

36.84807093

0.72469
0.6492040.67072
195

0.341154974

711.0323164.0121328.63
1734.558121

0.878381
0.4042850.5927060.625123609

303.034
5
1986.8381007.224
1099.032344-1.21911
1.18500069
1.12044349
0.329419-0.5104
0.29176
90.036929956

58.9150139
0.374555508
711.56783163.6531328.1881734.46
9421

Journal of Theoretical and Applied Information Technology
15th November 2019. Vol.97. No 21© 2005 – ongoing JATIT & LLS

ISSN: 1992-8645
Improve
ment

0
Iris
(150
obs.)

1

53
97

Average

0.5858920.505393
0.545642513

0.1598240.2468510.20333
7645

Improve
ment

Clustered Lasso

datasetClusterID012
Admissi
on
(500
obs.)

3
4
5

#
obs.
per
clus
ter

80
85
84
83
82
86

Average1
Boston
(506
obs.)

2

185
137
184

Average

California(20640obs.)

1
2
3
Average
Improve
ment

720
8
113
39
234
185
9

15.89965597

0.366475484

15.92727726

0.366358189

0.5383730.519073
0.528722806
0.032001092

0.1687450.2434140.206079331
0.013304032

0.580346
0.5166030.548474215
0.019913651

0.160891
0.2440380.202464357
0.00951
2148

Lasso

0.6094860.50930
60.559396339

0.1552040.2458730.20053
8486

Clustered
ElasticNet

ElasticNet

RMSE

R2 score

RMSE

R2 score

RMSE

R2 score

RMSE

0.3972480.4189750.013930.042890.2178020.2596220.206137182

0.1051230.10518
60.1042570.1023330.123983
0.15979
90.116780231

0.103192
0.218946
0.2470150.2848160.2951940.1282270.12195
50.0898450.0847430.624298
825
0.558337
418
0.590989
869
0.383689
678
0.653176
120.649727
827
0.576703
29

0.082995
0.0917080.0662170.07866
70.08255
80.1099140.085342992

0.476870
260.500034
743
0.636564
741
0.595595
155
0.645685
84
0.667472
07
0.587037
135
0.017603
393

0.09793
4
0.09757
3
0.0624190.06372
40.0834450.107093
0.085364642

0.504211
0.491277
0.7885130.594667172

6.2450795.6291894.1706395.348302053

Improvement

0E-ISSN: 1817-3195

R2 score

Improvement0

www.jatit.org

0.1945110.3474120.008360.0633510.149228941

0.9343280.9882850.906966
0.9517010.945319905

0.298720.241313579
0.14577048

0.11769
0.15552
30.116330496
0.00386
601

0.6312760.4709520.642887
0.581704998

5.3856785.7405375.4195585.515257787

0.022283071

0.03027161

0.2842080.2967320.3153680.28480
60.29527854

0.88077
1
1.0259430.7473290.8316190.871415265

0.49461637

0.08480992979

0.000253615

0.555916
205
0.489438
342
0.705213
571
0.583522
706

5.910471
5.6393534.923962
5.491261589

0.635021
837
0.473984
885
0.656423
965
0.588476895
0.008418
664

5.3582515.724061
5.3158455.466052459
0.00461194

0.398249064
0.450214
811
0.320412
819
0.325355
782
0.373558
119

0.8075660.9071080.7445710.8076990.816735984

0.422555
231
0.436138572
0.486811
721
0.420265792
0.441442
829

0.791088
0.9186470.647026
0.748732
0.77637343
0.05198858

0.153779
165

Journal of Theoretical and Applied Information Technology
15th November 2019. Vol.97. No 21© 2005 – ongoing JATIT & LLS

ISSN: 1992-8645

www.jatit.org

0180
Diabete
s
(442
obs.)

1
262
Average

0.01644943.35967

1.50228

69.1600
30.023830.00368829

36.48171

1.62935
1.56581185
0.997644487

58.46368

39.9206
8967

Improvement

0

Diamond(53940
obs.)

1
2

[3]
[4]

[5]

[6]
[7]

0.374400116

713.57950.743478863

440.100
9

565
4
153
23

0.40457
50.593488

1986.35
3
1006.2570.509160.2930333162.361327.0020.253176
236
0.402987
813

2224.60
1
1219.4490.624600687

1099.639291

0.036496857

1734.313961

0.466547
637

1294.716959

16.1138214

0.365951428

1
970.032110.2523186.74515

0.6911940.197810.1149626

0.3841490.189023.46708467
0.966841711

0.38273
70.318233686

REFERENCES:[2]

40.09702716

0.325622

53[1]

36.47947306.308
30

Improvement

43.714580.87573
9Improvement

Average63.81185592

0.000281
698
0.023699
919
0.011709
111

329
63

Average

Iris
(150
obs.)

E-ISSN: 1817-3195

T. Segaran, Programming CollectiveIntelligence: Building Smart Web 2.0
Applications. 2007.
D. Vallejo-Huanga, P. Morillo, and C.
Ferri, “Semi-Supervised Clustering
Algorithms for Grouping Scientific
Articles,” Procedia Comput. Sci., vol.
108, pp. 325–34, 2017.
V. K. Ayyadevara, Pro Machine
Learning Algorithms. 2018.
V. Cherkassky and F. M. Mulier,
Learning from Data : Concepts, Theory,and Methods / V. Cherkassky, F. Mulier.
1998.
W. A. Yousef and S. Kundu, “Learningalgorithms may perform worse withincreasing training set size: Algorithmdata incompatibility,” Comput. Stat.
Data Anal., vol. 74, pp. 181–97, 2014.
K. Chen and L. Liu, “The ‘Best K’ for
Entropy-based
Categorical
Data
Clustering.,” 2005, pp. 253–62.
M. N. Murty, P. J. Flynn, and A. K. Jain,“Data Clustering: A Review,” ACM
Comput. Surv., vol. 31, p. 43, 1999.

0.53696541
0.407347886

0.2523180.3841490.318233686

0.320155658
2.206191315
0.416165349
0.703393
874
1.663280
779
1.973883
919
0.367240443
0.803321
738
0.856890
97

91.87302
68.8260480.34953014
0.500967497

998.3974609.33
1205.916
2271.214188
0.42994502

0.4282990.2792060.353752519
0.10040588[8]

A. K. Jain and R. C. Dubes, “Algorithms
for Clustering data.” 1988.

[9]

Q. Yu and B. Li, “Regularization andEstimation in Regression with Cluster
Variables,” Open J. Stat., vol. 04, no. 10,
pp. 814–25, 2014.
N. D. of Statistics, “Diabetes Data-1-52019,” NCSU Department of Statistics.
[Online].
Available:
https://www4.stat.ncsu.edu/~boos/var.sel
ect/diabetes.html. [Accessed: 01-May2019].
M. S. Acharya, “Graduate Admissions1-5-2019,” Kaggle Inc, 2018. [Online].
Available:
https://www.kaggle.com/mohansacharya/graduate-admissions. [Accessed: 01May-2019].
C. Nugent, “California Housing Prices3-5-2019,” Kaggle Inc, 2017. [Online].
Available:
https://www.kaggle.com/camnugent/cali
fornia-housing-prices. [Accessed: 03May-2019].
M.
Shiva,“Diamonds-20-4-2019,”
Kaggle Inc, 2017. [Online]. Available:

[10]

[11]

[12]

[13]

2980

0.032111237
0.197813
953
0.114962
595

3.415714
253
2.644023
327
3.029868
79
0.996135
44

Journal of Theoretical and Applied Information Technology
15th November 2019. Vol.97. No 21© 2005 – ongoing JATIT & LLS

ISSN: 1992-8645

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

www.jatit.org

https://www.kaggle.com/shivam2503/diamonds. [Accessed: 20-Apr-2019].
I. Kaggle, “Boston Housing-25-4-2019.”[Online].
Available:
https://www.kaggle.com/c/bostonhousing. [Accessed: 25-Apr-2019].
I. Kaggle, “Iris Dataset-25-4-2019.”[Online].
Available:
https://www.kaggle.com/jchen2186/mac
hine-learning-with-iris-dataset.
[Accessed: 25-Apr-2019].
D. W. Gareth James Trevor Hastie,
Robert Tibshirani, An introduction tostatistical learning : with applications in
R. New York : Springer, [2013] ©2013.
Scikit-learn, “Ridgre Regression-26-42019.”[Online].
Available:
https://scikitlearn.org/stable/modules/generated/sklea
rn.linear_model.Ridge.html. [Accessed:
26-Apr-2019].
Scikit-learn, “Lasso Regression-27-42019.”[Online].
Available:
https://scikitlearn.org/stable/modules/linear_model.ht
ml. [Accessed: 27-Apr-2019].
Scikit-learn, “Elastic Net Regression-284-2019.”[Online].
Available:
https://scikitlearn.org/stable/modules/linear_model.ht
ml#elastic-net. [Accessed: 28-Apr2019].
I. B. Aydilek and A. Arslan, “A hybrid
method for imputation of missing values
using optimized fuzzy c-means withsupport vector regression and a genetic
algorithm,” Inf. Sci. (Ny)., vol. 233, pp.
25–35, 2013.
T. Beysolow II, Introduction to Deep
Learning Using R. Apress, 2017.

2981

E-ISSN: 1817-3195