Neural Network Regression Based on Falsity Input

In general, only the truth input is used to train neural network. This paper applies both truth and falsity input, which is the complement of the truth input, to train neural network to solve regression problems. Four neural networks are created. The first two networks are trained using the truth input to predict the truth and falsity outputs based on the truth and falsity targets, respectively. The last two are trained using the falsity input to predict the truth and falsity outputs as well. In order to add more diversity, ensemble of neural networks is applied. Each component in the ensemble contains four types of neural networks created based on our proposed techniques. Aggregation techniques are proposed to provide more accuracy results. Three classical benchmark data sets from the UCI machine learning repository are used in our experiments. These data sets are housing, concrete compressive strength, and computer hardware. It is found that the four proposed networks improve the prediction performance when compared to backpropagation neural network and complementary neural networks. Keywords— Feedforward Backpropagation Neural Network, Ensemble Neural Networks, Complementary Neural Networks, Regression Problems, Truth Neural Network, Falsity Neural Network I. I NTRODUCTION ONE of popular machines used to solve regression problems is neural network since it is found to give better accuracy results than statistical methods in various problem areas [1], [2], [3], [4], [5]. It is also found to provide better performance when compared to support vector regression (SVR) in many applications such as artificial nose regression problem [6], stock price prediction [7], water demand prediction [8], and approximation of the function with gaussian function and morlet wavelet RBF function [9]. Over the past year, neural networks have been used to solve several regression problems such as typhoon losses [10], technical target setting in QFD for Web service systems [11], wheat stripe rust [12], macroscopic water distribution system modeling [13], travel behavior analysis [14], autumn flood season in Danjiangkou reservoir basin [15], sheep growth prediction [16], software development effort estimation [17], and calibration of near-infrared spectra [18]. Several techniques have been used to improve the performance of neural network. For example, neural network was integrated with marginalized output weights to provide probabilistic predictions and to improve on the performance of sparse gaussian processes at the same computational cost as the traditional neural networks [19]. P. Kraipeerapun is with Department of Computer Science, Faculty of Science, Ramkhamhaeng University, Bangkok, Thailand (email: pawalai@yahoo.com) S. Amornsamankul (corresponding author) is with Department of Mathematics, Faculty of Science, Mahidol University, and is with Centre of Excellence in Mathematics, CHE, Bangkok, Thailand (email: scsam@mahidol.ac.th) In [20], neural network was designed to handle small training sets of high dimension by using a statistically based methodology. In [21], the number of nodes in one-hidden layer feedforward neural network were chosen based on the adaptive stochastic optimization and a linear regression method. Hou and Han [9] proposed a constructive approximation in which a single hidden layer decay RBF neural network with n+1 hidden neurons can interpolate n+1 multivariate samples and can approximate any multivariate functions without training. Their experiments demonstrated the faster convergence and better performance than support vector machines. Vizitiu et al. [22] proposed the use of full-genetic approach to train RBF neural networks. It assured optimization both connectivity and neural weights of neural networks. In [23], two implication rules in the truth table were applied to neural networks in order to increase performances of the prediction results. It was found to provide better performance when compared to backpropagation neural network and support vector regression with linear, polynomial, and radial basis function kernels. This technique was named complementary neural networks (CMTNN). In this paper, our proposed techniques will be created based on the concept of CMTNN and the ensemble technique. The ensemble of neural networks was proof to provide better results than using a single neural network in various application areas [24]. There are several ways to design an ensemble. One f the most popular and easiest ways is the use of bagging technique [25]. Therefore, we will apply bagging to CMTNN. The aim of this paper is to apply falsity input to CMTNN in order to enhance the prediction performance of neural network. The rest of this paper is organized as follows. Section II explains the basic concept of complementary neural network. Section III describes the concept of applying falsity input to complementary neural networks to solve single output regression problems. Section IV describes data sets and results of our experiments. Conclusions and future works are presented in Section V. II. COMPLEMENTARY NEURAL NETWORKS Complementary neural networks (CMTNN) consist of a pair of neural networks in which both networks have the same parameter values and they are trained using the same truth input. However, one network is trained using the falsity target value instead of the truth target value that is used to train another network. The falsity target value is the complement of the truth target value such as 0.2 and 0.8 for the falsity and the truth target values, respectively. Both networks are created based on the truth table for implication as shown in Table I. Let A andB be the input and the target of neural network, respectively. The training process of CMTNN are considered as the implication “ A → B”. The first two implication rules INTERNATIONAL JOURNAL OF MATHEMATICAL MODELS AND METHODS IN APPLIED SCIENCES Issue 5, Volume 5, 2011 874 TABLE I LOGICAL IMPLICATION Premise Conclusion Inference A B A → B True True True True False False False True True False False True shown in Table I are applied to CMTNN. The first implication rule is that if bothA andB are true then the inference is true. This rule is applied to the first network. It means that if the network is trained using the truth input and the truth target then we get the truth output. The second implication rule is applied to the second network in which if A is true butB is false then the inference is false. This means that if the network is trained using the truth input and the falsity target then we get the falsity output. Let Ttarget andFtarget be the truth and falsity target values. The falsity target value is considered as the complement of the truth target value. It can be computed as 1 − Ttarget. From these two neural networks, the falsity output should be complement to the truth output. In the testing phase, let T (xi) andF (xi) be the truth and falsity output values obtained from the first and the second networks, respectively. Both values are predicted from the input patternxi; i = 1, 2, 3, . , n wheren is the total number of input patterns in the testing phase. The combined output O1(xi) can be computed as follows. O1(xi) = T (xi) + (1 − F (xi)) 2 (1) In this paper, this technique is called CMTNN v.1. It was found that the combination of the truth output and the nonfalsity output obtained from both network provides better result when compared to backpropagation neural network (BPNN), and support vector regression (SVR) with linear, polynomial, and radial basis function kernels [23]. III. A PPLYING FALSITY INPUT TO COMPLEMENTARY NEURAL NETWORKS The third implication rule shown in Table I is applied to the proposed neural network. If A is false butB is true then the inference is true. This means that if the network is trained using the falsity input and the truth target then we get the truth output. Two novel techniques are proposed and described below. • CMTNN v.2 Instead of considering the truth and falsity target, only the truth target is used in this technique. However, we consider the truth and falsity input instead. Figure 1 shows CMTNN v.2 in the training phase. Two neural networks having the same architecture and parameter values are trained to predict degree of truth values. The first neural network is trained using the truth input whereas the second neural network is trained using the falsity input. This technique conforms to the first and the third implication rules shown in Table I. Figure 2 shows CMTNN v.2 in the testing phase. Let Tt(xi) and Tf(yi) be the truth outputs obtained from 1 NN Truth input