# Combination Estimation of Smoothing Spline and Fourier Series in Nonparametric Regression

Authors: Ni Putu Ayu Mirah Mariati; I. Nyoman Budiantara; Vita Ratnasari
Journal: Journal of Mathematics
 (2020)

---

## Abstract

So far, most of the researchers developed one type of estimator in nonparametric regression. But in reality, in daily life, data with mixed patterns were often encountered, especially data patterns which partly changed at certain subintervals, and some others followed a recurring pattern in a certain trend. The estimator method used for the data pattern was a mixed estimator method of smoothing spline and Fourier series. This regression model was approached by the component smoothing spline and Fourier series. From this process, the mixed estimator was completed using two estimation stages. The first stage was the estimation with penalized least squares (PLS), and the second stage was the estimation with least squares (LS). Those estimators were then implemented using simulated data. The simulated data were gained by generating two different functions, namely, polynomial and trigonometric functions with the size of the sample being 100. The whole process was then repeated 50 times. The experiment of the two functions was modeled using a mixture of the smoothing spline and Fourier series estimators with various smoothing and oscillation parameters. The generalized cross validation (GCV) minimum was selected as the best model. The simulation results showed that the mixed estimators gave a minimum (GCV) value of 11.98. From the minimum GCV results, it was obtained that the mean square error (MSE) was 0.71 andR2 was 99.48%. So, the results obtained indicated that the model was good for a mixture estimator of smoothing spline and Fourier series.


---

## Body

## 1. Introduction

Regression curve approaches that are often used are parametric regression and nonparametric regression approaches. However, not all relationship patterns can be approached with a parametric approach because there is no information about the form of the relationship between the response variable and the predictor variable. If the shape of the curve is unknown and the pattern is spread, it can be assumed that the regression curve uses an approach of the nonparametric regression model. Some nonparametric regression models that are widely used are spline [1] and Fourier series estimators [2].Estimation methods have attracted a lot attention of nonparametric regression researchers and become popular among them. One of the methods is the smoothing spline estimator. Smoothing spline estimates nonparametric regression functions that are assumed to be smooth in the sense that the function is included in a particular function space and is often assumed to be appropriate in the Sobolev space. Smoothing spline also has an excellent power to handle data of which the nature changes at certain subintervals [3–7]. In addition to the spline estimator, there is a popular estimation technique in nonparametric regression, namely, the Fourier series estimator. Fourier series is a trigonometric polynomial that has flexibility so that it can adapt effectively to the local nature of the data. The Fourier series estimator is generally used if the data investigated for the pattern are unknown and there is a tendency for repeated patterns [8, 9]. Fourier series is one model that has a good statistical and visual interpretation among the nonparametric regression models [10]. Advantages of estimating Fourier series are being able to handle data characters that follow repeated patterns at certain trend intervals and having good statistical interpretation. In previous research studies, only one type of estimator was developed.Along with the development of research on nonparametric regression, lately a mixed estimator has been developed in nonparametric regression. Sudiarsa et al. developed a combination estimator of Fourier series and truncated spline [11]. Budiantara et al., Rismal et al., and Ratnasari et al. developed a mixture of kernel and truncated spline estimators [12–14]. Other research on mixed estimators was conducted by Afifah et al. and Nisa et al. who developed a mixture of kernel and Fourier series [15, 16]. Recent research about mixture estimator is mix local polynomial and truncated spline which was developed by Suparti and Santoso [17]. Mixed smoothing spline and kernel estimator was developed by Hidayat et al. [18–20]. In daily life, mixed data patterns often appear, and some data patterns change at certain subintervals and some follow recurring patterns at certain trends. So as to handle the data pattern, in this study, we developed combination estimation of smoothing spline and Fourier series. Based on the description of previous research studies, focus of this paper will be emphasized on the nonparametric regression model that combines smoothing spline and Fourier series obtained through optimization of penalized least squares (PLS). Furthermore, this combined estimator is applied to the simulation data. These simulation data are generated from two different functions to represent two different patterns of predictor variables so that this condition is in accordance with the combined estimator that was developed.


## 2. Materials and Methods

The data provided is in pairsx1i,x2i,…,xpi,t1i,t2i,…,tqi,yi of which is assumed that the predictor variables were x1i,x2i,…,xpi,t1i,t2i,…,tqi and the respon variables yi. The relationship between the two variables follows nonparametric regression multivariable model:(1)yi=fx1i,x2i,…,xpi,t1i,t2i,…,tqi+εi,i=1,2,…,n.Assume the multivariable nonparametric regression model is additive, so the regression model is obtained as follows:(2)yi=∑j=1pgjxji+∑k=1qhktki+εi.In the estimation process using PLS function,hktki is a fixed model. Furthermore, the function hktki is approximated by the Fourier series function. Function gjxji is assumed to be in Sobolev space.Based on equation (2) for i=1,2,…,n, the following equation is obtained:(3)y1=g1x11+g2x21+⋯+gpxp1+h1t11+h2t21+⋯+hqtq1+ε1,y2=g1x12+g2x22+⋯+gpxp2+h1t12+h2t22+⋯+hqtq2+ε2,⋮yn=g1x1n+g2x2n+⋯+gpxpn+h1t1n+h2t2n+⋯+hqtqn+εn.Its matrix form can be written as(4)y1y2⋮yn=∑j=1pgjxj1∑j=1pgjxj2⋮∑j=1pgjxjn+∑k=1qhktk1∑k=1qhktk2⋮∑k=1qhktkn+ε1ε2⋮εn.The regression model in equation (4) can be written as(5)y1y2⋮yn=gx1gx2⋮gxn+ht1ht2⋮htn+ε1ε2⋮εn.Furthermore, equation (5) can be written as(6)y=g+h+ε,with(7)y=y1,y2,…,yn′,g=gx1,gx2,…,gxn′,h=ht1,ht2,…,htn′,ε=ε1,ε2,…,εn′.Component regression curvegjxji,j=1,2,…,p is assumed to be smooth in the sense that it is contained in the Sobolev space gjεW2m,j=1,2,…,p.Component regression curvehktki,k=1,2…,q is approached by using the Fourier series function:hktki=bktki+1/2α0k+∑u=1Kαukcosutki,k=1,2,…,q.Combined estimator smoothing spline and Fourier series in the nonparametric regression estimation method can be obtained through two stages. The first stage is done by completing a smoothing spline component using the PLS method, and the second phase is done by completing the Fourier series components using the LS method. To complete the smoothing spline components, equation (6) is modified to the following form:(8)z=g+ε,where z=y−h. Estimation of the smoothing spline component can be obtained by PLS optimization; penalty ∑j=1pλj∫ajbjgjxji2dxj for penalized least squares optimization:(9)MingjεW2maj,bjn−1∑i=1nyi−∑j=1pgjxji+∑k=1qbktki+12α0k+∑u=1Kαukcosutki2+∑j=1pλj∫ajbjgjxji2dxj,where goodness of fit is expressed as(10)∑j=1pgjxji+bktki+12α0k+∑u=1Kαukcosutki,∑j=1pλj∫ajbjgjxji2dxj,with0<λj<∞. Furthermore, the results estimated in the first stage are substituted into regression equation (8). The second stage, to get the estimation of the components of the Fourier series, is obtained by LS optimization. Furthermore, the results of two-stage estimation are substituted into equation (6) to obtain a combined smoothing spline and Fourier series estimator in multivariable nonparametric regression.


## 3. Results and Discussion

Functiong in equation (6) is a function whose form is unknown and assumed to be smooth in the sense of being contained in space W. Then, space W can be decomposed into direct sum of two spaces W0 and W1 which are perpendicular to each other, that is, W=W0⊕W1 with W0⊥W1. The following describes the shape of the functions g and goodness of fit which is explained in the following theorem.Theorem 1.
If given goodness of fit equation (9), i.e.,(11)n−1∑i=1nyi−∑j=1pgjxji+∑k=1qbktki+12α0k+∑u=1Kαukcosutki2,then goodness of fit can be written as(12)ϕc,d=n−1z−g′z−g,where(13)z=y−h,g=Uc+Vd.The proof of Theorem1 is provided in Appendix A.Lemma 1.
If given a penalty of PLS as equation (9), a penalty can be written as(14)∑j=1pλj∫ajbjgjmxj2dxj=d′ΛVd,where(15)d=dj1dj2…djn′,V=V10…00V2…0⋮⋮⋱⋮00…Vp,Λ=λ1I0…00λ2I…0⋮⋮⋱⋮00…λpI.The proof of Lemma1 is provided in Appendix B.Then, according to Theorem1 and Lemma 1, the estimator component smoothing spline g^λ,Kx,t, Fourier series h^λ,Kx,t, and combination smoothing spline and Fourier series μ^λ,Kx,t will be searched. The whole process is described in Theorem 2.Theorem 2.
If given the regression model:(16)yi=∑j=1pgjxji+∑k=1qhktki+εi,i=1,2,…,n,then the estimator for g^λ,Kx,t,h^λ,Kx,t and μ^λ,Kx,t obtained through the optimization PLS of equation (9) is given by(17)g^λ,Kx,t=Aλ,Ky,h^λ,Kx,t=Bλ,Ky,μ^λ,Kx,t=Cλ,Ky,where(18)Aλ,Ky=Dλ,KI−T2−Dλ,K′Dλ,KT−T−1Dλ,K′−II−Dλ,Ky,Bλ,Ky=T2−Dλ,K′Dλ,KT−T−1Dλ,K′−II−Dλ,Ky,Cλ,Ky=Aλ,K+Bλ,Ky,Dλ,K=UU′M−1U−1U′M−1+VM−1I−UU′M−1U−1U′M−1,M=V+nΛ.The proof of Theorem2 is provided in Appendix C.


### 3.1. Simulation

In this research, a simulation is conducted where it aims to show the ability of a combined estimator of smoothing spline and Fourier series in multivariable nonparametric regression. Data are generated from the polynomial function for smoothing spline; trigonometric functions for Fourier series and errors are normally distributed. The simulation about mixed estimator smoothing spline and Fourier series in nonparametric regression uses sample sizen = 100, oscillation parameters (K = 1, K = 2 and K = 3). Each oscillation parameter is repeated fifty times. The regression equation designed for this simulation study is as follows:(19)yi=g1x1i+h1t1i+εi,i=1,2,…,n,with(20)g1x1i=2x1i+4x1i−0.45,h1t1i=0.1t1i+12+0.9cos2πt1in.Furthermore, generation function errorεi is obtained from the distribution N0,1, xi∼U0,1, ti∼U0,1. Scatterplot simulation data are shown in Figure 1.Figure 1
Plot of the simulation data.
(a)(b)The plot of simulation data in two dimensions is shown in Figure1; from Figure 1, it can be seen that the data tend to change on a particular subinterval such as pattern smoothing spline and data tend to have a repeating pattern like the Fourier series pattern. Based on estimates using two-stage estimate for the simulated data, the best model for the combined estimator is modeled with smoothing parameter and oscillation parameters. After obtaining the optimal smoothing parameter and oscillation parameters, the minimum GCV value is chosen. The GCV values of the oscillation parameters K = 1, K = 2, and K = 3 are given in Table 1.Table 1
Comparison value of minimum GCV for various smoothing and oscillation parameters.
RepeatOscillation parameter (K)LambdaλGCVλ,K110.540.7820.740.2330.939.79420.341.5450.540.7960.640.49730.242.0880.441.1390.640.50Table1 and Figure 2 show that the GCV minimum value is 39.79, the optimal smoothing parameter is 0.9, and the oscillation parameter is K = 1. This model gives satisfying results with GCV = 39.79; MSE = 0.13; and R2 = 99.48%.Figure 2
GCV function.The plot between the estimation results and the original simulation data is presented in Figure3. Based on Figure 3(a), the estimated data plot is very close to the original data, so this model can be used to predict very well. Furthermore, on the left side of Figure 3(b), surface plots are formed from equation (C.15), which is an equation to generate simulated data. Figures 3(a) and 3(b) show the same meaning, so that the estimation formula in equation (C.15) can be used appropriately to estimate the simulation data.Figure 3
(a) Plot ofy and y^. (b) Plot between the estimation results and the original simulation data.
(a)(b)Based on the theory, it was obtained a mixed estimator of smoothing spline and Fourier series. This theory was then proved through simulation; based on the simulation which was conducted, the theory produced a result ofR2 whose value was high. This result of the research can be developed by the other research using different estimators on mixed patterned data. Besides that, optimization methods can be used to solve other mixed estimator problems, multiresponse model, and longitudinal data.


## 3.1. Simulation

In this research, a simulation is conducted where it aims to show the ability of a combined estimator of smoothing spline and Fourier series in multivariable nonparametric regression. Data are generated from the polynomial function for smoothing spline; trigonometric functions for Fourier series and errors are normally distributed. The simulation about mixed estimator smoothing spline and Fourier series in nonparametric regression uses sample sizen = 100, oscillation parameters (K = 1, K = 2 and K = 3). Each oscillation parameter is repeated fifty times. The regression equation designed for this simulation study is as follows:(19)yi=g1x1i+h1t1i+εi,i=1,2,…,n,with(20)g1x1i=2x1i+4x1i−0.45,h1t1i=0.1t1i+12+0.9cos2πt1in.Furthermore, generation function errorεi is obtained from the distribution N0,1, xi∼U0,1, ti∼U0,1. Scatterplot simulation data are shown in Figure 1.Figure 1
Plot of the simulation data.
(a)(b)The plot of simulation data in two dimensions is shown in Figure1; from Figure 1, it can be seen that the data tend to change on a particular subinterval such as pattern smoothing spline and data tend to have a repeating pattern like the Fourier series pattern. Based on estimates using two-stage estimate for the simulated data, the best model for the combined estimator is modeled with smoothing parameter and oscillation parameters. After obtaining the optimal smoothing parameter and oscillation parameters, the minimum GCV value is chosen. The GCV values of the oscillation parameters K = 1, K = 2, and K = 3 are given in Table 1.Table 1
Comparison value of minimum GCV for various smoothing and oscillation parameters.
RepeatOscillation parameter (K)LambdaλGCVλ,K110.540.7820.740.2330.939.79420.341.5450.540.7960.640.49730.242.0880.441.1390.640.50Table1 and Figure 2 show that the GCV minimum value is 39.79, the optimal smoothing parameter is 0.9, and the oscillation parameter is K = 1. This model gives satisfying results with GCV = 39.79; MSE = 0.13; and R2 = 99.48%.Figure 2
GCV function.The plot between the estimation results and the original simulation data is presented in Figure3. Based on Figure 3(a), the estimated data plot is very close to the original data, so this model can be used to predict very well. Furthermore, on the left side of Figure 3(b), surface plots are formed from equation (C.15), which is an equation to generate simulated data. Figures 3(a) and 3(b) show the same meaning, so that the estimation formula in equation (C.15) can be used appropriately to estimate the simulation data.Figure 3
(a) Plot ofy and y^. (b) Plot between the estimation results and the original simulation data.
(a)(b)Based on the theory, it was obtained a mixed estimator of smoothing spline and Fourier series. This theory was then proved through simulation; based on the simulation which was conducted, the theory produced a result ofR2 whose value was high. This result of the research can be developed by the other research using different estimators on mixed patterned data. Besides that, optimization methods can be used to solve other mixed estimator problems, multiresponse model, and longitudinal data.


## 4. Conclusions

Based on the discussion, the following conclusions can be drawn:(a)
Based on PLS optimization, the smoothing spline component estimator is obtained asg^λ,Kx,t, Fourier series component estimator is obtained as h^λ,Kx,t, and combined smoothing spline and Fourier series estimator is obtained as μ^λ,Kx,t, which are given by(21)g^λ,Kx,t=Aλ,Ky,h^λ,Kx,t=Bλ,Ky,μ^λ,Kx,t=Cλ,Ky.(b)
The result of the simulation is that mixed estimator smoothing spline and Fourier series is good because it hasR2 = 99.48% and MSE = 0.13.



---

*Source: 4712531-2020-07-01.xml*