# Forecasting Using Information and Entropy Based on Belief Functions

Authors: Woraphon Yamaka; Songsak Sriboonchitta
Journal: Complexity
 (2020)

---

## Abstract

This paper introduces an entropy-based belief function to the forecasting problem. While the likelihood-based belief function needs to know the distribution of the objective function for the prediction, the entropy-based belief function does not. This is because the observed data likelihood is somewhat complex in practice. We, thus, replace the likelihood function with the entropy. That is, we propose an approach in which a belief function is built from the entropy function. As an illustration, the proposed method is compared to the likelihood-based belief function in the simulation and empirical studies. According to the results, our approach performs well under a wide array of simulated data models and distributions. There are pieces of evidence that the prediction interval obtained from the frequentist method has a much narrower prediction interval, while our entropy-based method performs the widest. However, our entropy-based belief function still produces an acceptable range for prediction intervals as the true prediction value always lay in the prediction intervals.


---

## Body

## 1. Introduction

Analysts are concerned with the uncertainty of the future. Reliable and timely information about future conditions is important for policy makers and planners to propose an appropriate plan and make a correct decision in the current situation. In addition, the assessment of future conditions is essential to reach reliable results and rational decisions. For example, in the economic research field, predicting the future of the economy has become a common practice (see, Mitchell [1] and Lahiri and Monokroussos [2]). In recent years, the need to forecast in economics has become even more important due to high fluctuation in the global economy. Whatever the approach used, a forecast cannot be trusted unless it is accompanied by some measure of uncertainty. Thus, in this context, it is highly desirable to have subjective probabilities or predicted intervals for the prediction [3]. In addition to the economic field, the prediction is also important in finance, medicine, engineering, and biology in order to provide an efficient solution for any current problems.Since the introduction of the Dempster–Shafer theory of belief functions [4, 5], new formal frameworks for handling uncertainty have been proposed and have attracted increasing interest in various areas. In this approach, a piece of evidence can be represented by a belief function, which is mathematically equivalent to a random set [6]. Although the approach is conceptually simple and easy to apply in various problems and models, statistical methods based on belief functions have not gained much acceptance. Kanjanatarakul et al. [7] mentioned that this approach cannot be applied easily to the complex statistical models in many areas, such as machine learning or econometrics. Thus, they suggested Shafer’s approach to statistical inference using belief functions, based on the concept of likelihood. Shafer [5] extended Dempster’s seminal work and proposed a belief function that is constructed from the likelihood function. It was recently justified by Denoeux [8] from the following three basic principles: likelihood principle, compatibility with Bayesian inference, and the least commitment principle.The likelihood-based belief function has gained an increasing interest in the last few years. Previous works using the likelihood-based belief function in forecasting models are found in many studies such as Kanjanatarakul et al. [3], Kanjanatarakul et al. [7], Thianpaen et al. [9], and Chakpitak et al. [10]. However, it is often very difficult to specify the appropriate likelihood distribution for such models. If the distribution assumption is not specified properly, it might lead to model misspecification and bring about a prediction bias. To overcome this problem, an approach based on information theory, the classical maximum entropy (ME) principle, which was introduced by Jaynes [11], is proposed in this study. The ME principle has opened the way for a whole new class of estimation methods. The estimation allows us to extract all the available information from the observed data with minimum assumptions and avoid specifying a likelihood function. In this study, we consider the Generalized Maximum Entropy (GME). We modify the likelihood-based belief functions by using an entropy function in place of the likelihood function. Therefore, we propose an approach in which a belief function BelyΘ on Θ is built from the entropy function. The advantage of this method is that the statistical inference will not assume any distribution in the plausibility plyΘ which refers to the relative entropy. Hence, the method becomes more flexible to explain a wide range of variation in the observed data.Although the likelihood-based belief function has been proved to be a reliable interval prediction method (Denoeux [8]) applications to forecasting have remained limited in some applications. The reason might be that the likelihood-based belief function requires the statistician to provide the likelihood distribution function, which is problematic when no likelihood knowledge, or only weak information, is available. The purpose of this paper is to introduce a new approach for forecasting future observation as well as providing a prediction interval. Different from the likelihood-based belief function, we replace the parametric likelihood function with the entropy function to derive the entropy-based belief function, and thus the model misspecification is avoided since no distributional assumption is made.In this paper, we further explore this proposed approach by proving that, under weak distribution function information, the predictive entropy-based belief function converges to the true probability distribution of the not yet observed data. Several simulation studies are conducted to investigate the performance of our proposed method. Finally, we apply our proposed method to Autoregressive (AR) model in predicting the growth rate of Thailand’s GDP where data is scarce and involves high uncertainty. Besides, we further explore this method by showing that it can be used for predicting future observations based on the linear regression framework using the data from the operation of a plant for the oxidation of ammonia to nitric acid, measured on 21 consecutive days. We note that our efficient method allows for accurate decision-making based on reliable forecasts in various research fields.The remainder of the paper is organized as follows. Section2 provides the background on the linear entropy-based belief function concept and describes the estimation and prediction using the entropy-based belief function methods. Section 3 proposes a simulation study and real application study in various fields to evaluate the performance of our method through autoregressive and regression models. Section 4 summarizes the paper.


## 2. Methodology

This section briefly explains the necessary background notions related to the entropy approach and introduces the connection of entropy on the belief function. The Generalized Maximum Entropy approach and parameter estimation are first presented in Sections2.1 and 2.2, respectively. The entropy-based belief function is then introduced in Section 2.3. Finally, the basic concept for predictive belief function is presented in Section 2.4.


### 2.1. Generalized Maximum Entropy Approach

In this study, we propose using a maximum entropy estimator to estimate our unknown parameterpkms β=β1,...,βK in the econometric models, say AR and linear regression models. Before we discuss this estimator for regression and its statistical properties, we briefly summarize the entropy approach. The maximum entropy concept consists of inferring the probability distribution that maximizes information entropy given a set of various constraints. Let be a proper probability of variable k on support m and the vector zk=zk1,⋯,zkM be M-dimensional vector support of coefficients k,k=1,...,K. Following Shannon’s entropy (1948), the summation of the K entropies is(1)Hp=−∑k=1K∑m=1Mpkmlogpkm,where the probabilities P are the unknown and unobservable probabilities in the entropy and ∑m=1Mpkm=1. The entropy measures the uncertainty of distribution and reaches a maximum when pk1=pk2=⋯=pkm=1/M; in other words, if no constraints (data) are imposed, HP reaches its maximum value for uniform distribution.


### 2.2. Parameter Estimation

To better understand the entropy estimation, we give a simple example. Let us consider the simple autoregressive AR model. Letyt,t=1,...,T, be observed data which are expressed as a linear combination of past observations yt−k. In the AR (K), we have(2)yt=∑k=1Kβkyt,t−k+εt,where K and βk are the autoregressive order and the autoregressive coefficient, respectively, and εt is the independent and identically distributed random white noise that is not assumed to have any distribution and is assumed to be an uncorrelated random variable yt−k.To apply this concept to be an estimator of the regression model, we generalize the maximum entropy to the inverse problem in the regression framework. Rather than searching for the point estimatesβ, we can view these unknown parameters as expectations of random variables with M support values for each estimated parameter value k, Z=z1,...,zK, where zk=zk1,..,zkM for all k=1,...K, where 2≤M<∞, and zk1 and zkM are the lower and upper bounds of βk. Thus, we can express parameter βk as(3)βk=∑m=1Mpkmzkm,where pkm are the M-dimensional estimated probability defined on the support vector zkm. Next, similar to the above expression, εt is also constructed as the mean value of some random variable vt. Here, each εt is assumed to be a random vector with finite and discrete random variable with M support value, vt=vt1,...,vtM. Let wt be a M dimensioned proper probability weights defined on the set vtm such that(4)εt=∑m=1Mwtmvtm.Using the reparameterized unknownsβk and εt, we can rewrite equation (2) as(5)yt=∑k=1Kyt,t−k∑m=1Mpkmzkm+∑m=1Mwtmvtm,where the vector support zkm and vtm are convex set that is symmetric around zero. Then, we can construct our Generalized Maximum Entropy (GME) estimator as(6)Hp,w=argmaxp,wHp+Hw≡−∑k=1K∑m=1Mpkmlogpkm−∑t=1T∑m=1Mwtmlogwtm,subject to(7)yt=∑k=1Kyt,t−k∑m=1Mpkmzkm+∑m=1Mwtmvtm,(8)∑m=1Mpkm=1,∑m=1Mwtm=1,where pkm and wtm are on the interval [0, 1].This optimization problem can be solved using the Lagrangian method which takes the following form:(9)L=Hp,w+∑t=K+1Tλ1t′yt−∑k=1Kyt,t−k∑m=1Mpkmzkm−∑m=1Mwtmvtm+∑k=1Kλ2k′1−∑m=1Mpkm+∑t=K+1Tλ3t′1−∑m=1Mwtm,where the unknown multipliers λ1,λ2,λ3 are T×1, K×1, and T×1 vectors of Lagrangian multipliers. Thus, the resulting first-order conditions are(10)∂Lpkm=−logpkm−1−∑t=K+1Tyt−kλ1t′zkm−λ2k′=0,∂Lwtm=−logwtm−1−λ1t′vtm−λ3t′=0,∂Lλ1t=−yt−∑k=1Kyt,t−k∑m=1Mpkmzkm−∑m=1Mwtmvtm=0,∂Lλ2k=1−∑m=1Mpkm=0,∂Lλ3t=1−∑m=1Mwtm=0.Thus, we have(11)pkm=exp−1−λ⌢2k−∑t=K+1Tyt−kλ⌢1tzkm=0,(12)wtm=exp−1−λ⌢3t−λ⌢1tvtm=0.Since the constraint requires that∑m=1Mpkm=1 and ∑m=1Mwtm=1, we sum both pkm and wtm over m, and we have(13)∑m=1Mpkm=exp−1−λ⌢2k∑m=1Mexp−∑t=K+1Tyt−kλ⌢1tzkm=1,(14)∑m=1Mwtm=exp−1−λ⌢3t∑m=1Mexp−λ⌢1tvtm=1,where λ⌢ is the estimated multiplier. To estimate the Lagrangian multipliers, we can substitute the candidate probabilities, pkm and wtm, into the original Lagrangian (equation (9)) Then, the estimated λ⌢ can be obtained by differentiating the concentrated Lagrangian (equation (9)) (conditional on the candidates pkm and wtm) with respect to λ and set it to be equal be zero. Then, substitute these estimated parameters in equations (13)-(14). Note that exp−1−λ⌢2k and −1−λ⌢3t are constants for a given parameter, so the optimal probability fd15equations (11) and (12) can be rewritten as(15)p⌢km=exp−zkm∑T=K+1Tyt−kλ⌢1t∑m=1Mexp−zkm∑t=K+1Tyt−kλ⌢1t,w⌢tm=exp−λ⌢1tv1m∑m=1Mexp−λ⌢1tv1m,Summing up the above equations, we maximize the joint-entropy objective equation (6) subject to the regression equation (7), by adding restrictions equation (8). The solution to this maximization problem is unique by forming the Lagrangian and solving for the first-order conditions to obtain the optimal solutions p⌢km, w⌢tm, and λ⌢1 (estimated values). Then, these estimated probabilities are used to derive the point estimates for the regression coefficients and error term, see equations (3) and (4).Example 1.
To better understand the entropy estimation, we give a simple example. Let consider the first-order linear autoregressive AR(1) model as defined by(16)yt=β0+β1yt−1+εt,εt∼N0,1.
The true values forβ0 and β1 are set to be 1 and −1, respectively. The error is drawn randomly from a normal distribution with mean 0 and variance 1. We generate equation (16) using sample size T = 10. The support spaces of β0 and β1 are defined as −5,0,5 while εt is defined as −3,0,3. The simulated data are shown in Figure 1. And the results are reported in Tables 1 and 2.
To estimate equation (16), we need to construct our GME problem as(17)Hp,w=−∑k=12∑m=13pkmlogpkm−∑i=110∑m=13wimlogwim,yt=∑m=13p1mz1m+∑m=13p2mz2myt,t−1+∑m=13wimvim,t=1,...,10,∑m=13pkm=1,∑m=13wim=1,where pkm and wtm are the parameter estimates in this problem, see equations (3)-(4).Figure 1
Example data.Table 1
Estimated error terms of the example AR (1) model.
Errorwt1wt2wt3vtmεt=∑m=13wtmvtm10.34420.33320.3326[3, 0, 3]−0.065120.55390.29210.1540[3, 0, 3]−1.199330.12030.27070.1540[3, 0, 3]1.465640.38780.33050.2817[3, 0, 3]−0.318450.47610.31520.2087[3, 0, 3]−0.802160.33770.33330.3290[3, 0, 3]−0.026170.23780.32320.4390[3, 0, 3]0.603880.32790.33330.3388[3, 0, 3]0.032990.40590.32840.2657[3, 0, 3]−0.4209100.17100.30060.5284[3, 0, 3]1.0719Table 2
Estimated coefficients of the example AR(1) model.
pk1pk2pk2zkm=zk1,zk2,zk3EstimatesTrueβ0=∑m=13p1mz1m0.21730.31790.4648[−5, 0, 5]1.23701β1=∑m=13p2mz2m0.42560.32550.2489[−5, 0, 5]−0.8831−1


### 2.3. The Entropy-Based Belief Function

Following the three basic principles of the likelihood-based belief function, consisting the likelihood principle, compatibility with Bayesian inference, and the least commitment principle, which are justified by Denoeux [8], we modify these approaches by using an entropy function as the substitute for the likelihood function. Let P∈Θ be the parameter of interest, and we propose an approach in which a belief function BelyΘ on Θ is built from the entropy function. In this case, the entropy ratio refers to a “relative plausibility,” and the contour function can thus be readily calculated as(18)plyΡ=HΡHΡ¯,where P=pkm,wtm in which pkm and wtm are proper probability vector mass functions on the interval [0, 1] and HP is(19)HP=−∑k=1K∑m=1Mpkmlogpkm−∑t=1T∑m=1Mwtmlogwtm,subject to(20)yt=∑k=1Kyt,t−k∑m=1Mpkmzkm+∑m=1Mwtmvtm,(21)∑m=1Mpkm=1,∑m=1Mwtm=1.Meanwhile,HP¯ is defined as(22)HΡ¯=supΡ∈ΘHΡ,and the marginal contour functions on each individual parameter are(23)plpkm=supp−km,wtmplP,plwtm=suppkm,w−tmplP.Subject to the additional constraints equations (20)-(21), this belief function is called the entropy-based belief function on Θ induced by y. The corresponding plausibility function can be estimated by plyP as(24)PlyΘC=supΡ∈CplyΡ,for all C⊆Θ. The focal sets of BelyΘ are the level sets of plyP defined as(25)Γyω=Ρ∈ΘplyΡ≥ω,where ω is uniformly distributed in 0,1. This belief function BelyΘ is equivalent to the random set induced by the Lebesgue measure λ on 0,1 and the multivalued mapping Γy from 0,1⟶2Θ [3].


### 2.4. The Basic Method for Predictive Belief Function

To forecast future dataYf=YT+1,...,YT+h, the sampling model used by Dempster [12] is introduced here. In this model, the forecast data YT+1 is expressed as a function of the proper probability mass function P which is obtained by past observed data y=y1,...,yT and an unobserved auxiliary S∈S variable with known probability measure μ not depending on P. Note that S is the sample space of s:(26)Yf=φP,S,where φ is defined in such a way that the distribution of Yf is conditional on fixed P. When Yf is a continuous random variable, equation (26) can be computed by(27)Yf=Gy,P−1S=infyfFy,pyf≥S,where Gy,P−1 is the inverse conditional cumulative distribution function (cdf) of Yfy and s is uniformly distributed in 0,1. We note that, in the likelihood-based belief function case, Gy,P−1 becomes Gy,β−1 which is the inverse of some conditional cumulative distribution functions, say normal, student-t, or other parametric distributions. However, in the entropy-based belief function case, we do not need any distribution assumptions on Gy,P−1. Let us compose Γy:0,1×S⟶2Θ with φ, and we get a new multivalued mapping Γy′ form 0,1×S⟶2Y defined as(28)Γy′:0,1×S⟶2Y,ω,s⟶φΓyω,s.The predictive belief functionBely and plausibility Ply are then induced by the multivalued mapping Γy and λ⊗μ on 0,1×S as follows:(29)BelyYC=λ⊗μωi,siφyΓyωi,si⊆C,PlyYC=λ⊗μωi,siφyΓyωi,si∩C≠∅,for all C⊆R.


## 2.1. Generalized Maximum Entropy Approach

In this study, we propose using a maximum entropy estimator to estimate our unknown parameterpkms β=β1,...,βK in the econometric models, say AR and linear regression models. Before we discuss this estimator for regression and its statistical properties, we briefly summarize the entropy approach. The maximum entropy concept consists of inferring the probability distribution that maximizes information entropy given a set of various constraints. Let be a proper probability of variable k on support m and the vector zk=zk1,⋯,zkM be M-dimensional vector support of coefficients k,k=1,...,K. Following Shannon’s entropy (1948), the summation of the K entropies is(1)Hp=−∑k=1K∑m=1Mpkmlogpkm,where the probabilities P are the unknown and unobservable probabilities in the entropy and ∑m=1Mpkm=1. The entropy measures the uncertainty of distribution and reaches a maximum when pk1=pk2=⋯=pkm=1/M; in other words, if no constraints (data) are imposed, HP reaches its maximum value for uniform distribution.


## 2.2. Parameter Estimation

To better understand the entropy estimation, we give a simple example. Let us consider the simple autoregressive AR model. Letyt,t=1,...,T, be observed data which are expressed as a linear combination of past observations yt−k. In the AR (K), we have(2)yt=∑k=1Kβkyt,t−k+εt,where K and βk are the autoregressive order and the autoregressive coefficient, respectively, and εt is the independent and identically distributed random white noise that is not assumed to have any distribution and is assumed to be an uncorrelated random variable yt−k.To apply this concept to be an estimator of the regression model, we generalize the maximum entropy to the inverse problem in the regression framework. Rather than searching for the point estimatesβ, we can view these unknown parameters as expectations of random variables with M support values for each estimated parameter value k, Z=z1,...,zK, where zk=zk1,..,zkM for all k=1,...K, where 2≤M<∞, and zk1 and zkM are the lower and upper bounds of βk. Thus, we can express parameter βk as(3)βk=∑m=1Mpkmzkm,where pkm are the M-dimensional estimated probability defined on the support vector zkm. Next, similar to the above expression, εt is also constructed as the mean value of some random variable vt. Here, each εt is assumed to be a random vector with finite and discrete random variable with M support value, vt=vt1,...,vtM. Let wt be a M dimensioned proper probability weights defined on the set vtm such that(4)εt=∑m=1Mwtmvtm.Using the reparameterized unknownsβk and εt, we can rewrite equation (2) as(5)yt=∑k=1Kyt,t−k∑m=1Mpkmzkm+∑m=1Mwtmvtm,where the vector support zkm and vtm are convex set that is symmetric around zero. Then, we can construct our Generalized Maximum Entropy (GME) estimator as(6)Hp,w=argmaxp,wHp+Hw≡−∑k=1K∑m=1Mpkmlogpkm−∑t=1T∑m=1Mwtmlogwtm,subject to(7)yt=∑k=1Kyt,t−k∑m=1Mpkmzkm+∑m=1Mwtmvtm,(8)∑m=1Mpkm=1,∑m=1Mwtm=1,where pkm and wtm are on the interval [0, 1].This optimization problem can be solved using the Lagrangian method which takes the following form:(9)L=Hp,w+∑t=K+1Tλ1t′yt−∑k=1Kyt,t−k∑m=1Mpkmzkm−∑m=1Mwtmvtm+∑k=1Kλ2k′1−∑m=1Mpkm+∑t=K+1Tλ3t′1−∑m=1Mwtm,where the unknown multipliers λ1,λ2,λ3 are T×1, K×1, and T×1 vectors of Lagrangian multipliers. Thus, the resulting first-order conditions are(10)∂Lpkm=−logpkm−1−∑t=K+1Tyt−kλ1t′zkm−λ2k′=0,∂Lwtm=−logwtm−1−λ1t′vtm−λ3t′=0,∂Lλ1t=−yt−∑k=1Kyt,t−k∑m=1Mpkmzkm−∑m=1Mwtmvtm=0,∂Lλ2k=1−∑m=1Mpkm=0,∂Lλ3t=1−∑m=1Mwtm=0.Thus, we have(11)pkm=exp−1−λ⌢2k−∑t=K+1Tyt−kλ⌢1tzkm=0,(12)wtm=exp−1−λ⌢3t−λ⌢1tvtm=0.Since the constraint requires that∑m=1Mpkm=1 and ∑m=1Mwtm=1, we sum both pkm and wtm over m, and we have(13)∑m=1Mpkm=exp−1−λ⌢2k∑m=1Mexp−∑t=K+1Tyt−kλ⌢1tzkm=1,(14)∑m=1Mwtm=exp−1−λ⌢3t∑m=1Mexp−λ⌢1tvtm=1,where λ⌢ is the estimated multiplier. To estimate the Lagrangian multipliers, we can substitute the candidate probabilities, pkm and wtm, into the original Lagrangian (equation (9)) Then, the estimated λ⌢ can be obtained by differentiating the concentrated Lagrangian (equation (9)) (conditional on the candidates pkm and wtm) with respect to λ and set it to be equal be zero. Then, substitute these estimated parameters in equations (13)-(14). Note that exp−1−λ⌢2k and −1−λ⌢3t are constants for a given parameter, so the optimal probability fd15equations (11) and (12) can be rewritten as(15)p⌢km=exp−zkm∑T=K+1Tyt−kλ⌢1t∑m=1Mexp−zkm∑t=K+1Tyt−kλ⌢1t,w⌢tm=exp−λ⌢1tv1m∑m=1Mexp−λ⌢1tv1m,Summing up the above equations, we maximize the joint-entropy objective equation (6) subject to the regression equation (7), by adding restrictions equation (8). The solution to this maximization problem is unique by forming the Lagrangian and solving for the first-order conditions to obtain the optimal solutions p⌢km, w⌢tm, and λ⌢1 (estimated values). Then, these estimated probabilities are used to derive the point estimates for the regression coefficients and error term, see equations (3) and (4).Example 1.
To better understand the entropy estimation, we give a simple example. Let consider the first-order linear autoregressive AR(1) model as defined by(16)yt=β0+β1yt−1+εt,εt∼N0,1.
The true values forβ0 and β1 are set to be 1 and −1, respectively. The error is drawn randomly from a normal distribution with mean 0 and variance 1. We generate equation (16) using sample size T = 10. The support spaces of β0 and β1 are defined as −5,0,5 while εt is defined as −3,0,3. The simulated data are shown in Figure 1. And the results are reported in Tables 1 and 2.
To estimate equation (16), we need to construct our GME problem as(17)Hp,w=−∑k=12∑m=13pkmlogpkm−∑i=110∑m=13wimlogwim,yt=∑m=13p1mz1m+∑m=13p2mz2myt,t−1+∑m=13wimvim,t=1,...,10,∑m=13pkm=1,∑m=13wim=1,where pkm and wtm are the parameter estimates in this problem, see equations (3)-(4).Figure 1
Example data.Table 1
Estimated error terms of the example AR (1) model.
Errorwt1wt2wt3vtmεt=∑m=13wtmvtm10.34420.33320.3326[3, 0, 3]−0.065120.55390.29210.1540[3, 0, 3]−1.199330.12030.27070.1540[3, 0, 3]1.465640.38780.33050.2817[3, 0, 3]−0.318450.47610.31520.2087[3, 0, 3]−0.802160.33770.33330.3290[3, 0, 3]−0.026170.23780.32320.4390[3, 0, 3]0.603880.32790.33330.3388[3, 0, 3]0.032990.40590.32840.2657[3, 0, 3]−0.4209100.17100.30060.5284[3, 0, 3]1.0719Table 2
Estimated coefficients of the example AR(1) model.
pk1pk2pk2zkm=zk1,zk2,zk3EstimatesTrueβ0=∑m=13p1mz1m0.21730.31790.4648[−5, 0, 5]1.23701β1=∑m=13p2mz2m0.42560.32550.2489[−5, 0, 5]−0.8831−1


## 2.3. The Entropy-Based Belief Function

Following the three basic principles of the likelihood-based belief function, consisting the likelihood principle, compatibility with Bayesian inference, and the least commitment principle, which are justified by Denoeux [8], we modify these approaches by using an entropy function as the substitute for the likelihood function. Let P∈Θ be the parameter of interest, and we propose an approach in which a belief function BelyΘ on Θ is built from the entropy function. In this case, the entropy ratio refers to a “relative plausibility,” and the contour function can thus be readily calculated as(18)plyΡ=HΡHΡ¯,where P=pkm,wtm in which pkm and wtm are proper probability vector mass functions on the interval [0, 1] and HP is(19)HP=−∑k=1K∑m=1Mpkmlogpkm−∑t=1T∑m=1Mwtmlogwtm,subject to(20)yt=∑k=1Kyt,t−k∑m=1Mpkmzkm+∑m=1Mwtmvtm,(21)∑m=1Mpkm=1,∑m=1Mwtm=1.Meanwhile,HP¯ is defined as(22)HΡ¯=supΡ∈ΘHΡ,and the marginal contour functions on each individual parameter are(23)plpkm=supp−km,wtmplP,plwtm=suppkm,w−tmplP.Subject to the additional constraints equations (20)-(21), this belief function is called the entropy-based belief function on Θ induced by y. The corresponding plausibility function can be estimated by plyP as(24)PlyΘC=supΡ∈CplyΡ,for all C⊆Θ. The focal sets of BelyΘ are the level sets of plyP defined as(25)Γyω=Ρ∈ΘplyΡ≥ω,where ω is uniformly distributed in 0,1. This belief function BelyΘ is equivalent to the random set induced by the Lebesgue measure λ on 0,1 and the multivalued mapping Γy from 0,1⟶2Θ [3].


## 2.4. The Basic Method for Predictive Belief Function

To forecast future dataYf=YT+1,...,YT+h, the sampling model used by Dempster [12] is introduced here. In this model, the forecast data YT+1 is expressed as a function of the proper probability mass function P which is obtained by past observed data y=y1,...,yT and an unobserved auxiliary S∈S variable with known probability measure μ not depending on P. Note that S is the sample space of s:(26)Yf=φP,S,where φ is defined in such a way that the distribution of Yf is conditional on fixed P. When Yf is a continuous random variable, equation (26) can be computed by(27)Yf=Gy,P−1S=infyfFy,pyf≥S,where Gy,P−1 is the inverse conditional cumulative distribution function (cdf) of Yfy and s is uniformly distributed in 0,1. We note that, in the likelihood-based belief function case, Gy,P−1 becomes Gy,β−1 which is the inverse of some conditional cumulative distribution functions, say normal, student-t, or other parametric distributions. However, in the entropy-based belief function case, we do not need any distribution assumptions on Gy,P−1. Let us compose Γy:0,1×S⟶2Θ with φ, and we get a new multivalued mapping Γy′ form 0,1×S⟶2Y defined as(28)Γy′:0,1×S⟶2Y,ω,s⟶φΓyω,s.The predictive belief functionBely and plausibility Ply are then induced by the multivalued mapping Γy and λ⊗μ on 0,1×S as follows:(29)BelyYC=λ⊗μωi,siφyΓyωi,si⊆C,PlyYC=λ⊗μωi,siφyΓyωi,si∩C≠∅,for all C⊆R.


## 3. Simulation and Application: Entropy-Based Belief Function

Let us now turn our attention to examining the performance of the entropy-based belief function. In this section, the inference and forecasting methodology outlined in the previous section will be applied to the simulated data and real data. We apply the entropy-based belief function to the forecasting AR and regression models which will be presented in Sections3.1 and 3.2, respectively.


### 3.1. The Entropy-Based Belief Function in Autoregressive Model Problem

#### 3.1.1. Prediction

As we have seen in Section2, the estimation problem is to make statements about some probability after observing some data. To describe the prediction method for the AR model, let us consider the case where the K = 1; thus, the first-order linear autoregressive AR (1) model is defined by(30)YT+1=β1+β2yT+εT,where YT+1 is a predicted value of T+1 which is not yet observed and yT is the observed value at time T.The main idea is to estimateYT+1 as a function of β, which is viewed as an expectation of probability with M support and some pivotal variable S, whose distribution does not depend on β. We can write equivalently(31)YT+1=φyβ,s=β1+β2yT+F^−1s,s∼U0,1,where F^⋅−1 is the quantile function of empirical distribution based on the estimated residuals, ε1,...,εT. Then, in the case of entropy approach, we rewrite equation (30) as(32)YT+1=∑m=1Mp1mz1m+∑m=1Mp2mz2myT+F^−1s,where m is the number of supports of z and v. Consequently, in the entropy estimation, we rewrite the focal set equation (31) as(33)YT+1=φyP,s=∑m=1Mp1mz1m+∑m=1Mp2mz2myT+F^−1s.To forecastYT+1yT, the predictive belief function and plausibility function can then be approximated using Monte Carlo simulation as in the following procedure (Kanjanatarakul et al. [3] and Kanjanatarakul et al. [7]:(1)
Drawωi and si independently from uniform [0, 1] N draws. Then, compute (or approximate) the focal set which is the interval defined by the following lower and upper bounds:(34)φyΓyω,s=φP,splyP≥ωi=y¯isi,ωi,y¯isi,ωi,wherey¯isi,ωi and y¯isi,ωi are the lower and upper bounds which can be computed using a constrained nonlinear optimization algorithm. This optimization problem can be solved using the nonlinear optimization:(35)y¯isi,ωi=minφyP,ωi,si,(36)y¯isi,ωi=maxφyP,ωi,si,which subject to(37)plyΡ≥ωi,yt=∑m=1Mp1mz1m+∑m=1Mp2mz2myt,t−1+∑m=1Mwtmvtm,∑m=1Mp1m=1,∑m=1Mp2m=1,∑m=1Mwtm=1.(2)
We can then approximate the predictive belief (Bely) and plausibility (ply) functions on Yf as(38)BelyC=1N#i∈1,...,NφyΓyωi,si⊆C,PlyC=1N#i∈1,...,NφyΓyωi,si∩C≠∅,or allC⊆Θ. These lower and upper cdfs of the predictive belief function are approximated using N = 5000 randomly generated focal sets (Algorithm 1).Algorithm 1: Monte Carlo algorithm for approximating a predictive entropy-based belief function (random interval case).
Require: Desired number of focal sets N, number of support vector M, support spaces of coefficients zk=zk1,...,zkM and error terms vt=vt1,...,vtM.fori=1toNdoDrawωi,si from uniform [0, 1]Search fory¯i=minφyP,ωi,si such that plyP≥ωi,yt=∑m=1Mp1mz1m+∑m=1Mp2mz2myt−1+∑m=1Mwtmvtmand∑m=1Mp1m=1,∑m=1Mp2m=1,∑m=1Mwtm=1.Search fory¯isi,ωi=maxφyP,ωi,si such that plyP≥ωi,yt=∑m=1Mp1mz1m+∑m=1Mp2mz2myt−1+∑m=1Mwtmvtmand∑m=1Mp1m=1,∑m=1Mp2m=1,∑m=1Mwtm=1Bi⟵y¯i,y¯iend forY^T+1=∑i=1Ny¯i+y¯i/N


#### 3.1.2. Experiment Study

In this section, we conduct a simulation and experiment studies to investigate the finite sample performance of the proposed Shannon entropy-based belief methods. The simulated data are constructed from the AR model using different error distributions consisting of normal, Student-t, skewed Student-t, and uniform. We note that the normal and student-t are selected as symmetric distribution, while skewed student-t and uniform are typical examples of highly skewed distribution and flat distribution, respectively. In these experiments, we consider three competing likelihood functions: normal, student-t, and skewed student-t likelihoods.To compare these likelihood-based belief functions, the one-step-ahead forecast is computed from simulated data and compared with the true prediction value. We note that, as the functionply=Ply has a unique maximum y˜, it may be taken as a point prediction of the AR (1) model-based belief function approach. This is to say y˜ can be viewed as the most plausible value of Y given the past information. Thus, we need a criterion to gauge the performance of the method. Here, the mean square error MSE method is employed, and it can be defined as(39)MSE=R−1∑r=1RYtrue,r−Yf,r2,where R = 20 is the number of repetitions, Ytrue is the true value, and Yf=y˜ is the predicted value.In this Monte Carlo simulation, we consider sample sizeT=20 and T=40. R = 20 samples are generated for each case. To make a fair comparison, the error terms are generated from N0,σ2=1, t0,σ2=1,ν=4, skew−t0,σ2=1,ν=4,ξ=1.5, and U−2,2. Then, the sampling experiments are based on the AR (1) model:(40)yt=0.5−0.5yt−1+εt.In this experiment, the first 20 observations are used to predict the 21st observation. Likewise, the first 40 observations are used to predict the 41st observation. Then, the most plausible predicted values,Yf21st and Yf41st, are compared with Ytrue21st=0.5−0.5yT+εT+1,T=20, and Ytrue41st=0.5−0.5yT+εT+1,T=40, respectively. In the case of the Shannon entropy, the support z1m is initially set to (−5, 0, 5) and the supports for vtm to (−3σ, 0, 3σ), where σ is computed from the conventional AR model. Table 3 reports the prediction error MSE of our considered estimators. Consider the performance of the proposed estimator when the errors are generated from a uniform distribution, and we found that the error of the Shannon entropy-based belief function is smaller than the other methods over the one-step-ahead forecast. In addition, the performance of this method is better when the sample size increases from 20 to 40.Table 3
Forecasting performance.
Y˜fNormal likelihood-based belief functionStudent-t likelihood-based belief functionSkewed Student-t likelihood-based belief functionShannon entropy-based belief functionRenyi entropy-based belief functionTsallis entropy-based belief functionU−2,2T = 20MSE0.9346 (0.9541)1.2327 (1.8079)13.8763 (15.4710)0.5784 (0.5824)0.5796 (0.5831)0.5788 (0.5826)T = 40MSE0.6891 (0.7552)1.6996 (1.6252)8.7598 (12.2013)0.2596 (0.5914)0.2608 (0.5948)0.2605 (0.5953)N0,1T = 20MSE0.4601 (0.5491)1.2275 (1.4574)2.0607 (2.4733)0.5582 (0.7576)0.5688 (0.7589)0.5591 (0.7601)T = 40MSE0.4995 (0.3136)1.4068 (1.4647)1.2092 (1.1063)0.5294 (0.3787)0.5301 (0.3793)0.5312 (0.3798)t0,1,4T = 20MSE1.2850 (1.476)1.1815 (2.6127)1.6660 (1.8974)1.2191 (1.7849)1.2115 (1.7858)1.2114 (1.7857)T = 40MSE0.7604 (0.4309)0.6142 (0.3654)0.8783 (1.1256)0.7538 (0.2542)0.7542 (0.2550)0.7542 (0.2550)skew−t0,1,4,1.5T = 20MSE0.7278 (0.9846)2.6995 (2.0442)0.5878 (1.1307)0.6699 (1.6189)0.6703 (1.6208)0.6700 (1.6200)T = 40MSE0.5520 (0.4491)1.4772 (1.2513)0.4443 (0.8051)0.4513 (0.3370)0.4559 (0.3389)0.4564 (0.3401)() is standard deviation onMSE. The bold numbers present the best prediction result.Next, we investigate the performance of the proposed estimator when the errors are generated from normal, student-t, and skewed student-t distributions and compare with the parametric likelihood-based belief function. We observe that the overall result is different in Table 3. We find that the Shannon entropy-based belief function seems to outperform the likelihood-based belief function in terms of lowest MSE of the prediction when the U−2,2 error distribution is assumed. When the errors are generated from other parametric distributions, we find that the likelihood-based belief function method with the correct distribution function always has the best performance. However, we notice that our Shannon entropy-based belief function performs the second-best prediction over the wide range of error distributions.Thus, in this simulation study, we can conclude that our method performs well particularly when the error is unknown or has a uniform distribution. Nonetheless, when the error is generated from known distribution, the parametric likelihood-based belief function outperforms the entropy. This means that if the distribution is known and we estimate the model based on the true distribution, the accurate result is obtained. However, the error distribution is sometimes unknown in practice, these results confirm the advantage of the Shannon entropy-based belief function, which performs uniformly well over a wide range of error distributions.Furthermore, it would be more interesting to consider other entropy measures such as Renyi entropy [13] and Tsallis entropy [14]; we thus introduce these two entropy-based belief functions to investigate the performance of various entropy measures. We note that Renyi entropy is the generalization of Shannon entropy while Tsallis entropy is nonlogarithmic and is obtained through the joint generalization of the averaging procedures and the concept of information gain. These two entropies are indexed by an order d. The Renyi entropy is formulated as(41)HdRp=−11−d∑k=1K∑m=1Mlogpkmd,while Tsallis entropy can be defined as(42)HdTp=−c∑k=1K∑m=1Mpkmd−11−d,where the value of c is a positive constant and depends on the particular units used. For simplicity, we set c=1 and order d (see [15]). Note that both Renyi and Tsallis entropies become the Shannon entropy as a special case when d=2. To construct the Renyi entropy-based belief function and Tsallis entropy-based belief function, we can replace Shannon entropy in the relative plausibility (equation (18)) by Renyi and Tsallis, respectively.The results of these three entropies for comparison are shown in the last three columns of Table1, and it is evident that the Shannon entropy-based belief function still performs the best prediction performance in this simulation data. However, the performances of these three entropies are not much different in terms of MSE.Example 2.
Application to predicting GDP growth by the AR model.
In this example, the real data are also used to investigate the performance of our method. The dataset considered, derived from the Thomson Reuter Datastream and World Bank database, consist of yearly data, from the ending of 1995 to 2018, of Gross Domestic Product (GDP). The examined data series is transformed into the growth rate and plotted in Figure2. Then, we perform the Augmented Dickey–Fuller unit roots test on GDP growth and determine the lagging order number of the AR model according to Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). The ADF-statistic result is −4.337, which is greater than the critical values at 10% significance level and means that GDP is stationary. Then, the lag selection for the AR model is shown in Table 4. The result shows that the AR model with lag 1 provides the lowest AIC and BIC.
To evaluate the performance of prediction, the forecast evaluation period is from 2014 to 2018. We perform a recursive forecasting to update GDP growth year by year, totally 5 years, which give us enough data points to evaluate the out-of-sample forecast performance. For this evaluation period, we compute the root mean square forecast error (RMSE) and mean square error (MAE) of forecasts at each forecast horizon (from one to five years ahead). We can compute the RMSE and MAE as in the following:(43)MAE=1T∑t=1Tet,RMSE=1T∑t=1Tet2,where T = 5 and e=Ytrue−Yf.
The forecast comparison of the AR(1) model with the likelihood-based approach and the entropy-based approach, as well as the conventional frequentist method, performed in out-of-sample is reported in Table5. The out-of-sample forecasts made for the 2014–2018 period are reflected by the most plausible prediction values and their large upper-lower prediction intervals. The true prediction values are shown in the first column. The estimated conditional expectation as well as the frequentist 95% prediction intervals are shown in column 2. Columns 3–7 show the most plausible predictions and quantile intervals for the normal, student-t, skewed student-t, and the three entropy-based belief functions, respectively.
We can now consider the performance of different methods. First, we evaluate the entropy-based method regarding the prediction value. Second, we compare the performance regarding the range of the prediction intervals.
In the first evaluation, the smaller theMAE and RMSE, the superior the corresponding model’s performance. According to Table 5, we learn that all three entropy-based belief functions outperform the other likelihood-based methods. We can observe that the values of RMSE and MAE of entropy-based belief functions are a bit smaller than those of the likelihood-based belief functions. Again, the Shannon entropy still provides lower RMSE and MAE compared to the other two entropies. We expect that if the real data has a distribution and the assumption of the likelihood-based belief function is not correctly specified, a less accurate prediction is obtained. In addition, small sample size may affect the accuracy of the estimated parameters. Hence, in this study, we conclude that entropy-based belief function for AR (1) performs well in the out-of-sample forecasts relative to other competing methods.
In the second evaluation, for the prediction intervals for Thai GDP growth, the methods suggest reliable prediction intervals around the predicted value. We can observe that all prediction interval methods for the AR (1) model are reliable as the true values exist along with the range of the intervals, except for the 95% frequentist approach. The prediction interval based on the 95% frequentist for Thai GDP growth in 2014 does not cover the true value. This result reveals that the conventional frequentist approach may not provide the appropriate prediction value.
Regarding the prediction intervals, we can calculate the average of the prediction intervals for making a comparison. The results portray the difference between the methods exists. The frequentist method has a very narrow mean of the prediction intervals, while our entropy-based method performs the widest. We suspect that our approach relies on the empirical distribution assumption on the error distribution and leads to a larger simulated error in the Monte Carlo algorithm for approximating a predictive entropy-based belief function process. However, this experiment shows that our method produces an acceptable range for prediction intervals as the true values always lie in the intervals.
We now use the entropy-based belief function method to predict Thai economic growth in the future. The prediction is very important because of its relevance to the current policy debate on whether Thai GDP growth will be up or down. The forecasts made for 2019 is based on scant data, which is reflected by large upper-lower cdf intervals (Figure3). Figure 3 presents the lower and upper cumulative density functions (cdfs) for the prediction problem obtained from (black solid line) and (pink solid line). We also plot the vertical blue dotted line corresponding to the most plausibility value together with the quantile intervals for the GDP growth in 2019. We can observe that the value of growth rate is around 0.03961 (3.961%) in 2019 and the prediction interval 95% ranges (−0.0391 – 0.0558) (pink dashed line) and (0.0630 – 0.1124) (black dashed line).
Furthermore, to show the accuracy of our estimation in this study, we show the contour plots of plausibilityplyθ in two-dimensional parameter subspaces of each method, as shown in the Appendix. These contour plots show two-dimensional slices of the contour function, where one of the three parameters is fixed to its maximum normal likelihood, two of the four parameters are fixed to its maximum student-t likelihood, three of the five parameters are fixed to its maximum skewed student-t likelihood, and z+v×m−2 of the z+v×m probability parameters are fixed to its maximum entropy. The results of these contour plots show that the estimated parameters of each method are likely to reach the maximum plausibility, as presented by the red cross. Consider the contour plots of plyθ for our proposed method, entropy-based belief functions. We can summarize the process graphically by plotting the optimization on a contour plot of the plausibility plyP. We can observe that, for example, the estimated probabilities p11 and p12 reach a maximum plausibility plyP at the red cross.Figure 2
GDP growth rate of Thailand.Table 4
Lag selection for the AR model.
AR (1)AR (2)AR (3)AR (4)AIC−104.41−102.69−100.75−100.43BIC−101.00−98.14−95.076−93.61Table 5
Out-of-sample forecasting performance.
TRUEYfARYfnormYfstdYfsstdYfshannonYfRenyiYftsallisForecast 20140.0090.03570.03670.03310.03670.00420.00390.0030Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[0.0123, 0.0838] 0.0715[−0.0191, 0.0918] 0.1109[−0.0157, 0.1072] 0.1229[−0.0054, 0.1172] 0.1226[−0.0226, 0.0381] 0.0607[−0.0225, 0.0387] 0.0613[−0.0218, 0.0383] 0.0601Forecast 20150.0300.03310.03280.03380.03480.04040.04190.0410Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0150, 0.0812] 0.0962[−0.0187, 0.0882] 0.1069[−0.0153, 0.1006] 0.1159[0.0016, 0.1035] 0.1019[−0.0496, 0.1305] 0.1801[−0.0493, 0.1351] 0.1844[−0.0490, 0.1309] 0.1799Forecast 20160.0330.03400.03520.03380.03620.03980.04010.0407Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0128, 0.0808] 0.0936[−0.0162, 0.0887] 0.1049[−0.0133, 0.0976] 0.1109[0.0001, 0.1107] 0.1106[−0.0467, 0.1252] 0.1719[−0.0433, 0.1212] 0.1645[−0.0439, 0.1231] 0.1670Forecast 20170.0390.03410.03440.03390.03530.03950.03850.0386Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0115, 0.0798] 0.0913[−0.0162, 0.0838] 0.1000[−0.0116, 0.0950] 0.1066[0.0006, 0.1058] 0.1052[−0.0439, 0.1214] 0.1653[−0.0431, 0.1225] 0.1656[−0.0425, 0.1207] 0.1632Forecast 20180.0410.03400.03340.03440.03570.03950.03900.0405Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0099, 0.0793] 0.0892[−0.0151, 0.0810] 0.0961[−0.0099, 0.0941] 0.1040[0.0019, 0.0983] 0.0964[−0.0414, 0.1185] 0.1599[−0.0423, 0.1191] 0.1614[−0.0431, 0.1206] 0.1637MAE0.00380.00430.00340.00530.00230.00230.0024RMSE0.01260.01290.01530.01300.00590.00660.0066Mean of prediction range0.08840.10380.11190.10730.14760.14750.1467norm = normal likelihood, std = student-t likelihood, sstd = skewed student-t likelihood, and Yf is the point prediction.Figure 3
Lower and upper cdfs for one-step-ahead prediction obtained from Shannon entropy: the forecast made in 2018 for the growth rate of Thai GDP in 2019.


### 3.2. The Entropy-Based Belief Function in the Regression Problem

#### 3.2.1. Regression Problem

Consider the simple linear regression model:(44)yt=∑k=1Kβkxkt+εt,t=1,...,T,where yt is observed dependent variable at time t, xkt is an independent variable k at time t, and εt is the error that is not assumed to be any distribution when the entropy-based approach is used.


#### 3.2.2. Prediction with Certain Inputs

Similar to the AR model, the prediction with linear regression can be easily computed using the method described in Section3.1.2. Let YT+1 be the future value which cannot be observed at time T+1 and xkT be the observed inputs or covariates at time T:(45)YT+1=∑k=1KβkxkT+εT,where YT+1 is a predicted value which is not yet observed and yT is the observed value at time T. We can write equivalently,(46)φyβ,s=∑k=1KβkxkT+F^−1s,s∼U0,1.Under the entropy approach, we rewrite equations (45)-(46) as(47)YT+1=φyP,s=∑k=1KxkT∑m=1Mpkmzkm+F^−1s.To forecast the future observation conditional on all information of the observed inputs at time,T, YT+1xkT, the predictive belief function and plausibility function can then be approximated using Monte Carlo simulation as described in Section 3.1.2. We, then, compute the maximum and minimum of the linear regression function φyP,s subject to plyP≥ωi for minimum and maximum problems. The additional constraints are(48)yT+1=∑k=1KxkT∑m=1Mpkmzkm+∑m=1MwTmvTm,∑m=1Mpkm=1,∑m=1Mwtm=1.Example 3.
As an example, we illustrate the proposed method by applying them to the data from the operation of a plant for the oxidation of ammonia to nitric acid, measured on 21 consecutive days. The data are analyzed by Brownlee [16] in a simple regression setting, and it is already available within R software. The dataset consists of 4 variables, namely, air flow to the plant (AIR), cooling water inlet temperature (WATER), acid concentration (percentage minus 50 times 10 (ACID), and ammonia lost (percentage times 10) (LOSS). We consider LOSS as the dependent variable, and AIR, WATER, and ACID as independent variables. Thus, we consider the following linear regression model:(49)LOSSt=β0+β1AIRt+β2WATERt+β3ACIDt+εt.
By using the Maximum likelihood (ML) and GME estimations, we can obtain the estimated parameters, standard errors, andp values. Table 6 provides the estimated results of our ML and GME estimates altogether with some basic statistics (standard errors and p values) and the plausibilities plβk=0 (see [7]). However, in the case of GME estimation, the computation of the plausibilities plβk=0 is different. Recall that, in the entropy approach, the estimated parameter of the model cannot be obtained directly from the estimation, but it is computed as the summation of all expected support points. Thus, it would be reasonable to have pl∑m=1Mpkmzkm=0. We know that if pk1=pk2=⋯=pkM=1/M, then ∑m=1Mpkmzkm=0. Hence, we turn our attention to test plpkm=1/M. In our approach, suppose M=3 and we have βk=0, whenever pk1=pk3. To compute this, we need to maximize the entropy under the usual constraints and additional constraint pk1=pk3=1/3. Therefore, we can further compute the plβk=0=plpk1=pk3. In this test, we test whether plpk1=pk3 is less than plβk=β⌢k=plp⌢k1,p⌢k2,p⌢k3. Let p⌢km be the optimal probability of variable k at support m. If plpk1=pk3<plp⌢k1,p⌢k2,p⌢k3, the hypothesis of the coefficient βk equal to zero is rejected, indicating that this coefficient is significant.
Figure4 shows the marginal contour functions plβ3=0 for parameter ACID obtained from the likelihood-based approach, and we can observe that the plβ3=0 is greater than 0.10, indicating that the acid concentration variable is said to have insignificant effect to ammonia lost. In Figure 5, the marginal contour functions of plβk=0=plpk1=pk3 are plotted for parameters AIR k=1, WATER k=2, and ACID k=3. We can observe that(50)plp11=p13<plp⌢11,p⌢12,p⌢13,plp21=p23<plp⌢21,p⌢22,p⌢23,plp31=p33=plp⌢31,p⌢32,p⌢33.
This confirms that there is no significant effect of ACID to LOSS.Figure 4
Marginal contour functions for the coefficients of variables AIR obtained from (a) normal likelihood, (b) student-t likelihood, and (c) skewed student-t likelihood.
(a)(b)(c)Figure 5
Marginal plausibility of probabilities (Shannon entropy) plotted in blue lines and the plausibility ofplp⌢k1,p⌢k2,p⌢k3 plotted in red horizon line. Similar plots are also obtained for Renyi and Tsallis entropies.
(a)(b)(c)Example 4.
Continuing Example3, we then consider the task of prediction problem. We consider the ammonia lost as the dependent variable and only AIR and WATER as the inputs since they show a significant effect on LOSS variable. We have re-estimated the model, and the result is provided in Table 7 and Figure 6. We can observe that the similar significant result of the estimated parameters is obtained for all methods. We also make a prediction of LOSS when the inputs are known at time T; hence, the prediction equation is based on(51)LOSST=β0+β1AIRT+β2WATERT+εT,with εT which is assumed to be empirical distribution. We can write, equivalently,(52)LOSST∗=β0+β1AIRT+β2WATERT+ϕ−1s.
The predictive belief function onLOSST=21∗ can then be approximated using the methods described in Section 3.2.2. Figure 6 displays the lower and upper cdfs of the various predictive belief functions, approximated using N = 5000 randomly generated focal sets. The prediction value of LOSST=21∗ is plotted by the vertical blue dotted line as well as the α quantile intervals with α=0.05,0.95. We then compare the prediction performance based on prediction bias. The true value of LOSST=21 is 15; thus, we calculate prediction bias from LOSST=21∗−15. According to the last row of Table 7, we can observe that the Shannon entropy-based belief function method outperforms the others as the lowest bias is obtained. Moreover, we find that the prediction interval of the Shannon entropy method provides a reliable interval. Although the 0.05 quantile interval does not cover the true value, the lower bound of its interval is closer to the true value = 15. Therefore, to confirm the reliability of the method, we extend the lower quantile interval of the Shannon entropy-based method to be α=0.01 and we can observe that the true observed data is contained in the 0.01 lower quantile interval (green dashed line).Table 6
Regression results.
normstdsstdShannon entropyRenyi entropyTsallis entropyEstimateIntercept−43.7037 (0.0000) [0.0000]−43.3522 (0.0000) [0.0000]−41.3097 (0.0000) [0.0000]−38.6949 (0.0000)∗∗∗−38.6950 (0.0000)∗∗∗−38.7344 (0.0000)∗∗∗AIR0.8891 (0.0000) [0.0000]0.8980 (0.0000) [0.0000]0.8605 (0.0000) [0.0000]0.7894 (0.0000)∗∗∗0.7890 (0.0000)∗∗∗0.7317 (0.0000)∗∗∗WATER0.8166 (0.0049) [0.0723]0.7823 (0.0043) [0.1001]0.8154 (0.0079) [0.0000]1.1918 (0.0007)∗∗∗1.1918 (0.0007)∗∗∗1.2315 (0.0008)∗∗∗ACID−0.1071 (0.3361) [0.2179]−0.1094 (0.3114) [0.2357]−0.1155 (0.1733) [0.2495]−0.1821 (0.2631)−0.1821 (0.2630)−0.1828 (0.2644)() isp value, [] is Plβj=0, and ∗∗∗ is plpk1=pk3<plp⌢k1,p⌢k2,p⌢k3. For the case of entropy, the support z1m is initially set to (−50, 0, 50) and the supports for vtm to (−3σ, 0, 3σ), where σis computed form the conventional LS estimation.Figure 6
Lower and upper cdfs for the prediction problem in linear regression. (a) Normal. (b) Student-t. (c) Skewed Student. (d) Shannon Entropy. (e) Renyi Entropy. (f) Tsallis Entropy.
(a)(b)(c)(d)(e)(f)Table 7
Regression results of the reduced model with only two inputs.
normstdsstdShannon entropyRenyi entropyTsallis entropyEstimateIntercept−57.0752 (0.0000) [0.0000]−43.8121 (0.0000) [0.0000]−32.0274 (0.0000) [0.0000]−51.8325 (0.0000)∗∗∗−41.4330 (0.0000)∗∗∗−50.9705 (0.0000)∗∗∗AIR0.8629 (0.0000) [0.0000]0.8354 (0.0000) [0.0000]0.5923 (0.0000) [0.0000]0.7633 (0.0000)∗∗∗0.7561 (0.0000)∗∗∗0.7029 (0.0000)∗∗∗WATER0.8033 (0.0233) [0.0000]0.5056 (0.0709) [0.0000]0.7831 (0.0079) [0.0000]1.1479 (0.0000)∗∗∗1.1479 (0.0000)∗∗∗1.1358 (0.0000)∗∗∗Expected prediction26.5564 <19.1543, −32.4528>27.0138 <8.3351, −32.3311>24.5375 <18.4833, −32.1598>23.5138 <17.9744, 34.1935>26.1581 <18.1415, 34.8077>24.0054 <17.0348, 34.5871>Prediction bias11.556412.01389.53758.513811.15819.0054<> is the prediction interval, ( ) isp value, [ ] is Plβj=0, and ∗∗∗ is plpk1=pk3<plp⌢k1,p⌢k2,p⌢k3. For the case of Entropy, the support z1m is initially set to (−100, 0, 100) and the supports for vtm to (−3σ, 0, 3σ), where σ is computed from the conventional LS estimation.


## 3.1. The Entropy-Based Belief Function in Autoregressive Model Problem

### 3.1.1. Prediction

As we have seen in Section2, the estimation problem is to make statements about some probability after observing some data. To describe the prediction method for the AR model, let us consider the case where the K = 1; thus, the first-order linear autoregressive AR (1) model is defined by(30)YT+1=β1+β2yT+εT,where YT+1 is a predicted value of T+1 which is not yet observed and yT is the observed value at time T.The main idea is to estimateYT+1 as a function of β, which is viewed as an expectation of probability with M support and some pivotal variable S, whose distribution does not depend on β. We can write equivalently(31)YT+1=φyβ,s=β1+β2yT+F^−1s,s∼U0,1,where F^⋅−1 is the quantile function of empirical distribution based on the estimated residuals, ε1,...,εT. Then, in the case of entropy approach, we rewrite equation (30) as(32)YT+1=∑m=1Mp1mz1m+∑m=1Mp2mz2myT+F^−1s,where m is the number of supports of z and v. Consequently, in the entropy estimation, we rewrite the focal set equation (31) as(33)YT+1=φyP,s=∑m=1Mp1mz1m+∑m=1Mp2mz2myT+F^−1s.To forecastYT+1yT, the predictive belief function and plausibility function can then be approximated using Monte Carlo simulation as in the following procedure (Kanjanatarakul et al. [3] and Kanjanatarakul et al. [7]:(1)
Drawωi and si independently from uniform [0, 1] N draws. Then, compute (or approximate) the focal set which is the interval defined by the following lower and upper bounds:(34)φyΓyω,s=φP,splyP≥ωi=y¯isi,ωi,y¯isi,ωi,wherey¯isi,ωi and y¯isi,ωi are the lower and upper bounds which can be computed using a constrained nonlinear optimization algorithm. This optimization problem can be solved using the nonlinear optimization:(35)y¯isi,ωi=minφyP,ωi,si,(36)y¯isi,ωi=maxφyP,ωi,si,which subject to(37)plyΡ≥ωi,yt=∑m=1Mp1mz1m+∑m=1Mp2mz2myt,t−1+∑m=1Mwtmvtm,∑m=1Mp1m=1,∑m=1Mp2m=1,∑m=1Mwtm=1.(2)
We can then approximate the predictive belief (Bely) and plausibility (ply) functions on Yf as(38)BelyC=1N#i∈1,...,NφyΓyωi,si⊆C,PlyC=1N#i∈1,...,NφyΓyωi,si∩C≠∅,or allC⊆Θ. These lower and upper cdfs of the predictive belief function are approximated using N = 5000 randomly generated focal sets (Algorithm 1).Algorithm 1: Monte Carlo algorithm for approximating a predictive entropy-based belief function (random interval case).
Require: Desired number of focal sets N, number of support vector M, support spaces of coefficients zk=zk1,...,zkM and error terms vt=vt1,...,vtM.fori=1toNdoDrawωi,si from uniform [0, 1]Search fory¯i=minφyP,ωi,si such that plyP≥ωi,yt=∑m=1Mp1mz1m+∑m=1Mp2mz2myt−1+∑m=1Mwtmvtmand∑m=1Mp1m=1,∑m=1Mp2m=1,∑m=1Mwtm=1.Search fory¯isi,ωi=maxφyP,ωi,si such that plyP≥ωi,yt=∑m=1Mp1mz1m+∑m=1Mp2mz2myt−1+∑m=1Mwtmvtmand∑m=1Mp1m=1,∑m=1Mp2m=1,∑m=1Mwtm=1Bi⟵y¯i,y¯iend forY^T+1=∑i=1Ny¯i+y¯i/N


### 3.1.2. Experiment Study

In this section, we conduct a simulation and experiment studies to investigate the finite sample performance of the proposed Shannon entropy-based belief methods. The simulated data are constructed from the AR model using different error distributions consisting of normal, Student-t, skewed Student-t, and uniform. We note that the normal and student-t are selected as symmetric distribution, while skewed student-t and uniform are typical examples of highly skewed distribution and flat distribution, respectively. In these experiments, we consider three competing likelihood functions: normal, student-t, and skewed student-t likelihoods.To compare these likelihood-based belief functions, the one-step-ahead forecast is computed from simulated data and compared with the true prediction value. We note that, as the functionply=Ply has a unique maximum y˜, it may be taken as a point prediction of the AR (1) model-based belief function approach. This is to say y˜ can be viewed as the most plausible value of Y given the past information. Thus, we need a criterion to gauge the performance of the method. Here, the mean square error MSE method is employed, and it can be defined as(39)MSE=R−1∑r=1RYtrue,r−Yf,r2,where R = 20 is the number of repetitions, Ytrue is the true value, and Yf=y˜ is the predicted value.In this Monte Carlo simulation, we consider sample sizeT=20 and T=40. R = 20 samples are generated for each case. To make a fair comparison, the error terms are generated from N0,σ2=1, t0,σ2=1,ν=4, skew−t0,σ2=1,ν=4,ξ=1.5, and U−2,2. Then, the sampling experiments are based on the AR (1) model:(40)yt=0.5−0.5yt−1+εt.In this experiment, the first 20 observations are used to predict the 21st observation. Likewise, the first 40 observations are used to predict the 41st observation. Then, the most plausible predicted values,Yf21st and Yf41st, are compared with Ytrue21st=0.5−0.5yT+εT+1,T=20, and Ytrue41st=0.5−0.5yT+εT+1,T=40, respectively. In the case of the Shannon entropy, the support z1m is initially set to (−5, 0, 5) and the supports for vtm to (−3σ, 0, 3σ), where σ is computed from the conventional AR model. Table 3 reports the prediction error MSE of our considered estimators. Consider the performance of the proposed estimator when the errors are generated from a uniform distribution, and we found that the error of the Shannon entropy-based belief function is smaller than the other methods over the one-step-ahead forecast. In addition, the performance of this method is better when the sample size increases from 20 to 40.Table 3
Forecasting performance.
Y˜fNormal likelihood-based belief functionStudent-t likelihood-based belief functionSkewed Student-t likelihood-based belief functionShannon entropy-based belief functionRenyi entropy-based belief functionTsallis entropy-based belief functionU−2,2T = 20MSE0.9346 (0.9541)1.2327 (1.8079)13.8763 (15.4710)0.5784 (0.5824)0.5796 (0.5831)0.5788 (0.5826)T = 40MSE0.6891 (0.7552)1.6996 (1.6252)8.7598 (12.2013)0.2596 (0.5914)0.2608 (0.5948)0.2605 (0.5953)N0,1T = 20MSE0.4601 (0.5491)1.2275 (1.4574)2.0607 (2.4733)0.5582 (0.7576)0.5688 (0.7589)0.5591 (0.7601)T = 40MSE0.4995 (0.3136)1.4068 (1.4647)1.2092 (1.1063)0.5294 (0.3787)0.5301 (0.3793)0.5312 (0.3798)t0,1,4T = 20MSE1.2850 (1.476)1.1815 (2.6127)1.6660 (1.8974)1.2191 (1.7849)1.2115 (1.7858)1.2114 (1.7857)T = 40MSE0.7604 (0.4309)0.6142 (0.3654)0.8783 (1.1256)0.7538 (0.2542)0.7542 (0.2550)0.7542 (0.2550)skew−t0,1,4,1.5T = 20MSE0.7278 (0.9846)2.6995 (2.0442)0.5878 (1.1307)0.6699 (1.6189)0.6703 (1.6208)0.6700 (1.6200)T = 40MSE0.5520 (0.4491)1.4772 (1.2513)0.4443 (0.8051)0.4513 (0.3370)0.4559 (0.3389)0.4564 (0.3401)() is standard deviation onMSE. The bold numbers present the best prediction result.Next, we investigate the performance of the proposed estimator when the errors are generated from normal, student-t, and skewed student-t distributions and compare with the parametric likelihood-based belief function. We observe that the overall result is different in Table 3. We find that the Shannon entropy-based belief function seems to outperform the likelihood-based belief function in terms of lowest MSE of the prediction when the U−2,2 error distribution is assumed. When the errors are generated from other parametric distributions, we find that the likelihood-based belief function method with the correct distribution function always has the best performance. However, we notice that our Shannon entropy-based belief function performs the second-best prediction over the wide range of error distributions.Thus, in this simulation study, we can conclude that our method performs well particularly when the error is unknown or has a uniform distribution. Nonetheless, when the error is generated from known distribution, the parametric likelihood-based belief function outperforms the entropy. This means that if the distribution is known and we estimate the model based on the true distribution, the accurate result is obtained. However, the error distribution is sometimes unknown in practice, these results confirm the advantage of the Shannon entropy-based belief function, which performs uniformly well over a wide range of error distributions.Furthermore, it would be more interesting to consider other entropy measures such as Renyi entropy [13] and Tsallis entropy [14]; we thus introduce these two entropy-based belief functions to investigate the performance of various entropy measures. We note that Renyi entropy is the generalization of Shannon entropy while Tsallis entropy is nonlogarithmic and is obtained through the joint generalization of the averaging procedures and the concept of information gain. These two entropies are indexed by an order d. The Renyi entropy is formulated as(41)HdRp=−11−d∑k=1K∑m=1Mlogpkmd,while Tsallis entropy can be defined as(42)HdTp=−c∑k=1K∑m=1Mpkmd−11−d,where the value of c is a positive constant and depends on the particular units used. For simplicity, we set c=1 and order d (see [15]). Note that both Renyi and Tsallis entropies become the Shannon entropy as a special case when d=2. To construct the Renyi entropy-based belief function and Tsallis entropy-based belief function, we can replace Shannon entropy in the relative plausibility (equation (18)) by Renyi and Tsallis, respectively.The results of these three entropies for comparison are shown in the last three columns of Table1, and it is evident that the Shannon entropy-based belief function still performs the best prediction performance in this simulation data. However, the performances of these three entropies are not much different in terms of MSE.Example 2.
Application to predicting GDP growth by the AR model.
In this example, the real data are also used to investigate the performance of our method. The dataset considered, derived from the Thomson Reuter Datastream and World Bank database, consist of yearly data, from the ending of 1995 to 2018, of Gross Domestic Product (GDP). The examined data series is transformed into the growth rate and plotted in Figure2. Then, we perform the Augmented Dickey–Fuller unit roots test on GDP growth and determine the lagging order number of the AR model according to Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). The ADF-statistic result is −4.337, which is greater than the critical values at 10% significance level and means that GDP is stationary. Then, the lag selection for the AR model is shown in Table 4. The result shows that the AR model with lag 1 provides the lowest AIC and BIC.
To evaluate the performance of prediction, the forecast evaluation period is from 2014 to 2018. We perform a recursive forecasting to update GDP growth year by year, totally 5 years, which give us enough data points to evaluate the out-of-sample forecast performance. For this evaluation period, we compute the root mean square forecast error (RMSE) and mean square error (MAE) of forecasts at each forecast horizon (from one to five years ahead). We can compute the RMSE and MAE as in the following:(43)MAE=1T∑t=1Tet,RMSE=1T∑t=1Tet2,where T = 5 and e=Ytrue−Yf.
The forecast comparison of the AR(1) model with the likelihood-based approach and the entropy-based approach, as well as the conventional frequentist method, performed in out-of-sample is reported in Table5. The out-of-sample forecasts made for the 2014–2018 period are reflected by the most plausible prediction values and their large upper-lower prediction intervals. The true prediction values are shown in the first column. The estimated conditional expectation as well as the frequentist 95% prediction intervals are shown in column 2. Columns 3–7 show the most plausible predictions and quantile intervals for the normal, student-t, skewed student-t, and the three entropy-based belief functions, respectively.
We can now consider the performance of different methods. First, we evaluate the entropy-based method regarding the prediction value. Second, we compare the performance regarding the range of the prediction intervals.
In the first evaluation, the smaller theMAE and RMSE, the superior the corresponding model’s performance. According to Table 5, we learn that all three entropy-based belief functions outperform the other likelihood-based methods. We can observe that the values of RMSE and MAE of entropy-based belief functions are a bit smaller than those of the likelihood-based belief functions. Again, the Shannon entropy still provides lower RMSE and MAE compared to the other two entropies. We expect that if the real data has a distribution and the assumption of the likelihood-based belief function is not correctly specified, a less accurate prediction is obtained. In addition, small sample size may affect the accuracy of the estimated parameters. Hence, in this study, we conclude that entropy-based belief function for AR (1) performs well in the out-of-sample forecasts relative to other competing methods.
In the second evaluation, for the prediction intervals for Thai GDP growth, the methods suggest reliable prediction intervals around the predicted value. We can observe that all prediction interval methods for the AR (1) model are reliable as the true values exist along with the range of the intervals, except for the 95% frequentist approach. The prediction interval based on the 95% frequentist for Thai GDP growth in 2014 does not cover the true value. This result reveals that the conventional frequentist approach may not provide the appropriate prediction value.
Regarding the prediction intervals, we can calculate the average of the prediction intervals for making a comparison. The results portray the difference between the methods exists. The frequentist method has a very narrow mean of the prediction intervals, while our entropy-based method performs the widest. We suspect that our approach relies on the empirical distribution assumption on the error distribution and leads to a larger simulated error in the Monte Carlo algorithm for approximating a predictive entropy-based belief function process. However, this experiment shows that our method produces an acceptable range for prediction intervals as the true values always lie in the intervals.
We now use the entropy-based belief function method to predict Thai economic growth in the future. The prediction is very important because of its relevance to the current policy debate on whether Thai GDP growth will be up or down. The forecasts made for 2019 is based on scant data, which is reflected by large upper-lower cdf intervals (Figure3). Figure 3 presents the lower and upper cumulative density functions (cdfs) for the prediction problem obtained from (black solid line) and (pink solid line). We also plot the vertical blue dotted line corresponding to the most plausibility value together with the quantile intervals for the GDP growth in 2019. We can observe that the value of growth rate is around 0.03961 (3.961%) in 2019 and the prediction interval 95% ranges (−0.0391 – 0.0558) (pink dashed line) and (0.0630 – 0.1124) (black dashed line).
Furthermore, to show the accuracy of our estimation in this study, we show the contour plots of plausibilityplyθ in two-dimensional parameter subspaces of each method, as shown in the Appendix. These contour plots show two-dimensional slices of the contour function, where one of the three parameters is fixed to its maximum normal likelihood, two of the four parameters are fixed to its maximum student-t likelihood, three of the five parameters are fixed to its maximum skewed student-t likelihood, and z+v×m−2 of the z+v×m probability parameters are fixed to its maximum entropy. The results of these contour plots show that the estimated parameters of each method are likely to reach the maximum plausibility, as presented by the red cross. Consider the contour plots of plyθ for our proposed method, entropy-based belief functions. We can summarize the process graphically by plotting the optimization on a contour plot of the plausibility plyP. We can observe that, for example, the estimated probabilities p11 and p12 reach a maximum plausibility plyP at the red cross.Figure 2
GDP growth rate of Thailand.Table 4
Lag selection for the AR model.
AR (1)AR (2)AR (3)AR (4)AIC−104.41−102.69−100.75−100.43BIC−101.00−98.14−95.076−93.61Table 5
Out-of-sample forecasting performance.
TRUEYfARYfnormYfstdYfsstdYfshannonYfRenyiYftsallisForecast 20140.0090.03570.03670.03310.03670.00420.00390.0030Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[0.0123, 0.0838] 0.0715[−0.0191, 0.0918] 0.1109[−0.0157, 0.1072] 0.1229[−0.0054, 0.1172] 0.1226[−0.0226, 0.0381] 0.0607[−0.0225, 0.0387] 0.0613[−0.0218, 0.0383] 0.0601Forecast 20150.0300.03310.03280.03380.03480.04040.04190.0410Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0150, 0.0812] 0.0962[−0.0187, 0.0882] 0.1069[−0.0153, 0.1006] 0.1159[0.0016, 0.1035] 0.1019[−0.0496, 0.1305] 0.1801[−0.0493, 0.1351] 0.1844[−0.0490, 0.1309] 0.1799Forecast 20160.0330.03400.03520.03380.03620.03980.04010.0407Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0128, 0.0808] 0.0936[−0.0162, 0.0887] 0.1049[−0.0133, 0.0976] 0.1109[0.0001, 0.1107] 0.1106[−0.0467, 0.1252] 0.1719[−0.0433, 0.1212] 0.1645[−0.0439, 0.1231] 0.1670Forecast 20170.0390.03410.03440.03390.03530.03950.03850.0386Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0115, 0.0798] 0.0913[−0.0162, 0.0838] 0.1000[−0.0116, 0.0950] 0.1066[0.0006, 0.1058] 0.1052[−0.0439, 0.1214] 0.1653[−0.0431, 0.1225] 0.1656[−0.0425, 0.1207] 0.1632Forecast 20180.0410.03400.03340.03440.03570.03950.03900.0405Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0099, 0.0793] 0.0892[−0.0151, 0.0810] 0.0961[−0.0099, 0.0941] 0.1040[0.0019, 0.0983] 0.0964[−0.0414, 0.1185] 0.1599[−0.0423, 0.1191] 0.1614[−0.0431, 0.1206] 0.1637MAE0.00380.00430.00340.00530.00230.00230.0024RMSE0.01260.01290.01530.01300.00590.00660.0066Mean of prediction range0.08840.10380.11190.10730.14760.14750.1467norm = normal likelihood, std = student-t likelihood, sstd = skewed student-t likelihood, and Yf is the point prediction.Figure 3
Lower and upper cdfs for one-step-ahead prediction obtained from Shannon entropy: the forecast made in 2018 for the growth rate of Thai GDP in 2019.


## 3.1.1. Prediction

As we have seen in Section2, the estimation problem is to make statements about some probability after observing some data. To describe the prediction method for the AR model, let us consider the case where the K = 1; thus, the first-order linear autoregressive AR (1) model is defined by(30)YT+1=β1+β2yT+εT,where YT+1 is a predicted value of T+1 which is not yet observed and yT is the observed value at time T.The main idea is to estimateYT+1 as a function of β, which is viewed as an expectation of probability with M support and some pivotal variable S, whose distribution does not depend on β. We can write equivalently(31)YT+1=φyβ,s=β1+β2yT+F^−1s,s∼U0,1,where F^⋅−1 is the quantile function of empirical distribution based on the estimated residuals, ε1,...,εT. Then, in the case of entropy approach, we rewrite equation (30) as(32)YT+1=∑m=1Mp1mz1m+∑m=1Mp2mz2myT+F^−1s,where m is the number of supports of z and v. Consequently, in the entropy estimation, we rewrite the focal set equation (31) as(33)YT+1=φyP,s=∑m=1Mp1mz1m+∑m=1Mp2mz2myT+F^−1s.To forecastYT+1yT, the predictive belief function and plausibility function can then be approximated using Monte Carlo simulation as in the following procedure (Kanjanatarakul et al. [3] and Kanjanatarakul et al. [7]:(1)
Drawωi and si independently from uniform [0, 1] N draws. Then, compute (or approximate) the focal set which is the interval defined by the following lower and upper bounds:(34)φyΓyω,s=φP,splyP≥ωi=y¯isi,ωi,y¯isi,ωi,wherey¯isi,ωi and y¯isi,ωi are the lower and upper bounds which can be computed using a constrained nonlinear optimization algorithm. This optimization problem can be solved using the nonlinear optimization:(35)y¯isi,ωi=minφyP,ωi,si,(36)y¯isi,ωi=maxφyP,ωi,si,which subject to(37)plyΡ≥ωi,yt=∑m=1Mp1mz1m+∑m=1Mp2mz2myt,t−1+∑m=1Mwtmvtm,∑m=1Mp1m=1,∑m=1Mp2m=1,∑m=1Mwtm=1.(2)
We can then approximate the predictive belief (Bely) and plausibility (ply) functions on Yf as(38)BelyC=1N#i∈1,...,NφyΓyωi,si⊆C,PlyC=1N#i∈1,...,NφyΓyωi,si∩C≠∅,or allC⊆Θ. These lower and upper cdfs of the predictive belief function are approximated using N = 5000 randomly generated focal sets (Algorithm 1).Algorithm 1: Monte Carlo algorithm for approximating a predictive entropy-based belief function (random interval case).
Require: Desired number of focal sets N, number of support vector M, support spaces of coefficients zk=zk1,...,zkM and error terms vt=vt1,...,vtM.fori=1toNdoDrawωi,si from uniform [0, 1]Search fory¯i=minφyP,ωi,si such that plyP≥ωi,yt=∑m=1Mp1mz1m+∑m=1Mp2mz2myt−1+∑m=1Mwtmvtmand∑m=1Mp1m=1,∑m=1Mp2m=1,∑m=1Mwtm=1.Search fory¯isi,ωi=maxφyP,ωi,si such that plyP≥ωi,yt=∑m=1Mp1mz1m+∑m=1Mp2mz2myt−1+∑m=1Mwtmvtmand∑m=1Mp1m=1,∑m=1Mp2m=1,∑m=1Mwtm=1Bi⟵y¯i,y¯iend forY^T+1=∑i=1Ny¯i+y¯i/N


## 3.1.2. Experiment Study

In this section, we conduct a simulation and experiment studies to investigate the finite sample performance of the proposed Shannon entropy-based belief methods. The simulated data are constructed from the AR model using different error distributions consisting of normal, Student-t, skewed Student-t, and uniform. We note that the normal and student-t are selected as symmetric distribution, while skewed student-t and uniform are typical examples of highly skewed distribution and flat distribution, respectively. In these experiments, we consider three competing likelihood functions: normal, student-t, and skewed student-t likelihoods.To compare these likelihood-based belief functions, the one-step-ahead forecast is computed from simulated data and compared with the true prediction value. We note that, as the functionply=Ply has a unique maximum y˜, it may be taken as a point prediction of the AR (1) model-based belief function approach. This is to say y˜ can be viewed as the most plausible value of Y given the past information. Thus, we need a criterion to gauge the performance of the method. Here, the mean square error MSE method is employed, and it can be defined as(39)MSE=R−1∑r=1RYtrue,r−Yf,r2,where R = 20 is the number of repetitions, Ytrue is the true value, and Yf=y˜ is the predicted value.In this Monte Carlo simulation, we consider sample sizeT=20 and T=40. R = 20 samples are generated for each case. To make a fair comparison, the error terms are generated from N0,σ2=1, t0,σ2=1,ν=4, skew−t0,σ2=1,ν=4,ξ=1.5, and U−2,2. Then, the sampling experiments are based on the AR (1) model:(40)yt=0.5−0.5yt−1+εt.In this experiment, the first 20 observations are used to predict the 21st observation. Likewise, the first 40 observations are used to predict the 41st observation. Then, the most plausible predicted values,Yf21st and Yf41st, are compared with Ytrue21st=0.5−0.5yT+εT+1,T=20, and Ytrue41st=0.5−0.5yT+εT+1,T=40, respectively. In the case of the Shannon entropy, the support z1m is initially set to (−5, 0, 5) and the supports for vtm to (−3σ, 0, 3σ), where σ is computed from the conventional AR model. Table 3 reports the prediction error MSE of our considered estimators. Consider the performance of the proposed estimator when the errors are generated from a uniform distribution, and we found that the error of the Shannon entropy-based belief function is smaller than the other methods over the one-step-ahead forecast. In addition, the performance of this method is better when the sample size increases from 20 to 40.Table 3
Forecasting performance.
Y˜fNormal likelihood-based belief functionStudent-t likelihood-based belief functionSkewed Student-t likelihood-based belief functionShannon entropy-based belief functionRenyi entropy-based belief functionTsallis entropy-based belief functionU−2,2T = 20MSE0.9346 (0.9541)1.2327 (1.8079)13.8763 (15.4710)0.5784 (0.5824)0.5796 (0.5831)0.5788 (0.5826)T = 40MSE0.6891 (0.7552)1.6996 (1.6252)8.7598 (12.2013)0.2596 (0.5914)0.2608 (0.5948)0.2605 (0.5953)N0,1T = 20MSE0.4601 (0.5491)1.2275 (1.4574)2.0607 (2.4733)0.5582 (0.7576)0.5688 (0.7589)0.5591 (0.7601)T = 40MSE0.4995 (0.3136)1.4068 (1.4647)1.2092 (1.1063)0.5294 (0.3787)0.5301 (0.3793)0.5312 (0.3798)t0,1,4T = 20MSE1.2850 (1.476)1.1815 (2.6127)1.6660 (1.8974)1.2191 (1.7849)1.2115 (1.7858)1.2114 (1.7857)T = 40MSE0.7604 (0.4309)0.6142 (0.3654)0.8783 (1.1256)0.7538 (0.2542)0.7542 (0.2550)0.7542 (0.2550)skew−t0,1,4,1.5T = 20MSE0.7278 (0.9846)2.6995 (2.0442)0.5878 (1.1307)0.6699 (1.6189)0.6703 (1.6208)0.6700 (1.6200)T = 40MSE0.5520 (0.4491)1.4772 (1.2513)0.4443 (0.8051)0.4513 (0.3370)0.4559 (0.3389)0.4564 (0.3401)() is standard deviation onMSE. The bold numbers present the best prediction result.Next, we investigate the performance of the proposed estimator when the errors are generated from normal, student-t, and skewed student-t distributions and compare with the parametric likelihood-based belief function. We observe that the overall result is different in Table 3. We find that the Shannon entropy-based belief function seems to outperform the likelihood-based belief function in terms of lowest MSE of the prediction when the U−2,2 error distribution is assumed. When the errors are generated from other parametric distributions, we find that the likelihood-based belief function method with the correct distribution function always has the best performance. However, we notice that our Shannon entropy-based belief function performs the second-best prediction over the wide range of error distributions.Thus, in this simulation study, we can conclude that our method performs well particularly when the error is unknown or has a uniform distribution. Nonetheless, when the error is generated from known distribution, the parametric likelihood-based belief function outperforms the entropy. This means that if the distribution is known and we estimate the model based on the true distribution, the accurate result is obtained. However, the error distribution is sometimes unknown in practice, these results confirm the advantage of the Shannon entropy-based belief function, which performs uniformly well over a wide range of error distributions.Furthermore, it would be more interesting to consider other entropy measures such as Renyi entropy [13] and Tsallis entropy [14]; we thus introduce these two entropy-based belief functions to investigate the performance of various entropy measures. We note that Renyi entropy is the generalization of Shannon entropy while Tsallis entropy is nonlogarithmic and is obtained through the joint generalization of the averaging procedures and the concept of information gain. These two entropies are indexed by an order d. The Renyi entropy is formulated as(41)HdRp=−11−d∑k=1K∑m=1Mlogpkmd,while Tsallis entropy can be defined as(42)HdTp=−c∑k=1K∑m=1Mpkmd−11−d,where the value of c is a positive constant and depends on the particular units used. For simplicity, we set c=1 and order d (see [15]). Note that both Renyi and Tsallis entropies become the Shannon entropy as a special case when d=2. To construct the Renyi entropy-based belief function and Tsallis entropy-based belief function, we can replace Shannon entropy in the relative plausibility (equation (18)) by Renyi and Tsallis, respectively.The results of these three entropies for comparison are shown in the last three columns of Table1, and it is evident that the Shannon entropy-based belief function still performs the best prediction performance in this simulation data. However, the performances of these three entropies are not much different in terms of MSE.Example 2.
Application to predicting GDP growth by the AR model.
In this example, the real data are also used to investigate the performance of our method. The dataset considered, derived from the Thomson Reuter Datastream and World Bank database, consist of yearly data, from the ending of 1995 to 2018, of Gross Domestic Product (GDP). The examined data series is transformed into the growth rate and plotted in Figure2. Then, we perform the Augmented Dickey–Fuller unit roots test on GDP growth and determine the lagging order number of the AR model according to Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). The ADF-statistic result is −4.337, which is greater than the critical values at 10% significance level and means that GDP is stationary. Then, the lag selection for the AR model is shown in Table 4. The result shows that the AR model with lag 1 provides the lowest AIC and BIC.
To evaluate the performance of prediction, the forecast evaluation period is from 2014 to 2018. We perform a recursive forecasting to update GDP growth year by year, totally 5 years, which give us enough data points to evaluate the out-of-sample forecast performance. For this evaluation period, we compute the root mean square forecast error (RMSE) and mean square error (MAE) of forecasts at each forecast horizon (from one to five years ahead). We can compute the RMSE and MAE as in the following:(43)MAE=1T∑t=1Tet,RMSE=1T∑t=1Tet2,where T = 5 and e=Ytrue−Yf.
The forecast comparison of the AR(1) model with the likelihood-based approach and the entropy-based approach, as well as the conventional frequentist method, performed in out-of-sample is reported in Table5. The out-of-sample forecasts made for the 2014–2018 period are reflected by the most plausible prediction values and their large upper-lower prediction intervals. The true prediction values are shown in the first column. The estimated conditional expectation as well as the frequentist 95% prediction intervals are shown in column 2. Columns 3–7 show the most plausible predictions and quantile intervals for the normal, student-t, skewed student-t, and the three entropy-based belief functions, respectively.
We can now consider the performance of different methods. First, we evaluate the entropy-based method regarding the prediction value. Second, we compare the performance regarding the range of the prediction intervals.
In the first evaluation, the smaller theMAE and RMSE, the superior the corresponding model’s performance. According to Table 5, we learn that all three entropy-based belief functions outperform the other likelihood-based methods. We can observe that the values of RMSE and MAE of entropy-based belief functions are a bit smaller than those of the likelihood-based belief functions. Again, the Shannon entropy still provides lower RMSE and MAE compared to the other two entropies. We expect that if the real data has a distribution and the assumption of the likelihood-based belief function is not correctly specified, a less accurate prediction is obtained. In addition, small sample size may affect the accuracy of the estimated parameters. Hence, in this study, we conclude that entropy-based belief function for AR (1) performs well in the out-of-sample forecasts relative to other competing methods.
In the second evaluation, for the prediction intervals for Thai GDP growth, the methods suggest reliable prediction intervals around the predicted value. We can observe that all prediction interval methods for the AR (1) model are reliable as the true values exist along with the range of the intervals, except for the 95% frequentist approach. The prediction interval based on the 95% frequentist for Thai GDP growth in 2014 does not cover the true value. This result reveals that the conventional frequentist approach may not provide the appropriate prediction value.
Regarding the prediction intervals, we can calculate the average of the prediction intervals for making a comparison. The results portray the difference between the methods exists. The frequentist method has a very narrow mean of the prediction intervals, while our entropy-based method performs the widest. We suspect that our approach relies on the empirical distribution assumption on the error distribution and leads to a larger simulated error in the Monte Carlo algorithm for approximating a predictive entropy-based belief function process. However, this experiment shows that our method produces an acceptable range for prediction intervals as the true values always lie in the intervals.
We now use the entropy-based belief function method to predict Thai economic growth in the future. The prediction is very important because of its relevance to the current policy debate on whether Thai GDP growth will be up or down. The forecasts made for 2019 is based on scant data, which is reflected by large upper-lower cdf intervals (Figure3). Figure 3 presents the lower and upper cumulative density functions (cdfs) for the prediction problem obtained from (black solid line) and (pink solid line). We also plot the vertical blue dotted line corresponding to the most plausibility value together with the quantile intervals for the GDP growth in 2019. We can observe that the value of growth rate is around 0.03961 (3.961%) in 2019 and the prediction interval 95% ranges (−0.0391 – 0.0558) (pink dashed line) and (0.0630 – 0.1124) (black dashed line).
Furthermore, to show the accuracy of our estimation in this study, we show the contour plots of plausibilityplyθ in two-dimensional parameter subspaces of each method, as shown in the Appendix. These contour plots show two-dimensional slices of the contour function, where one of the three parameters is fixed to its maximum normal likelihood, two of the four parameters are fixed to its maximum student-t likelihood, three of the five parameters are fixed to its maximum skewed student-t likelihood, and z+v×m−2 of the z+v×m probability parameters are fixed to its maximum entropy. The results of these contour plots show that the estimated parameters of each method are likely to reach the maximum plausibility, as presented by the red cross. Consider the contour plots of plyθ for our proposed method, entropy-based belief functions. We can summarize the process graphically by plotting the optimization on a contour plot of the plausibility plyP. We can observe that, for example, the estimated probabilities p11 and p12 reach a maximum plausibility plyP at the red cross.Figure 2
GDP growth rate of Thailand.Table 4
Lag selection for the AR model.
AR (1)AR (2)AR (3)AR (4)AIC−104.41−102.69−100.75−100.43BIC−101.00−98.14−95.076−93.61Table 5
Out-of-sample forecasting performance.
TRUEYfARYfnormYfstdYfsstdYfshannonYfRenyiYftsallisForecast 20140.0090.03570.03670.03310.03670.00420.00390.0030Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[0.0123, 0.0838] 0.0715[−0.0191, 0.0918] 0.1109[−0.0157, 0.1072] 0.1229[−0.0054, 0.1172] 0.1226[−0.0226, 0.0381] 0.0607[−0.0225, 0.0387] 0.0613[−0.0218, 0.0383] 0.0601Forecast 20150.0300.03310.03280.03380.03480.04040.04190.0410Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0150, 0.0812] 0.0962[−0.0187, 0.0882] 0.1069[−0.0153, 0.1006] 0.1159[0.0016, 0.1035] 0.1019[−0.0496, 0.1305] 0.1801[−0.0493, 0.1351] 0.1844[−0.0490, 0.1309] 0.1799Forecast 20160.0330.03400.03520.03380.03620.03980.04010.0407Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0128, 0.0808] 0.0936[−0.0162, 0.0887] 0.1049[−0.0133, 0.0976] 0.1109[0.0001, 0.1107] 0.1106[−0.0467, 0.1252] 0.1719[−0.0433, 0.1212] 0.1645[−0.0439, 0.1231] 0.1670Forecast 20170.0390.03410.03440.03390.03530.03950.03850.0386Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0115, 0.0798] 0.0913[−0.0162, 0.0838] 0.1000[−0.0116, 0.0950] 0.1066[0.0006, 0.1058] 0.1052[−0.0439, 0.1214] 0.1653[−0.0431, 0.1225] 0.1656[−0.0425, 0.1207] 0.1632Forecast 20180.0410.03400.03340.03440.03570.03950.03900.0405Ply−∞,q0.05L,Bely−∞,q0.95U prediction range[−0.0099, 0.0793] 0.0892[−0.0151, 0.0810] 0.0961[−0.0099, 0.0941] 0.1040[0.0019, 0.0983] 0.0964[−0.0414, 0.1185] 0.1599[−0.0423, 0.1191] 0.1614[−0.0431, 0.1206] 0.1637MAE0.00380.00430.00340.00530.00230.00230.0024RMSE0.01260.01290.01530.01300.00590.00660.0066Mean of prediction range0.08840.10380.11190.10730.14760.14750.1467norm = normal likelihood, std = student-t likelihood, sstd = skewed student-t likelihood, and Yf is the point prediction.Figure 3
Lower and upper cdfs for one-step-ahead prediction obtained from Shannon entropy: the forecast made in 2018 for the growth rate of Thai GDP in 2019.


## 3.2. The Entropy-Based Belief Function in the Regression Problem

### 3.2.1. Regression Problem

Consider the simple linear regression model:(44)yt=∑k=1Kβkxkt+εt,t=1,...,T,where yt is observed dependent variable at time t, xkt is an independent variable k at time t, and εt is the error that is not assumed to be any distribution when the entropy-based approach is used.


### 3.2.2. Prediction with Certain Inputs

Similar to the AR model, the prediction with linear regression can be easily computed using the method described in Section3.1.2. Let YT+1 be the future value which cannot be observed at time T+1 and xkT be the observed inputs or covariates at time T:(45)YT+1=∑k=1KβkxkT+εT,where YT+1 is a predicted value which is not yet observed and yT is the observed value at time T. We can write equivalently,(46)φyβ,s=∑k=1KβkxkT+F^−1s,s∼U0,1.Under the entropy approach, we rewrite equations (45)-(46) as(47)YT+1=φyP,s=∑k=1KxkT∑m=1Mpkmzkm+F^−1s.To forecast the future observation conditional on all information of the observed inputs at time,T, YT+1xkT, the predictive belief function and plausibility function can then be approximated using Monte Carlo simulation as described in Section 3.1.2. We, then, compute the maximum and minimum of the linear regression function φyP,s subject to plyP≥ωi for minimum and maximum problems. The additional constraints are(48)yT+1=∑k=1KxkT∑m=1Mpkmzkm+∑m=1MwTmvTm,∑m=1Mpkm=1,∑m=1Mwtm=1.Example 3.
As an example, we illustrate the proposed method by applying them to the data from the operation of a plant for the oxidation of ammonia to nitric acid, measured on 21 consecutive days. The data are analyzed by Brownlee [16] in a simple regression setting, and it is already available within R software. The dataset consists of 4 variables, namely, air flow to the plant (AIR), cooling water inlet temperature (WATER), acid concentration (percentage minus 50 times 10 (ACID), and ammonia lost (percentage times 10) (LOSS). We consider LOSS as the dependent variable, and AIR, WATER, and ACID as independent variables. Thus, we consider the following linear regression model:(49)LOSSt=β0+β1AIRt+β2WATERt+β3ACIDt+εt.
By using the Maximum likelihood (ML) and GME estimations, we can obtain the estimated parameters, standard errors, andp values. Table 6 provides the estimated results of our ML and GME estimates altogether with some basic statistics (standard errors and p values) and the plausibilities plβk=0 (see [7]). However, in the case of GME estimation, the computation of the plausibilities plβk=0 is different. Recall that, in the entropy approach, the estimated parameter of the model cannot be obtained directly from the estimation, but it is computed as the summation of all expected support points. Thus, it would be reasonable to have pl∑m=1Mpkmzkm=0. We know that if pk1=pk2=⋯=pkM=1/M, then ∑m=1Mpkmzkm=0. Hence, we turn our attention to test plpkm=1/M. In our approach, suppose M=3 and we have βk=0, whenever pk1=pk3. To compute this, we need to maximize the entropy under the usual constraints and additional constraint pk1=pk3=1/3. Therefore, we can further compute the plβk=0=plpk1=pk3. In this test, we test whether plpk1=pk3 is less than plβk=β⌢k=plp⌢k1,p⌢k2,p⌢k3. Let p⌢km be the optimal probability of variable k at support m. If plpk1=pk3<plp⌢k1,p⌢k2,p⌢k3, the hypothesis of the coefficient βk equal to zero is rejected, indicating that this coefficient is significant.
Figure4 shows the marginal contour functions plβ3=0 for parameter ACID obtained from the likelihood-based approach, and we can observe that the plβ3=0 is greater than 0.10, indicating that the acid concentration variable is said to have insignificant effect to ammonia lost. In Figure 5, the marginal contour functions of plβk=0=plpk1=pk3 are plotted for parameters AIR k=1, WATER k=2, and ACID k=3. We can observe that(50)plp11=p13<plp⌢11,p⌢12,p⌢13,plp21=p23<plp⌢21,p⌢22,p⌢23,plp31=p33=plp⌢31,p⌢32,p⌢33.
This confirms that there is no significant effect of ACID to LOSS.Figure 4
Marginal contour functions for the coefficients of variables AIR obtained from (a) normal likelihood, (b) student-t likelihood, and (c) skewed student-t likelihood.
(a)(b)(c)Figure 5
Marginal plausibility of probabilities (Shannon entropy) plotted in blue lines and the plausibility ofplp⌢k1,p⌢k2,p⌢k3 plotted in red horizon line. Similar plots are also obtained for Renyi and Tsallis entropies.
(a)(b)(c)Example 4.
Continuing Example3, we then consider the task of prediction problem. We consider the ammonia lost as the dependent variable and only AIR and WATER as the inputs since they show a significant effect on LOSS variable. We have re-estimated the model, and the result is provided in Table 7 and Figure 6. We can observe that the similar significant result of the estimated parameters is obtained for all methods. We also make a prediction of LOSS when the inputs are known at time T; hence, the prediction equation is based on(51)LOSST=β0+β1AIRT+β2WATERT+εT,with εT which is assumed to be empirical distribution. We can write, equivalently,(52)LOSST∗=β0+β1AIRT+β2WATERT+ϕ−1s.
The predictive belief function onLOSST=21∗ can then be approximated using the methods described in Section 3.2.2. Figure 6 displays the lower and upper cdfs of the various predictive belief functions, approximated using N = 5000 randomly generated focal sets. The prediction value of LOSST=21∗ is plotted by the vertical blue dotted line as well as the α quantile intervals with α=0.05,0.95. We then compare the prediction performance based on prediction bias. The true value of LOSST=21 is 15; thus, we calculate prediction bias from LOSST=21∗−15. According to the last row of Table 7, we can observe that the Shannon entropy-based belief function method outperforms the others as the lowest bias is obtained. Moreover, we find that the prediction interval of the Shannon entropy method provides a reliable interval. Although the 0.05 quantile interval does not cover the true value, the lower bound of its interval is closer to the true value = 15. Therefore, to confirm the reliability of the method, we extend the lower quantile interval of the Shannon entropy-based method to be α=0.01 and we can observe that the true observed data is contained in the 0.01 lower quantile interval (green dashed line).Table 6
Regression results.
normstdsstdShannon entropyRenyi entropyTsallis entropyEstimateIntercept−43.7037 (0.0000) [0.0000]−43.3522 (0.0000) [0.0000]−41.3097 (0.0000) [0.0000]−38.6949 (0.0000)∗∗∗−38.6950 (0.0000)∗∗∗−38.7344 (0.0000)∗∗∗AIR0.8891 (0.0000) [0.0000]0.8980 (0.0000) [0.0000]0.8605 (0.0000) [0.0000]0.7894 (0.0000)∗∗∗0.7890 (0.0000)∗∗∗0.7317 (0.0000)∗∗∗WATER0.8166 (0.0049) [0.0723]0.7823 (0.0043) [0.1001]0.8154 (0.0079) [0.0000]1.1918 (0.0007)∗∗∗1.1918 (0.0007)∗∗∗1.2315 (0.0008)∗∗∗ACID−0.1071 (0.3361) [0.2179]−0.1094 (0.3114) [0.2357]−0.1155 (0.1733) [0.2495]−0.1821 (0.2631)−0.1821 (0.2630)−0.1828 (0.2644)() isp value, [] is Plβj=0, and ∗∗∗ is plpk1=pk3<plp⌢k1,p⌢k2,p⌢k3. For the case of entropy, the support z1m is initially set to (−50, 0, 50) and the supports for vtm to (−3σ, 0, 3σ), where σis computed form the conventional LS estimation.Figure 6
Lower and upper cdfs for the prediction problem in linear regression. (a) Normal. (b) Student-t. (c) Skewed Student. (d) Shannon Entropy. (e) Renyi Entropy. (f) Tsallis Entropy.
(a)(b)(c)(d)(e)(f)Table 7
Regression results of the reduced model with only two inputs.
normstdsstdShannon entropyRenyi entropyTsallis entropyEstimateIntercept−57.0752 (0.0000) [0.0000]−43.8121 (0.0000) [0.0000]−32.0274 (0.0000) [0.0000]−51.8325 (0.0000)∗∗∗−41.4330 (0.0000)∗∗∗−50.9705 (0.0000)∗∗∗AIR0.8629 (0.0000) [0.0000]0.8354 (0.0000) [0.0000]0.5923 (0.0000) [0.0000]0.7633 (0.0000)∗∗∗0.7561 (0.0000)∗∗∗0.7029 (0.0000)∗∗∗WATER0.8033 (0.0233) [0.0000]0.5056 (0.0709) [0.0000]0.7831 (0.0079) [0.0000]1.1479 (0.0000)∗∗∗1.1479 (0.0000)∗∗∗1.1358 (0.0000)∗∗∗Expected prediction26.5564 <19.1543, −32.4528>27.0138 <8.3351, −32.3311>24.5375 <18.4833, −32.1598>23.5138 <17.9744, 34.1935>26.1581 <18.1415, 34.8077>24.0054 <17.0348, 34.5871>Prediction bias11.556412.01389.53758.513811.15819.0054<> is the prediction interval, ( ) isp value, [ ] is Plβj=0, and ∗∗∗ is plpk1=pk3<plp⌢k1,p⌢k2,p⌢k3. For the case of Entropy, the support z1m is initially set to (−100, 0, 100) and the supports for vtm to (−3σ, 0, 3σ), where σ is computed from the conventional LS estimation.


## 3.2.1. Regression Problem

Consider the simple linear regression model:(44)yt=∑k=1Kβkxkt+εt,t=1,...,T,where yt is observed dependent variable at time t, xkt is an independent variable k at time t, and εt is the error that is not assumed to be any distribution when the entropy-based approach is used.


## 3.2.2. Prediction with Certain Inputs

Similar to the AR model, the prediction with linear regression can be easily computed using the method described in Section3.1.2. Let YT+1 be the future value which cannot be observed at time T+1 and xkT be the observed inputs or covariates at time T:(45)YT+1=∑k=1KβkxkT+εT,where YT+1 is a predicted value which is not yet observed and yT is the observed value at time T. We can write equivalently,(46)φyβ,s=∑k=1KβkxkT+F^−1s,s∼U0,1.Under the entropy approach, we rewrite equations (45)-(46) as(47)YT+1=φyP,s=∑k=1KxkT∑m=1Mpkmzkm+F^−1s.To forecast the future observation conditional on all information of the observed inputs at time,T, YT+1xkT, the predictive belief function and plausibility function can then be approximated using Monte Carlo simulation as described in Section 3.1.2. We, then, compute the maximum and minimum of the linear regression function φyP,s subject to plyP≥ωi for minimum and maximum problems. The additional constraints are(48)yT+1=∑k=1KxkT∑m=1Mpkmzkm+∑m=1MwTmvTm,∑m=1Mpkm=1,∑m=1Mwtm=1.Example 3.
As an example, we illustrate the proposed method by applying them to the data from the operation of a plant for the oxidation of ammonia to nitric acid, measured on 21 consecutive days. The data are analyzed by Brownlee [16] in a simple regression setting, and it is already available within R software. The dataset consists of 4 variables, namely, air flow to the plant (AIR), cooling water inlet temperature (WATER), acid concentration (percentage minus 50 times 10 (ACID), and ammonia lost (percentage times 10) (LOSS). We consider LOSS as the dependent variable, and AIR, WATER, and ACID as independent variables. Thus, we consider the following linear regression model:(49)LOSSt=β0+β1AIRt+β2WATERt+β3ACIDt+εt.
By using the Maximum likelihood (ML) and GME estimations, we can obtain the estimated parameters, standard errors, andp values. Table 6 provides the estimated results of our ML and GME estimates altogether with some basic statistics (standard errors and p values) and the plausibilities plβk=0 (see [7]). However, in the case of GME estimation, the computation of the plausibilities plβk=0 is different. Recall that, in the entropy approach, the estimated parameter of the model cannot be obtained directly from the estimation, but it is computed as the summation of all expected support points. Thus, it would be reasonable to have pl∑m=1Mpkmzkm=0. We know that if pk1=pk2=⋯=pkM=1/M, then ∑m=1Mpkmzkm=0. Hence, we turn our attention to test plpkm=1/M. In our approach, suppose M=3 and we have βk=0, whenever pk1=pk3. To compute this, we need to maximize the entropy under the usual constraints and additional constraint pk1=pk3=1/3. Therefore, we can further compute the plβk=0=plpk1=pk3. In this test, we test whether plpk1=pk3 is less than plβk=β⌢k=plp⌢k1,p⌢k2,p⌢k3. Let p⌢km be the optimal probability of variable k at support m. If plpk1=pk3<plp⌢k1,p⌢k2,p⌢k3, the hypothesis of the coefficient βk equal to zero is rejected, indicating that this coefficient is significant.
Figure4 shows the marginal contour functions plβ3=0 for parameter ACID obtained from the likelihood-based approach, and we can observe that the plβ3=0 is greater than 0.10, indicating that the acid concentration variable is said to have insignificant effect to ammonia lost. In Figure 5, the marginal contour functions of plβk=0=plpk1=pk3 are plotted for parameters AIR k=1, WATER k=2, and ACID k=3. We can observe that(50)plp11=p13<plp⌢11,p⌢12,p⌢13,plp21=p23<plp⌢21,p⌢22,p⌢23,plp31=p33=plp⌢31,p⌢32,p⌢33.
This confirms that there is no significant effect of ACID to LOSS.Figure 4
Marginal contour functions for the coefficients of variables AIR obtained from (a) normal likelihood, (b) student-t likelihood, and (c) skewed student-t likelihood.
(a)(b)(c)Figure 5
Marginal plausibility of probabilities (Shannon entropy) plotted in blue lines and the plausibility ofplp⌢k1,p⌢k2,p⌢k3 plotted in red horizon line. Similar plots are also obtained for Renyi and Tsallis entropies.
(a)(b)(c)Example 4.
Continuing Example3, we then consider the task of prediction problem. We consider the ammonia lost as the dependent variable and only AIR and WATER as the inputs since they show a significant effect on LOSS variable. We have re-estimated the model, and the result is provided in Table 7 and Figure 6. We can observe that the similar significant result of the estimated parameters is obtained for all methods. We also make a prediction of LOSS when the inputs are known at time T; hence, the prediction equation is based on(51)LOSST=β0+β1AIRT+β2WATERT+εT,with εT which is assumed to be empirical distribution. We can write, equivalently,(52)LOSST∗=β0+β1AIRT+β2WATERT+ϕ−1s.
The predictive belief function onLOSST=21∗ can then be approximated using the methods described in Section 3.2.2. Figure 6 displays the lower and upper cdfs of the various predictive belief functions, approximated using N = 5000 randomly generated focal sets. The prediction value of LOSST=21∗ is plotted by the vertical blue dotted line as well as the α quantile intervals with α=0.05,0.95. We then compare the prediction performance based on prediction bias. The true value of LOSST=21 is 15; thus, we calculate prediction bias from LOSST=21∗−15. According to the last row of Table 7, we can observe that the Shannon entropy-based belief function method outperforms the others as the lowest bias is obtained. Moreover, we find that the prediction interval of the Shannon entropy method provides a reliable interval. Although the 0.05 quantile interval does not cover the true value, the lower bound of its interval is closer to the true value = 15. Therefore, to confirm the reliability of the method, we extend the lower quantile interval of the Shannon entropy-based method to be α=0.01 and we can observe that the true observed data is contained in the 0.01 lower quantile interval (green dashed line).Table 6
Regression results.
normstdsstdShannon entropyRenyi entropyTsallis entropyEstimateIntercept−43.7037 (0.0000) [0.0000]−43.3522 (0.0000) [0.0000]−41.3097 (0.0000) [0.0000]−38.6949 (0.0000)∗∗∗−38.6950 (0.0000)∗∗∗−38.7344 (0.0000)∗∗∗AIR0.8891 (0.0000) [0.0000]0.8980 (0.0000) [0.0000]0.8605 (0.0000) [0.0000]0.7894 (0.0000)∗∗∗0.7890 (0.0000)∗∗∗0.7317 (0.0000)∗∗∗WATER0.8166 (0.0049) [0.0723]0.7823 (0.0043) [0.1001]0.8154 (0.0079) [0.0000]1.1918 (0.0007)∗∗∗1.1918 (0.0007)∗∗∗1.2315 (0.0008)∗∗∗ACID−0.1071 (0.3361) [0.2179]−0.1094 (0.3114) [0.2357]−0.1155 (0.1733) [0.2495]−0.1821 (0.2631)−0.1821 (0.2630)−0.1828 (0.2644)() isp value, [] is Plβj=0, and ∗∗∗ is plpk1=pk3<plp⌢k1,p⌢k2,p⌢k3. For the case of entropy, the support z1m is initially set to (−50, 0, 50) and the supports for vtm to (−3σ, 0, 3σ), where σis computed form the conventional LS estimation.Figure 6
Lower and upper cdfs for the prediction problem in linear regression. (a) Normal. (b) Student-t. (c) Skewed Student. (d) Shannon Entropy. (e) Renyi Entropy. (f) Tsallis Entropy.
(a)(b)(c)(d)(e)(f)Table 7
Regression results of the reduced model with only two inputs.
normstdsstdShannon entropyRenyi entropyTsallis entropyEstimateIntercept−57.0752 (0.0000) [0.0000]−43.8121 (0.0000) [0.0000]−32.0274 (0.0000) [0.0000]−51.8325 (0.0000)∗∗∗−41.4330 (0.0000)∗∗∗−50.9705 (0.0000)∗∗∗AIR0.8629 (0.0000) [0.0000]0.8354 (0.0000) [0.0000]0.5923 (0.0000) [0.0000]0.7633 (0.0000)∗∗∗0.7561 (0.0000)∗∗∗0.7029 (0.0000)∗∗∗WATER0.8033 (0.0233) [0.0000]0.5056 (0.0709) [0.0000]0.7831 (0.0079) [0.0000]1.1479 (0.0000)∗∗∗1.1479 (0.0000)∗∗∗1.1358 (0.0000)∗∗∗Expected prediction26.5564 <19.1543, −32.4528>27.0138 <8.3351, −32.3311>24.5375 <18.4833, −32.1598>23.5138 <17.9744, 34.1935>26.1581 <18.1415, 34.8077>24.0054 <17.0348, 34.5871>Prediction bias11.556412.01389.53758.513811.15819.0054<> is the prediction interval, ( ) isp value, [ ] is Plβj=0, and ∗∗∗ is plpk1=pk3<plp⌢k1,p⌢k2,p⌢k3. For the case of Entropy, the support z1m is initially set to (−100, 0, 100) and the supports for vtm to (−3σ, 0, 3σ), where σ is computed from the conventional LS estimation.


## 4. Conclusion

In recent studies, forecasting is an important tool for decision-making and strategic planning. Describing the uncertainty of the forecasts accurately is thus a very important issue. The approach developed in this paper is to model estimation uncertainty using a belief function. The belief functionBely is just another piece of evidence. It can be combined with the likelihood belief function and the joint belief function by Dempster’s rule. The combined belief function is then, as before, marginalized on φyp,s to get the upper and lower predictive belief function.As the information of the distribution function is generally unknown, we thus have a concern that the estimation and prediction problems that are solved in likelihood function-based belief function framework might have a bias if the observed data are not normally distributed. Therefore, this study takes advantage of the entropy approach to construct a belief function and illustrate these solutions under the autoregressive (AR) and regression contexts. Specifically, we replace the parametric likelihood function with the entropy measure (Shannon, Renyi, and Tsallis) to derive the entropy-based belief function and thus the model misspecification is avoided since no distributional assumption is made.To validate the performance of our proposed method, the simulation study is conducted. The results reveal that our method performs remarkably well in comparison with a host of competing methods. We would like to highlight that if the distribution is known and we estimate the model based on the true distribution, the accurate result is obtained. However, the error distribution is sometimes unknown in practice, these results confirm the advantage of the Shannon entropy-based belief function, which performs uniformly well over a wide range of error distributions.Furthermore, our entropy-based belief function is also applied to real data. In the first real application model, we conduct the out-of-sample forecasts from the year 2014 to 2018 and the results are displayed in Table5. We find that the entropy-based belief function for AR (1) exhibits good performance in the out-of-sample forecasts relative to the other competing methods. We then apply our method to predict the growth rate of GDP in 2019 and find that the growth rate of the Thai GDP is around 3.961%. In the second real application, we apply our method to predict the future value using the linear regression with certain inputs. We can observe that the estimates under the likelihood-based methods and entropy-based methods exhibit slight differences, as expected. The forecasting bias of our entropy-based belief is lower than those via likelihood-based belief, in particular the Shannon entropy. This suggests that the entropy-based belief seems to produce more accurate estimates of the prediction under the regression context as well. From these results, it is encouraging that the use of entropy-based belief offers a better alternative in the prediction problem.In brief, this study proposes the entropy-based belief function as an alternative method in place of likelihood predictive distributions. The method is very general and can be used with any parametric distributions. However, as the computational complexity is quite high, we may face the higher computational cost in the large datasets. We leave the performance of the computation of our method to future work. In addition, for the further studies, to make a more accurate prediction, this approach may be extended to combine a random interval encoding expert assessments of the prediction from several government agencies, investors, business enterprises, and private/public institutions. Furthermore, the forecasting performance of entropy-based belief function could be improved by increasing the orderd in the Renyi and Tsallis entropies. Therefore, we suggest the future study to conduct the higher order of Renyi and Tsallis entropies. Finally, it deserves comparing our approach with the posterior distribution-based belief function in the Bayesian context and Deng entropy-based belief function to confirm the performance of our entropy-based method.



---

*Source: 3269647-2020-09-01.xml*