# Empirical Likelihood-Based Inference in Regressive Model with Moment Restrictions

Authors: Zhao Zhiwen; He Jinghua
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

In this paper, we consider the parameter estimation problem of linear regression model when the auxiliary information can be denoted by moment restrictions. We use the weighted least squares method to estimate the model parameters and to obtain the weights based on the auxiliary information by using the empirical likelihood method. The limiting distribution of the estimator is established, and the simulation studies are carried out to demonstrate the feasibility of our theoretical results.


---

## Body

## 1. Introduction

SupposeX1,Y1,X2,Y2,…,Xn,Yn is an independent and identically distributed random sample from the following regression model:(1)Yi=ατXi+εi,i=1,2,…,n,where Xi is p×1 random vector and Yi is a scalar response variable; α=α1,…,αpτ is a p×1 vector of regression coefficients, and the error εi is a sequence of i.i.d. random variables with Eεi=0 and Eεi2=σ2. Moreover, we assume that Xi,i=1,2,…, is independent of εi,i=1,2,…,.Linear regression model is widely used in empirical work in economics, medicine, and many other disciplines due to its simple form. In this paper, we consider the parameter estimating problem of linear regression model when auxiliary information is available. We care about how to decrease the estimation bias by using the auxiliary information based on the empirical likelihood method. The customary design-based estimator does not make use of auxiliary population information at the estimation stage. Observe that auxiliary information can be used to increase the precision of estimators in sample surveys. Therefore, Chambers and Dunstan [1] propose a simple method for estimating the population distribution functions which allows auxiliary population information to be directly incorporated into the estimation process. Rao et al. [2] obtain the ratio and difference estimators of a population distribution function under a general sampling design by using auxiliary population information. Chen and Qin [3] show that the empirical likelihood method can be naturally applied to finite population inference problems by making effective use of the auxiliary information. Zhang [4] employs the method of empirical likelihood to construct confidence intervals for M-functionals in the presence of auxiliary information. Zhong and Rao [5] show that making effective use of auxiliary population information can lead to efficient estimators for making inferences on finite population parameters. GarcíA and Cebrián [6] derive confidence intervals for medians in a finite population by using multiauxiliary information through a multivariate regression-type estimator of the population distribution function. By assuming auxiliary information on the unknown distribution of the data, Crudu and Porcu [7] introduce a weighted Z-estimator for moment condition models. Moreover, Tang and Leng [8] propose a weighted least squares estimation which incorporates auxiliary information to the efficiency of the estimator. Hellerstein and Imbens [9] analyze the estimation of coefficients in regression models under moment restrictions in which the moment restrictions are derived from auxiliary data. But the moment restriction does not include nuisance parameter. We further consider the parameter estimation problem of linear regression model when the auxiliary information can be denoted by moment restrictions which include nuisance parameter based on the empirical likelihood method. Specifically, the auxiliary information can be expressed as moment constraint EgXi,θ0=0, where gXi,θ=g1Xi,θ,…,grXi,θ, θ∈Rd and r≥d. It is worth mentioning that θ can be different from α. Therefore, moment constraint EgXi,θ0=0 can denote a broad class of information. We will use the least squares method to estimate the model parameter and apply empirical likelihood method to obtain the weights by using the auxiliary information.As a nonparametric method, empirical likelihood method is widely used in statistical inference of various models. Empirical likelihood method is first proposed by Owen [10–12] and is further generalized by Qin and Lawless [13]. Empirical likelihood method is used for statistical inference of various models (see Chen and Keilegom [14] and Nordman and Lahiri [15]). In addition, some statisticians have also begun to pay attention to the statistical inference of semiparametric regression model with constraints (see Amini and Roozbeh [16], Roozbeh and Hamzah [17], and Roozbeh et al. [18]).This paper proceeds as follows. We first introduce the methodology and the main results. Next, we will undertake a simulation study to illustrate the feasibility of our method. Lastly, we give the proofs of the main results.The symbols “⟶d” and “⟶p” denote convergence in distribution and convergence in probability, respectively. Convergence “almost surely” is written as “a.s.” Furthermore, A⊗B denotes the Kronecker product of matrices A and B, ⋅ denotes Euclidean norm of the matrix or vector, Aτ denotes the transposition of the matrix or the column vector A, op1 denotes a random variable that converges to zero in probability, and Op1 denotes a random variable that is bounded in probability.


## 2. Methods and Main Results

In this section, we will introduce our method and give the main results of this paper. Firstly, we obtain the weights of the weighed least squares estimator based on the moment constraintEgXi,θ0=0 by using the empirical likelihood method (see Owen [12]). Specifically, let(2)Lθ=sup∏i=1nωi:ωi≥0,∑i=1nωi=1,∑i=1nωigXi,θ=0,where θ is unknown. Combining with the least squares method, we can obtain the following weighed least squares estimator:(3)α^=argminα∑i=1nωiYi−ατXi2.Introducing Lagrange multipliersλθ0∈Rr in Lθ0, we obtain the following weights:(4)ωiθ0=1n11+λθ0τgXi,θ0,where λθ0 satisfies(5)1n∑i=1ngXi,θ01+λθ0τgXi,θ0=0.Therefore, by (3) and (4), we know that(6)α^=∑i=1nωiθ0XiXiτ−1∑i=1nωiθ0YiXi.In order to obtain the limiting distribution ofα^, we assume that the following conditions hold:A1EX1X1τ=W, and W>0.A2EgXi,θ0gτXi,θ0=Σ>0. There exists a neighborhood of θ0 and an integrable function W˜x such that, in this neighborhood, ∂gx,θ/∂θ is continuous, ∂gx,θ/∂θ≤W˜x, gx,θ3≤W˜x, and the rank of E∂gX1,θ/∂θ is d.ConditionA2 can be found in a study by Qin and Lawless [13]. The following theorem presents the asymptotic properties of α^.Theorem 1.
UnderA1 and A2 and assuming that EX14<∞, if α0 is the true value of α, then(7)nα^−α0⟶dN0,W−1Λ−Λ12Σ−1θ0Λ12τW−1,where Λ=EX1X1τY1−X1τα02 and Λ12=EX1gτX1,θ0Y1−X1τα0.
We know that the asymptotic variance of the least squares estimatorα˜=∑i=1nXiXiτ−1∑i=1nYiXi is W−1ΛW−1. Note that matrix Σ−1θ0 is nonnegative definite. Hence, Theorem 1 implies that the weighted least squares estimator has a smaller asymptotic variance compared with the least squares estimator. That is to say, using the moment restrictions does improve the effectiveness of the estimation.
α^ contains unknown parameters. Therefore, we need to further estimate the unknown parameter θ. Let θ^=argmaxθLθ in (2). Using results in a study by Qin and Lawless [13], we know that(8)ωiθ^=1n11+λθ^τgτXi,θ^,where λθ^ satisfies(9)1n∑t=1ngτXi,θ^1+λθ^τgτXi,θ^=0,and λθ^,θ^ solves(10)1n∑t=1n∂gτXi,θ^/∂θτλθ^1+λθ^τgτXi,θ^=0.
Let(11)α^′=argminβ∑t=1nωtθ^Yi−ατXi2.
There are no unknown parameters inα^′, so it can be used to estimate unknown parameters α in practice. Next, we investigate the limiting properties of α^′. For convenience sake, let Γθ0=E∂gτXi,θ0/∂θ, Ωθ0=Γθ0Σ−1θ0Γτθ0−1 and B=Σ−1θ0I−Γθ0Ωθ0Γτθ0Σ−1θ0, where I is the unit matrix. The following theorem gives the limiting property of α^′.Theorem 2.
UnderA1 and A2 and assuming that EX14<∞, if α0 is the true value of α, then(12)nα^′−α0⟶dN0,W−1Λ−Λ12BΛ12τW−1.
Similarly, since matrixB is nonnegative definite, the weighted least squares estimator α^′ also has a smaller asymptotic variance compared with the least squares estimator.


## 3. Simulation Studies

In this section, we study the finite sample properties of the above estimator by simulation. We consider the following models:Model 1.Yi=α1+α2Xi+εi,i=1,2,…,n, where the distribution of Xi is exponential with unit mean and the distribution of εi is standard normal.Model 2.Yi=α1+α2Xi+εi,i=1,2,…,n, where the distribution of Xi is exponential with unit mean. εi=λiξi+1−λiζi, where the distribution of ξi is standard normal, the distribution function of ζi is N(0,9), and the probability distribution of λi is Pλi=1=1−Pλi=0=1−p.Model 3.Yi=α1X1i+α2X2i+εi,i=1,2,…,n, where the distribution of X1i is exponential with mean 0.5, the distribution of X2i is exponential with unit mean, and the distribution of εi is standard normal.In order to illustrate the feasibility of our proposed method, we compute the mean absolute deviation ratio of the estimator given by (11) to the least squares estimator. For model 1 and model 2, we use gXi,θ=θ1Xi−1,θ1Xi2−2,θ2Xi−1 as moment restrictions. For model 3, we use gX2i,θ=θ1X2i−1,θ1X2i2−2,θ2X2i−1 as moment restrictions.For each experiment, we conduct 1000 repetitions, and three alternative sample sizes (n = 30,100 and 300) are considered. The simulation results for model 1 are summarized in Table 1. For model 2, we take different pollution level p=0.2,0.4, and the simulation results are presented in Tables 2 and 3, respectively. Moreover, the simulation results for model 3 are summarized in Table 4.Table 1
The simulation results for model 1.
α1,α2n = 30n = 100n = 3000,10.1503,0.0962×10−30.4429,0.0805×10−30.3574,0.0265×10−30,2(0.0013,0.0001)0.5059,0.1128×10−30.6640,0.0125×10−30,3(0.0018,0.0001)(0.0014,0.0001)(0.0018,0.0000)0,40.3198,0.0327×10−3(0.0066,0.0002)(0.8846,0.0121)0,5(0.0234,0.0012)(0.0094,0.0001)(0.2887,0.0035)3,1(0.0016,0.0037)0.1341,0.4235×10−30.5841,0.8164×10−33,20.6284,0.6922×10−3(0.0013,0.0016)(0.0009,0.0012)3,3(0.0622,0.1554)0.6129,0.5704×10−30.2794,0.2578×10−33,4(0.0028,0.0018)0.2891,0.1785×10−3(0.0013,0.0009)3,5(0.0027,0.0014)0.1241,0.0512×10−30.1882,0.1078×10−36,10.0589,0.9486×10−3(0.0003,0.0021)0.0692,0.4257×10−36,2(0.0003,0.0011)0.0529,0.2192×10−30.1971,0.6406×10−36,30.1244,0.2126×10−30.3027,0.7651×10−30.2588,0.9244×10−36,40.0604,0.1164×10−30.2382,0.3704×10−30.2163,0.3758×10−36,50.2337,0.3729×10−303247,0.5159×10−40.1075,0.1288×10−3Table 2
The simulation results for model 2 (p=0.2).
α1,α2n = 30n = 100n = 3000,10.7218,0.1631×10−30.2068,0.3868×10−30.4816,0.0390×10−30,2(0.0023,0.0004)(0.0730,0.0198)(0.0016,0.0002)0,30.5612,0.2035×10−40.4688,0.0697×10−3(0.2909,0.0131)0,4(0.0044,0.0003)0.7965,0.0492×10−3(0.0090,0.0002)0,50.4604,0.1192×10−3(0.4088,0.0280)(0.1139,0.0025)3,1(0.0015,0.0068)(0.0044,0.0139)0.1750,0.0806×10−33,2(0.0129,0.0143)(0.0013,0.0020)0.5347,0.9881×10−33,3(0.0011,0.0009)(0.0095,0.0068)(0.0025,0.0036)3,40.2909,0.3911×10−3(0.0011,0.0009)0.8104,0.8536×10−33,50.1006,0.0220×10−30.7710,0.6039×10−3(0.0047,0.0028)6,1(0.0005,0.0032)0.0283,0.3442×10−3(0.0002,0.0001)6,20.0262,0.3567×10−30.0566,0.3088×10−3(0.0006,0.0021)6,30.1006,0.0679×10−3(0.0009,0.0023)(0.0009,0.0022)6,40.2790,0.5487×10−30.1418,0.2085×10−3(0.0029,0.0045)6,5(0.0007,0.0011)0.4147,0.5056×10−30.0713,0.3029×10−3Table 3
The simulation results for model 2 (p=0.4).
α1,α2n = 30n = 100n = 3000,10.0097,0.4296×10−3(0.0010,0.0008)0.1758,0.1850×10−30,20.9817,0.3201×10−30.4836,0.0195×10−3(0.0011,0.0000)0,3(0.0072,0.0065)(0.0027,0.0010)(0.0037,0.0001)0,4(0.0116,0.0014)(0.0012,0.0002)(0.3652,0.0319)0,5(0.0027,0.0003)(0.0064,0.0007)(0.0059,0.0001)3,10.1768,0.5159×10−3(0.0020,0.0059)(0.0303,0.0777)3,2(0.0081,0.0105)(0.0025,0.0038)(0.0104,0.0211)3,3(0.0101,0.0088)(0.0011,0.0008)0.0517,0.1450×10−33,40.6036,0.3940×10−3(0.0017,0.0012)(0.0036,0.0027)3,5(0.0019,0.0011)(0.0033,0.0020)(0.0058,0.0028)6,1(0.0008,0.0020)(0.0007,0.0056)(0.0007,0.0056)6,20.0726,0.1935×10−30.3039,0.8903×10−3(0.0006,0.0020)6,30.3542,0.1108×10−30.4838,0.2409×10−40.0006,0.0011×10−36,4(0.0008,0.0012)(0.0056,0.0102)(0.0013,0.0020)6,50.1707,0.2671×10−30.3853,0.4861×10−30.2360,0.2885×10−3Table 4
The simulation results for model 3.
α1,α2n = 30n = 100n = 3000,10.5517,0.0218×10−30.7855,0.1065×10−30.6022,0.0350×10−30,2(0.0015,0.0001)(0.0028,0.0001)0.6271,0.0008×10−30,3(0.0024,0.0000)0.4016,0.0078×10−3(0.0041,0.0000)0,4(0.0042,0.0001)(0.0073,0.0001)(0.0118,0.0002)0,5(0.0026,0.0000)(0.0088,0.0001)(0.0131,0.0001)3,10.0113,0.5320×10−30.0175,0.2675×10−3(0.0306,0.5866)3,20.0671,0.1866×10−30.0302,0.3476×10−30.0799,0.6932×10−33,30.1395,0.5421×10−30.0174,0.1121×10−30.0232,0.2970×10−33,40.1309,0.3677×10−30.1905,0.1747×10−3(0.0129,0.0251)3,50.0390,0.2233×10−30.0205,0.3749×10−30.6382,0.1322×10−46,10.0057,0.2006×10−30.1223,0.2038×10−30.0544,0.1760×10−36,20.0140,0.3297×10−30.0864,0.4524×10−30.0568,0.7259×10−36,30.0060,0.2511×10−30.0181,0.2149×10−30.0092,0.2824×10−36,40.0143,0.2344×10−30.0378,0.2712×10−30.2272,0.0901×10−36,50.0078,0.1755×10−30.0235,0.2951×10−30.7367,0.3097×10−3It can be seen from Tables1–4 that the ratio of the mean absolute deviations of the least squares estimation with moment restrictions to that of the ordinary least squares estimation is very small. This implies that, for the different sample sizes, the different error, and the different parameters, the least squares estimation with moment restrictions is a more precise estimator. Hence, utilizing the auxiliary information indeed improves the efficiency of estimation.


## 4. Proofs of the Main Results

In order to prove Theorem1, we first present several lemmas.Lemma 1.
IfA1 and A2 hold, then(13)max1≤i≤ngXi,θ0=opn1/2.Proof.
Let(14)Σnθ0=1n∑i=1ngXi,θ0gτXi,θ0.
From the law of large numbers, we know that(15)Σnθ0⟶a.s.Σθ,asn⟶∞.
Using the method of the proof of Lemma2 in [19], we can prove that (13) holds. Lemma 1 is established.Lemma 2.
UnderA1 and A2 and assuming that EX14<∞, if α0 is the true value of α, then(16)1n∑i=1nXiτα0,gτXi,θ0τ⟶dN0,M,where(17)M=ΛΛ12Λ12τΣ.Proof.
According to the central limit theorem of independent and identical distribution, it is easy to know that Lemma2 holds.Lemma 3.
UnderA1 and A2 and assuming that EX14<∞, we have(18)λθ0=Opn−1/2.Proof.
Letλθ0=ξβ0, where β0=1,ξ=λθ0. By (5), we know that(19)0=β0τn∑i=1ngXi,θ01+ξβ0τgXi,θ0=β0τn∑i=1ngXi,θ0−ξn∑i=1nβ0τgXi,θ021+ξβ0τgXi,θ0≤β0τn∑i=1ngXi,θ0−ξ1+ξmax1≤i≤ngXi,θ0β0τΣnθ0β0.
Therefore, we have(20)ξβ0τΣnθ0β0−max1≤i≤ngXi,θ0β0τn∑i=1ngXi,θ0≤β0τn∑i=1ngXi,θ0.
Note that(21)β0τn∑i=1ngXi,θ0≤1n∑i=1ngXi,θ0=OPn−1/2.
Furthermore, by Lemma1, we obtain(22)max1≤i≤ngXi,θ0β0τn∑i=1ngXi,θ0=oP1.
By the law of large numbers for independent and identically distributed random variables, we have(23)β0τΣnθ0β0⟶a.s.β0τΣθ0β0.
Combining with (21) and (22), we know that Lemma 3 holds.Proof.
of Theorem 1.
By (5), we know that(24)λθ0=Σnθ0−11n∑i=1ngXi,θ0+Σnθ0−1Rnθ0,where(25)Rnθ0=1n∑i=1ngτXi,θ0λθ0τgXi,θ021+λθ0τgXi,θ0.
By Lemmas1–3, we know that(26)Rnθ0=oPn−1/2.
By using Taylor’s formula for1/1+λθ0τgXi,θ0, we have(27)11+λθ0τgXi,θ0=1−λθ0τgXi,θ0+oλθ0τgXi,θ0.
So, we have(28)ωi=1n1−λθ0τgXi,θ0+oλθ0τgXi,θ0.
By Lemma1 and Lemma 3, we have(29)max1≤i≤nλθ0τgXi,θ0=oP1.
Therefore, we have that uniformly fori,(30)oλθ0τgXi,θ0λθ0τgXi,θ0=oP1.
Note that(31)nα^−α0=Tn−1Sn,where(32)Tn=∑i=1nωiθ0XiXiτ,Sn=n∑i=1nωiθ0XiYi−α0τXi.
Firstly, we prove that(33)Tn⟶PW.
After simple algebra calculation, we have(34)Tn=1n∑i=1n1−λθ0τgXi,θ01+oP1XiXiτ=1n∑i=1nXiXiτ−1n∑i=1nXiXiτ⊗Σnθ0−1Rnθ0τgXi,θ01+oP1−1n∑i=1nXiXiτ⊗Σnθ0−1×1n∑i=1ngXi,θ0τgXi,θ01+oP1≜U1−U2−U3.
By the law of large numbers for independent and identically distributed random variables, we have(35)U1⟶PW.
Furthermore, by (26) and Lemma 2, we know that(36)U2=oP1,(37)U3=oP1.
Combined with (36) and (37), this establishes (33).
In the following, we considerSn. By (26) and (26), we know that(38)Sn=1n∑i=1n1−λθ0τgXi,θ0×1+oP1XiYi−α0τXi=1n∑i=1nXiYi−α0τXi−1n∑i=1nYi−α0τXiXigτXi,θ0Σnθ0−1×1n∑i=1ngXi,θ0+oP1.
By the law of large numbers for independent and identically distributed random variables, we know that(39)1n∑i=1nYi−α0τXiXigτXi,θ0⟶a.s.Λ12.
Furthermore, by Lemma2, we know that(40)Sn⟶dN0,Λ12Σ−1Λ12τ.
Combined with (31), we know that Theorem 1 holds.Proof.
of Theorem 2
Using the results in [13], we know that(41)λθ^=B1n∑i=1ngτXi,θ0+oPn−1/2.
After simple algebra calculation, we have(42)nα^′−α0=T˜n−1S˜n,where T˜n=∑i=1nωiθ^XiXiτ,S˜n=n∑i=1nωiθ^XiYi−α0τXi. Similar to the proof of Theorem 1, we know that(43)T˜n⟶PW,S˜n⟶dN0,Λ−Λ12BΛ12τ.
Thus, Theorem2 is established.


## 5. Conclusions

In this paper, we discuss how to use auxiliary information to improve the efficiency of regression model parameter estimation when auxiliary information exists. First of all, we use the auxiliary information to establish the estimation equation. Based on this estimation equation, we use empirical likelihood method to obtain the weight of weighted least squares estimation of regression model parameters and then give the weighted least squares estimation of model parameters. Secondly, it is noticed that the auxiliary information contains unknown parameters, so the weighted least squares estimator also contains unknown parameters. Therefore, we use the maximum empirical likelihood estimation to further estimate the unknown parameters contained in the auxiliary information. Therefore, we then obtain the weighted least squares estimation that can be used in practice. Finally, we use simulation analysis to illustrate the feasibility of our method.



---

*Source: 3415150-2020-12-29.xml*