Kolmogorov-Smirnov type test for generated
variables

Taisuke Otsu and Go Taniguchi
July 21, 2019
KEO Discussion Paper No. 148

KEO Discussion Paper No. 148

KOLMOGOROV-SMIRNOV TYPE TEST FOR GENERATED VARIABLESTAISUKE OTSU AND GO TANIGUCHI
Abstract. Distribution homogeneity testing, particularly based on the Kolmogorov-Smirnov
statistic, has been applied in various empirical studies. In empirical economic analysis, it isoften the case that economic variables of interest are obtained as fitted values or residuals of
preliminary model fits, called generated variables. In this paper, we extend the KolmogorovSmirnov type homogeneity test to accommodate such generated variables, and propose an
asymptotically valid bootstrap procedure. A small simulation study illustrates that it is crucial
for reliable inference to account for estimation errors in the generated variables. The proposed
method is applied to compare the total factor productivities across diﬀerent countries.

1. Introduction
In various areas of empirical studies, researchers are often interested in testing homogeneityof distributions across diﬀerent samples. The most popular approach for distribution homogeneity testing is based on the Kolmogorov-Smirnov statistic, which is obtained as the largest
discrepancy of the empirical distribution functions by these samples, and statistical theory of
the Kolmogorov-Smirnov test is well known (e.g., Lehmann and Romano, 2005). An obvious
premise behind this statistical theory is that the samples of interest are all observable.
In empirical economic analysis, however, it is often the case that researchers are interested
in the distributions of latent or theoretical variables, which are unobservable but may be estimated by observable data. Such variables, called generated variables, are often obtained as
fitted values or residuals of preliminary regression fitting. Common examples include expectedvalues of prices or sales, total factor productivity, relative quality of firms, among others. A
major econometric issue of use of generated variables is that its statistical inference requires toaccount for estimation errors contained in the generated variables. Pagan (1984) investigated
how to modify standard errors for regression analysis using generated regressors. Hahn and
Ridder (2013) studied standard error formulae for semiparametric estimators that involve generated variables. Matsushita and Otsu (2016) proposed a likelihood-based inference method
for semiparametric models with generated variables.
In this paper, we consider distribution homogeneity testing of generated variables. The test
statistic is the Kolmogorov-Smirnov statistic using the generated variables. However, the null
distribution is diﬀerent from the conventional one due to the estimation errors to construct
the generated variables, and takes a somewhat complicated form. Therefore, we propose abootstrap procedure to compute the critical value for the Kolmogorov-Smirnov statistic. A
1

key ingredient of our bootstrap procedure is to recenter for the bootstrap statistic to impose
the null hypothesis (cf. Whang, 2001). By adapting the asymptotic theory developed in
Linton, Maasoumi and Whang (2005), the asymptotic validity of our bootstrap procedure canbe established.
The proposed homogeneity test is illustrated by Monte Carlo simulation and an empiricalapplication. Our simulation study illustrates that it is crucial for reliable inference to account for
estimation errors in the generated variables. Especially the conventional Kolmogorov-Smirnov
test may exhibit severe size distortions. Also our empirical application on comparisons of thetotal factor productivities across diﬀerent countries illustrate usefulness of the proposed test.

2. Distribution homogeneity test
Consider scalar latent variables Xki for k = 1, . . . , K, which are not directly observable.
Suppose that the latent variables are specified as Xki = g(Zki , ✓k0 ), where g is a known function
up to the unknown parameters ✓k0 and Zki is a vector of observables. Let Fk be the cumulative distribution function of Xk for k = 1, . . . , K. We wish to conduct hypothesis testing on
distribution homogeneity of generated variables
H0 : F1 (x) = · · · = FK (x) for all x,
against H1 : H0 is false.
If {Xki }ni=1 is directly observable, then various homogeneity tests are available in the literature. In this paper we focus on the situation where some estimates ✓ˆk for the parameters ✓k0 are available to the researcher so that the generated variables X̂ki = g(Zki , ✓ˆk ) can
be constructed for i = 1, . . . , n. Typical examples of generated variables are fitted values
(1)
(2)0
0 ˆ
(X̂ki = Zki✓k0 ) and residuals (X̂ki = Zki
Zki ✓ˆk0 ) by preliminary OLS linear regression fitP
ting. Let F̂k (x) = n 1 ni=1 I{X̂ki  x} be the empirical distribution function based on thegenerated variables {X̂ki }ni=1 , where I{·} is the indicator function. F̂k (x) is a consistent estimator of Fk (x) under mild regularity conditions as far as ✓ˆk is consistent for ✓k0 . In orderto test the distribution homogeneity hypothesis H0 , we employ the Kolmogorov-Smirnov typestatistics
KS1 = max supp

KS2 = max supp

k6=l x2X

k

x2X

n|F̂k (x)F̂l (x)|,n F̂k (x)1 X
F̂l (x) ,
K l=1

(1)

K

where X is a given set. The first statistic KS1 is the maximal pairwise sup-norm distance ofthe empirical distributions {F̂k (·)}K
k=1 . The second statistic KS2 is obtained by the maximalPK1deviation from the average K
l=1 F̂l (x), which is equivalent to the empirical distribution
by the pooled sample {X̂ki }ni=1 over k = 1, . . . , K. When the number of diﬀerent samples K islarge, KS2 is computationally more attractive.

2

The limiting distributions of the Kolmogorov-Smirnov statistics under the null hypothesis
H0 are obtained as follows.

Proposition. For each k = 1, . . . , K, suppose(i): {Zki }ni=1 is a strictly stationary and ↵-mixing sequence with the mixing coeﬃcient↵(m) = O(m a ) for some a > max{(q1)(q + 1), 1 + 2/r}, where q is an even integer

satisfying q > 3(max{dim ✓1 , . . . , dim ✓K } + 1)/2 and r appears in (ii) below. For aneighborhood Nk around ✓k0 , Fk (x, ✓k ) = P {g(Zki , ✓k )  x} is diﬀerentiable on ✓k 2
Nk , supx2X ,✓:|✓ ✓k0 | n |@Fk (x, ✓k )/@✓k@Fk (x, ✓k0 )/@✓k | ! 0 for any positive sequence2
n ! 0, supx2X |@Fk (x, ✓k0 )/@✓k | < 1, and E[sup✓k 2Nk |@g(Zki , ✓k )/@✓k | ] < 1 ;Pp(ii): the estimator ✓ˆk satisfies n(✓ˆk ✓k0 ) = n 1/2 ni=1 k (Zki , ✓k0 )+op (1) with E[ k (Zki , ✓k0 )] =0 and E[| k (Zki , ✓k0 )|2+r ] < 1 for some r > 0.

Then under H0 ,d

KS1 ! max sup ⌫kl (x) +k6=l x2X

KS2@Fk (x, ✓k0 )⇠k@✓k0@Fl (x, ✓l0 )
⇠l ,
@✓l0

K ⇢
1 X
@Fk (x, ✓k0 )! max sup⌫kl (x) +⇠k
k x2X K
@✓k0
l=1
d

(2)

@Fl (x, ✓l0 )
⇠l
@✓l0

,

where (⌫kl (·), ⇠k0 , ⇠l0 ) is a mean zero Gaussian process with the covariance kernel Ckl (x1 , x2 ) =limn!1 nE[Skl,n (x1 )Skl,n (x2 )0 ],n

Skl,n (x) =⌫ˆk (x, ✓k0 )⌫ˆk (x, ✓k ) = n 1/2

n
X
i=1

⇤

1 X
⌫ˆl (x, ✓l0 ), pn i=1{I{g(Zki , ✓k )  x}

n

1 X
0
k (Zki , ✓k0 ) , p
n i=1l (Zli , ✓l0 )

0

!0

,

Fk (x, ✓k )}.

If Xki ’s are directly observable, then the limiting null distributions of KS1 and KS2 reduceP
to maxk6=l supx2X |⌫kl (x)| and maxk supx2X |K 1 K
l=1 ⌫kl (x)|, respectively. Thus, the terms in(2) containing ⇠k ’s are considered as correction terms to account for the estimation errors of
the parameter estimators, ✓ˆk ’s.
The assumptions for this proposition are adapted from Linton, Maasoumi and Whang (2005)
to our setup. Also, the proof is obtained by modifying that of Linton, Maasoumi and Whang(2005, Theorem 1). Here we sketch the proof for the case of KS1 . The proof for KS2 is similar.
Under H0 (i.e., Fk (x, ✓k0 )KS1 = max sup |ˆ⌫k (x, ✓ˆk )k6=l x2X

Fl (x, ✓l0 ) = 0), KS1 can be written aspp⌫ˆl (x, ✓ˆl ) + n{Fk (x, ✓ˆk ) Fl (x, ✓ˆl )}n{Fk (x, ✓k0 )3

Fl (x, ✓l0 )}|.

By Linton, Maasoumi and Whang (2005, Lemmas 2 and 3),p

sup |ˆ
⌫k (x, ✓ˆk )⌫ˆk (x, ✓k0 )| ! 0,

x2X

sup

p

x2X

n

n Fk (x, ✓ˆk )@Fk (x, ✓k0 ) 1 X
@✓k0
n i=1

Fk (x, ✓k0 )for k = 1, . . . , K. Therefore, we obtain
KS1 = max sup
k6=l x2X⌫ˆk (x, ✓k0 )⌫ˆl (x, ✓l0 )@Fk (x,✓k0 ) 1 Pn
+ @✓0
i=1 k (Zki , ✓k0 )nkk (Zki , ✓k0 )@Fl (x,✓l0 ) 1 Pn
i=1
@✓l0
n

p

! 0,

+ op (1),l (Zli , ✓l0 )(3)

and the conclusion in (2) follows by the weak convergence of the empirical process ⌫ˆk (x, ✓k0 )and continuous mapping theorem.
We can also show that the Kolmogorov-Smirnov tests based on KS1 and KS2 are consistent under fixed alternative hypotheses and have non-trivial power under the local alternative
hypotheses at the rate of n 1/2 .
Due to the correction terms associated with ⇠k ’s, the limiting null distributions of theKolmogorov-Smirnov type statistics are not asymptotically pivotal and somewhat complicated.
Thus, we suggest the following bootstrap procedure to approximate the critical values.
Bootstrap procedure for critical values:⇤⇤(1) Draw the bootstrap resample {Z1i, . . . , ZKi}ni=1 from the joint empirical distribution of{Z1i , . . . , ZKi }n , and compute the estimator {✓ˆ⇤ , . . . , ✓ˆ⇤ } by using {Z ⇤ , . . . , Z ⇤ }n .
i=1

1

k6=l

KS2⇤ = max sup
k

x

p

x

K

n F̂k⇤ (x)

Pn

1 X ⇤
F̂ (x)
K l=1 l(F̂k (x)Ki i=1F̂l (x)}|,K⇤ ˆ⇤
i=1 I{g(Zki , ✓k )  x}.
⇤ B⇤ B(3) Repeat (2) B times to obtain {KS1,b}b=1 or {KS2,b}b=1 .

where F̂k⇤ (x) = n 1

1i

K(2) Then compute the recentered bootstrap statistics
p
KS1⇤ = max sup n|F̂k⇤ (x) F̂l⇤ (x) {F̂k (x)1 X
F̂l (x)
K l=1

)

,

Their quantiles provide thebootstrap critical values of KS1 or KS2 to test H0 .

Note that the recentering in KS1⇤ or KS2⇤ is crucial to impose the null hypothesis H0 . The
idea of recentering was suggested in the literature by Hall and Horowitz (1996), Whang (2001),
and Linton, Maasoumi and Whang (2005), for example. To obtain an intuition, consider thebootstrap counterpart KS1⇤ . We can show that KS1⇤ satisfies
KS1⇤ = max sup
k6=l x2X⌫ˆk⇤ (x, ✓ˆk )+⌫ˆl⇤ (x, ✓ˆl )@Fk (x,✓k0 ) 1 Pn
@✓k0

n

i=1

⇤ ˆ
k (Zki , ✓k )@Fl (x,✓l0 ) 1 Pn
i=1
@✓l0
n

⇤ ˆ
l (Zli , ✓l )+ op (1),conditional on {Z1i , . . . , ZKi }ni=1 with probability one, whereP⇤⌫ˆk⇤ (x, ✓k ) = n 1/2 ni=1 [I{g(Zki, ✓k )  x} I{g(Zki , ✓k )  x}]. This expression is analogous to4(3) and guarantees the asymptotic validity of our bootstrap procedure. Without recentering in
KS1⇤ , we would have additional terms in the above expansion of KS1⇤ that may diverge.
3. Simulation
To illustrate the finite sample performance of our Kolmogorov-Smirnov type tests, we conduct
a small simulation study. We consider two regression models
Y1i = 1 + (1, 1)X1i + e1i ,Y2i = 1 + (1, 1)X2i + e2i ,(4)for i = 1, . . . , n, where X1i and X2i are bivariate regressors generated from
!
!!
!
!!
0
1 0.5
0
1 0.7
X1i ⇠ N,,
X2i ⇠ N
,
.
0
0.5 1
0
0.7 1
In this setup, we consider homogeneity testing of the distribution functions of the error terms
e1 and e2 , i.e.,
H0 : Fe1 (t) = Fe2 (t) for all t,
based on the OLS residuals ê1i and ê2i for the regressions from Y1 on X1 and Y2 on X2 ,respectively.
For the test statistic KS1 in (1), we compare the proposed bootstrap procedure with the conventional Kolmogorov-Smirnov critical value that does not take into account for the estimation
errors ê1i

e1i and ê2i

e2i . Note that the conventional critical value is asymptotically invalid

and is used to illustrate importance of accommodating the estimation errors for the generated
variables.
To evaluate the size properties, we consider three distributions of e1 and e2 : N (0, 1), standardized t(3), standardized

2(3), and standardized LN (0, 1) (log-normal generated by exp(Z)for Z ⇠ N (0, 1)). For the sample size, we consider n =100, 200, and 500. The number of boot-strap replications is 99 and the number of Monte Carlo replications is 1000. The nominal sizeis 0.05. Table 1 presents the rejection frequencies under the null hypotheses. Our bootstrap
procedure works well for all cases although it shows under-coverage for most cases. On theother hand, the conventional Kolmogorov-Smirnov critical value clearly fails to control the sizefor the case of LN (0, 1). This indicates that it is crucial to take into account for the estimation
errors of the generated variables to conduct homogeneity testing.
We next evaluate power properties of the proposed test. Now the error terms in (4) are
generate by
e1i =

p

⇢✏0i +

e2i =p⇢✏0i +pp1⇢✏1i ,

1⇢✏2i ,

for i = 1, . . . , n, where ⇢ 2 {0.0, 0.2, 0.4}, ✏0i ⇠ N (0, 1), ✏1i ⇠ N (0, 1), and ✏2i follows thestandardized t(3), standardized

e2i become similar as ⇢ increases.

2(3), and standardized LN (0, 1). The distributions of e1i and5

Table 1. Rejection Frequencies (Size)
F e1 = F e2
N (0, 1)
t(3)
2

(3)

LN (0, 1)

MethodBootstrap
Conventional KS
Bootstrap
Conventional KS
Bootstrap
Conventional KS
Bootstrap
Conventional KS

100
.001
.001
.009
.052
.009
.005
.043
.390

n
200
.003
.001
.007
.064
.017
.072
.048
.441

500
.006
.002
.013
.078
.018
.069
.053
.506

Table 2 presents the rejection frequencies under the alternative hypotheses based on 1000
Monte Carlo replications. The proposed bootstrap test shows reasonable power properties
when the sample size is large enough. The power of the conventional Kolmogorov-Smirnov
for LN (0, 1) is spurious because of severe over-rejection under the null hypothesis. Also, as⇢ increases, the power of the conventional Kolmogorov-Smirnov deteriorates faster than thebootstrap test. Finally, although the results are not reported, we find that the bootstrap
statistic without recentering has zero power for all cases.
Table 2. Rejection Frequencies (Power)
n = 100⇢
F ✏2
Method
0.0 0.2 0.4t(3)Bootstrap.087 .024 .007
Conventional KS .128 .027 .007
2
(3)
Bootstrap.114 .045 .016
Conventional KS .255 .063 .012
LN (0, 1)
Bootstrap.312 .195 .038
Conventional KS .979 .384 .067

n = 200
n = 500⇢⇢
0.0 0.2 0.4 0.0 0.2 0.4.378 .108 .026 .946 .579 .191
.495 .107 .022 .987 .539 .084
.605 .243 .061 1 .883 .416
.779 .239 .049 1 .841 .255
.780 .719 .237 .997 .997 .901
1 .884 .260 1
1 .855

4. Empirical illustration: Total factor productivity
We apply our Kolmogorov-Smirnov type test to compare the total factor productivities among
diﬀerent countries. Following Solow’s (1957) classical approach, we specify the production
function as Yt = At Kt↵ Lt , where Yt is total output, Kt and Lt are capital and labor inputs,
respectively, and At is total factor productivity. Solow (1957) specified the production function as Yt = At F (Kt , Lt ), where Yt is total output, Kt and Lt are capital and labor inputs,
respectively, and At is total factor productivity.1 If we specify F by the Cobb-Douglas function
F (Kt , Lt ) = Kt↵ Lt , we can derivelog(At ) =log(Yt )↵ log(Kt )1The capital input K

log(Lt ).

(5))Kt 1 , where It andt is computed by the permanent inventory method, i.e., Kt = It +(1are gross investment and depreciation rate, respectively. We use the data on gross fixed capital formulation
for It , and set the depreciation rate as = 0.05. The initial capital stock K0 is calculated by K0 = I0 / .
6

We use the datasets oﬀered by the Federal Reserve Bank of St. Louis (FRED), OECD, United
Nations Statistics Division, and World Bank (see Table 3).
Table 3. List of variables
VariableY
I
L

Measurement
Gross domestic products in the country (World Bank)
Gross fixed capital formation in the country (FRED)
Population (OECD)⇥ Percentage of working age population (OECD)⇥ Average annual hours worked by persons engaged for the country (FRED)

Using these data, we calculate the annual growth rate of total factor productivity from 1971 to2014 for 15 countries (Australia, Austria, Belgium, Canada, Denmark, France, Germany, Italy,
Japan, Netherlands, Norway, South Korea, Sweden, United Kingdom, and United States). In
particular,log(At ) is estimated by evaluating ↵ andin (5) with the OLS estimator.

We apply the Kolmogorov-Smirnov type test for generated variables to test homogeneity ofpairs of distributions oflog(At ) from 15 countries. As in the simulation study, we compare

our bootstrap method with the conventional Kolmogorov-Smirnov critical value that does nottake into account for the estimation errors. The results are presented in Table 4. We cansee that the conclusions of the tests are diﬀerent for several cases (indicated by bold letters).
Also those conclusions can be diﬀerent in either ways. For reliable inference, it is critical toincorporate estimation errors for generated variables as in our bootstrap method.

7

8Australia
Austria
Belgium
CanadaDenmark
FranceGermany
Italy
Japan
Korea
Netherlands
NorwaySwedenUK

Austria Belg. Canada Denm. France Germ.
Italy
Japan
Korea
Neth.
(i) (ii) (i) (ii) (i) (ii) (i) (ii) (i) (ii) (i) (ii) (i) (ii) (i) (ii) (i) (ii) (i) (ii)
R R R R N N R N R R R R N N N N N N N R
N N N N N N N N N N N N R R R R N N
N N N N N N N N N N R R N N N N
N N R N R R N N N N N N N N
N N N N N N R N R R N N
N N N N R R R R N N
R N R R R R N N
N N R N N N
N N R R
R R

Norway Sweden
UKUS(i) (ii) (i) (ii) (i) (ii) (i) (ii)
N N R R R R R N
R N N N N N N N
N N N N N N N N
N N N N N N N N
R N N N N N N N
R R N N N N N N
R R N N N N R N
N N N N N N N N
N N R R N N R R
R N R R R R R R
R R N N N N N N
R R N N N N
N N N N
N N

Table 4. Test results for (i) Bootstrap (ii) Conventional KS (R=reject, N=not reject)References[1] Hahn, J. and G. Ridder (2013) Asymptotic variance of semiparametric estimators with generated regressors,
Econometrica, 81, 315-340.
[2] Hall, P. and J. L. Horowitz (1996) Bootstrap critical values for tests based on generalized-method-of-moments
estimators, Econometrica, 64, 891-916.
[3] Lehmann, E. L. and J. P. Romano (2005) Testing Statistical Hypotheses, 3rd ed., Springer.
[4] Linton, O., Maasoumi, E. and Y.-J. Whang (2005) Consistent testing for stochastic dominance under general
sampling schemes, Review of Economic Studies, 72, 735-765.
[5] Matsushita, Y. and T. Otsu (2016) Likelihood inference on semiparametric models with generated regressors,
Working paper.
[6] Pagan, A. (1984) Econometric issues in the analysis of regressions with generated regressors, International
Economic Review, 25, 221-247.
[7] Solow, R. M. (1957) Technical change and the aggregate production function, Review of Economics andStatistics, 39, 312-320.
[8] Whang, Y.-J. (2001) Consistent specification testing for conditional moment restrictions, Economics Letters,
71, 299-306.
Department of Economics, London School of Economics, Houghton Street, London, WC2A
2AE, UK, and Keio Economic Observatory (KEO), 2-15-45 Mita, Minato-ku, Tokyo 108-8345,
Japan.
Email address: t.otsu@lse.ac.uk
Research Center for Policy and Economy, Mitsubishi Research Institute, Inc., 10-3, Nagatacho 2-chome, Chiyoda-ku, Tokyo 100-8141, Japan.
Email address: go_taniguchi@mri.co.jp

9