JOINT RESEARCH CENTER FOR PANEL STUDIESSELECTED DISCUSSION PAPER SERIESSDP2012-003
March, 2013

Effect of Adding Web-Mode on Unit-Response in Sample Survey:Evaluating
the Effect of Adding an Alternative When No Control Group Exists

Koji Yamamoto*
Michio Naoi**【Abstract】
The first wave of Japan Household Panel Survey (JHPS) was designed to letparticipants respond to questionnaires via the Web rather than paper if they desired. In
this study, we investigate whether introducing the Web option altered the response rateof the survey. We emphasize that eventual Web respondents would likely have
responded even in the absence of this mode. In general, when gauging the effect oftreatments, such as policy-related treatments, a control group is usually required,regardless of whether the design is experimental or quasi-experimental. However, insome cases, a control group cannot be assigned into or observed.We show that theeffectof adding an alternative option can be estimated without a control group,undercertain assumptions. We use this approach to estimate increase in unit-response due toadding the Web mode.The result shows that the options “Web response” and “paper
response” were mutually easily substitutable, and, only few respondents would not have
responded unless Web mode was available. In short, the net rise in unit-response
introduced by Web mode was only limited. Inaddition, in this paper, we show thecharacteristics of those who had PC and Internet access, which enabled online
responses, and those who actually responded via Web mode.

* JSPS Fellow, Chuo Univeristy**

Assistant Professor, Tokyo University of Marine Science and Technology

Joint Research Center for Panel Studies
Keio University

Effect of Adding Web-Mode on Unit-Response in Sample Survey:
Evaluating the Effect of Adding an Alternative
When No Control Group Exists 
Koji Yamamoto† and Michio Naoi‡(Issued March 27, 2011)
Abstract
The first wave of Japan Household Panel Survey (JHPS) was designed to letparticipants respond to questionnaires via the Web rather than paper if they
desired. In this study, we investigate whether introducing the Web option
altered the response rate of the survey. We emphasize that eventual Web
respondents would likely have responded even in the absence of this mode. In
general, when gauging the effect of treatments, such as policy-related
treatments, a control group is usually required, regardless of whether thedesign is experimental or quasi-experimental. However, in some cases, acontrol group cannot be assigned into or observed. We show that the effect ofadding an alternative option can be estimated without a control group, undercertain assumptions. We use this approach to estimate increase in unitresponse due to adding the Web mode. The result shows that the options‚Web response‛ and ‚paper response‛ were mutually easily substitutable,
and, only few respondents would not have responded unless Web mode wasavailable. In short, the net rise in unit-response introduced by Web mode wasonly limited. In addition, in this paper, we show the characteristics of those
who had PC and Internet access, which enabled online responses, and those
who actually responded via Web mode.
This English version was translated into by an external translation service supplier fromthe Japanese version issued on March 27, 2011. Note that information included in thispaper is as of March 2011.
The earlier versions of this paper were presented at the Workshop of the Joint ResearchCenter for the Panel Studies at Keio University (November 18, 2009), Methods-andApplications Workshop (November 20, 2009 and October 8, 2010), and Japanese
Association for Mathematical Sociology (March 8, 2011). The authors are grateful toparticipants of these meetings for useful comments. In particular, the authors would like tooffer a warm thanks to Tamaki Miyauchi for carefully guiding the first author, and to
Hiroshi Ishida, Colin R. McKenzie, Satoshi Miwa, Hiroshi Taroumaru, and Takahiro
Tsuchiya.
† Part-time Researcher, Joint Research Center for Panel Studies, Keio University‡ Assistant Professor, Faculty of Marine Technology, Tokyo University of Marine Scienceand Technology
11. Introduction
The type of sample surveys utilizing multiple data collection methods has become
increasingly popular in recent years. In this paper, we evaluate the extent to which thistype of design improves the response rate. In general, in order to measure the effect oftreatments, for example policy-related ones, we usually need a control group, regardless ofwhether the design is experimental or quasi-experimental. In some cases, however, wecannot make assignment for, or observe, a control group. We propose a method to evaluate
the effect of particular type of treatments, even in the absence of a control group.
In this study, we investigate the effect of mixed-mode survey design on participant
response. In general, sample survey can employ a variety of survey methods, such as faceto-face interviewing, visitation and questionnaire retention, postal correspondence, andonline survey. These different methods are called modes, and a survey involving multiple
modes is called a mixed-mode survey. According to Groves et al. (2004: 163-165), mixedmode survey enables participants to select the mode most convenient to them, thereby
increasing the unit-response1. However, some scholars argue that adding Web mode in
mixed-mode surveys has little effect on the response rate (Couper and Miller 2008: 834).
Hence, we question the extent to which mixed-mode survey improves the response rate.
From this perspective, we present a method for evaluating the effect of mixed-mode survey
on response rate. Our analysis uses the data obtained by the method of visitation, retention,and self-administered answering, with Web-mode option.
The quality of a sample survey is not measured only by the response rate
(Yamamoto and Ishida 2010); however, aiming to improve the response rate is extremely
important. The quality of sample survey is improved by minimizing the errors (conditionalon fixed cost). Improving the response rate reduces sampling error by increasing thesample size. Moreover, the improvement of response rate can reduce the non-response
error, especially when the types of those with particularly low response rate are made more

1

In Japan, although it was not a sample survey, in 2010 the National Census introduced
postal mode and, partially, Web mode in addition to the existing method of visitationcollection of paper questionnaires (Ministry of Internal Affairs and Communications 2010).
Presumably, the purpose was to facilitate responses (although because the Census is a
‚population‛ survey, it was not stated that the aim was ‚the improvement of response
rate‛).
2

likely to respond2. Hence, we have good reasons to seek methods that can improve theresponse rate, and, evaluate them.
The most powerful method for evaluating the impact of a survey mode on the
response rate is to randomly assign the targeted sample into different survey modes andcompare the response rates of the modes (for example, Israel 2009)3. However, in mixedmode surveys, the impact of additional modes on the response rate should be assessedagainst a control group without those options. If we set a control group with a smaller
number of response modes, we might be afraid that the response rate of that group may
worsen and substantial information we obtain may be poorer. For this reason, such amethod with assignment for a control group has shortcomings, unless we are willing to
pay a lot of cost to investigate survey methodology itself.
Alternatively, the effect of adding modes in mixed-mode surveys may be evaluatedby requesting non-respondents at the initial round to respond via the particular alternativemode. In this approach, an option is added after a time gap from the initial round and its
effect is evaluated. Generally speaking, this approach is recommended, not merely in orderto evaluate the effect of a mode, but also to improve the performance of the whole survey.4
This type of approach was adopted by McCabe et al. (2006) and Dillman et al. (2009).
2

We should be careful to discuss how we can reduce non-response error. When the
response rate of types of people with particularly poor response rate does not improve,even when the response rate of the entire target sample improves, the non-response error is
not necessarily reduced. Furthermore, adjustment techniques against non-response error
may depend on the source of the non-response. Roughly speaking, we can argue like thefollowings. On one hand, suppose the individual’s probability to respond is correlated only
to the information that can be observed before the survey. This situation is equivalent to asituation referred to as missing at random (MAR) or selection-on-observables in missing
data analysis, and we do have at hand some remedies for the non-response error. On theother hand, the situation in which the probability of response is related to factors thatcannot be observed in advance (thus not observed for non-respondents) is equivalent to astatus known as non-ignorable (NI) or selection-on-unobservables. In this situation we canonly have remedies to adequately analyze data with even stronger assumptions than in theformer situation, making it more difficult to analyze. For particular methods and issues on
non-response in panel surveys, see Naoi (2009) for example. We should be aware that, evenwhen an approach in sample survey improves response rate, if it strains the assumption ofMAR, the approach may not be ideal for the purpose of reducing non-response error.
3 Although not directly related to the evaluation of mixed-mode surveys, Kaplowitz et al.
(2004) assigned target sample to postal and Web surveys, as well as to different methods ofprior notification and follow-up, and compared the response rates among the groups.
4 See Dillman (1999: 240-241).
3

However, this approach also has limitations. Adding the alternative option after a time gap
lengthens the survey period and may reduce the validity of information we might beinterested in. Therefore, surveys aimed at gaining substantial knowledge may not benefit
from this type of approaches5. We also note that, in some sense, even this approach uses acontrol group for comparison, if we regard the target sample before adding the alternative
and the sample after adding it, as a control group and a treatment group, respectively.
In this paper, we present an evaluation method that assesses the effect of asimultaneously presented alternative with no control group. More specifically, we analyze
to reveal the net rise in the response rate when Web mode is added to the paper-based selfadministered mode with visitation and retention. The analysis uses the data from the firstwave of Japan Household Panel Survey conducted in 2009 (hereinafter abbreviated to‚JHPS2009‛). In JHPS2009, the Web-based option was introduced and available toindividuals who desire to use it. Its primary aim was to improve the response rate among
the sections of target individuals with previously poor response rate, and also to reduce thenon-response error (Naoi, Yamamoto, and Miyauchi 2010: 35-36). Below we evaluate
whether this type of setup indeed improves the response rate. Because a group without
Web mode (which can be seen as a control group) was not set in this survey, in order toevaluate the effect of Web mode we construct a novel method. In doing this, we emphasize
the fact that, the individuals who eventually made response via Web might have maderesponse even via other mode, if Web mode had been unavailable.

2. Summary of Survey and Data
JHPS2009 is the first wave of the panel survey. The details of data collection
methods and sample properties in this survey are given by Naoi and Yamamoto (2010) andNaoi, Yamamoto, and Miyauchi (2010). The following analyses use both of the main dataset
and the survey confirmation form dataset. The main dataset are made from the responsesof the target individuals (including the spouses of sampled individuals). This dataset
comprises 4,022 valid cases. The survey confirmation form dataset is made from records of5

Moreover, in general, when an alternative A1 is presented initially and another alternative
A2 is added after a time gap, if the decision makers are from the beginning aware that A2
will be added and they prefer A2, they may not choose A1 initially, willing to choose A2,
but without hating to choose A1. When this occurs, those who chose the additional
alternative A2 may have chosen alternative A1 in the absence of A2.
4

collection agents’ visits to the target individuals’ residences during the survey period. The
collection agents recorded demographic data (such as housing type) of the entire target
sample on the survey form (collection agents’ survey confirmation form)6. This dataset
contains 12,549 cases. Unlike the main dataset, which includes information only of theindividuals who validly made response in the survey, the survey confirmation form dataset
embraces information from the entire targeted sample, including those individuals who
refused to partake in the survey.
JHPS2009 was conducted via visitation and retention method. To be precise,although part of the target sample was interviewed face-to-face, only a fraction of the items
was surveyed in this manner, whereas other items were surveyed using self-administeredpaper questionnaires also for the interviewed group. The same items were surveyed in bothgroups, and individuals to be interviewed were randomly selected. No significant
difference in the response rate was observed between the interviewed and non-interviewed
groups (Naoi, Yamamoto, and Miyauchi 2010: 41).
In JHPS2009, the target sample was able to respond via Web instead of completing
paper questionnaires. Participants freely selected whether to deliver their responses on
paper or online7. Importantly, the Web option was not randomly assigned in the target
sample by the administrator. If some target individuals were provided with Web mode andothers were not, the effect of Web mode on the response rate could be analyzed bycomparing the two groups. Because Web mode was in fact designed to be available to theentire target sample, we require a novel method to estimate its effect on the response rate.
Hereafter, the responses returned via paper questionnaires shall be referred to as ‚paper
response‛, whereas responses via Web shall be called ‚Web response‛, Invalid or null
responses shall be referred to as ‚non-response‛. Furthermore, the cases where thecollection agents had made contact with the target individuals or their family shall becalled ‚contacted‛ 8 . ‚Contacted‛ comprises two subgroups: ‚cooperation‛, when valid
answers were obtained, and ‚non-cooperation‛, when contacted participants failed tocooperate.
6

For more information on this dataset, see also Naoi and Yamamoto (2010: 8).
See Naoi and Yamamoto (2010: 11-12) and Naoi, Yamamoto, and Miyauchi (2010: 35-36).
8 We consider the following cases as contacted: the cases recorded, in the survey
confirmation form, as either ‚(the collection agent had) spoken through the intercom‛,‚met with family members other than the target person or her/his spouse‛, ‚met the
spouse‛, or ‚met with the target person‛. Individuals who refused cooperation in advance(e.g., via telephone) were excluded.
75

Of the 4,022 valid survey responses, 91 were returned online. The 4,022 participants
were asked whether or not they use a PC at home, and whether they have home internet
access. This information together with Web response status is summarized in Table 1. Table
1 is equivalent to Table 1–6 from Naoi and Yamamoto (2010: 12). Not surprisingly, mostWeb respondents use a PC at home and have home internet access.
[Table 1]
The analysis in Section 3 utilizes the abovementioned information regarding PC use
and home internet access. As such, it uses the main dataset including 4,022 valid cases. On
the other hand, the analysis in Section 6 requires explanatory variables even for the noncooperation cases. In that section, the survey confirmation form dataset with 12,549 cases(including the entire target sample) is utilized. Also, we assume that, when a target
individual was not contacted, she/he would never have an alternative to response via Web
mode. Hence, the analysis in Section 6 was conducted on the 9,621 contacted individuals.

3. Characteristics of Web Respondents: PC Use and Internet Access
The effect of Web-mode addition and comparisons between the Web and paper
respondents will be investigated in Section 6. Before them, in this section we put focus on
the characteristics of Web respondents in JHPS2009, which have been studied by Naoi,
Yamamoto, and Miyauchi (2010: 54-59). Naoi et al. compared the distribution of eachvariable between Web and paper respondents, and obtained the following results. Web
respondents are more likely to be male, young, single, and university graduates. Also, thisgroup comprised a larger proportion of those who are ‚mainly working‛. Moreover, Web
respondents were more likely to be working full-time, largely in the clerical andinformation processing occupations. The income of Web respondents was relatively high.
Based on these results of Naoi et al. (2010), our questions here are twofold. First,
although they showed the traits of the Web respondents by presenting distribution of eachsingle variable, does a relationship exist between each variable and Web response when thevariable used are mutually controlled? Second, when respondents with particularattributes tend to respond online, is it because they only tend to ‚use a home PC and havehome internet access‛, or, are they likely to be online respondents even among those who

6use a home PC and have home internet access?
In considering these questions, two types of probit models were estimated using
the main dataset of JHPS2009 9. The first model explains the case of ‚PC use and homeinternet access available‛. In this model, the dependent variable is, 1 when home PC is usedand home internet is accessible, and 0 otherwise. The explanatory variables used arevariables of respondents’ attributions. The second model explains ‚Web response‛. Thismodel limits the cases to those where a home PC is used and home internet access areavailable. The dependent variable is 1 when a Web response is made and 0 when response
is provided via paper questionnaire. Also in this model, as in the former, the respondents’attributions are used as the explanatory variable. For both of two models, the explanatory
variables are the respondent’s sex, age, marital status, education level, occupation, and9

Two additional points should be made to the estimates provided in this section. First,
selection bias may exist in the estimated results because the ‚Web response‛ model
analyzes only the cases for which ‚PC use and internet access available‛ is true. Now
suppose that, even if the respondents without ‚PC use and internet access‛ would have
‚PC use and internet access‛, other conditions being equal, then such respondents areunlikely to make Web response. In this case, the estimates of our model do not
appropriately capture the Web response trend. However, there is no crucial need toconsider the Web response trend of the individuals not in the state ‚PC use/internet accessavailable‛. This is because we do not intend, for example, to measure the effect of policy
where we provide PC and internet access with people and teach them to use PC andinternet. In addition, it is extremely unlikely that the target individuals would ‚buy a PC
and setup an internet access just in order to be able to respond via Web in the survey‛.
Thus, the method here is appropriate to seek the type of individuals, among those with
‚PC use and internet access available‛, who are likely respond via Web. Note that thenature of the data here is different from that used in, for example, the analyses on factors
determining wages, where ‚potential wages affect employment but only the wages of theemployed can be observed‛. The second point relates to the analysis method of Section 6.
In that section, we estimate nested logit model (details for which we will explain in later
sections). Even in the analysis of this section, the choice between ‚Web response‛ and‚paper response‛ depends largely on the state ‚PC use and internet access available‛.
Thus, the choice structure is nested, and some might think nested logit model is anappropriate model such a structure. It is true that we can analyze the choice structure hereby nested logit model. However, we have reasons not to do that. Nested logit model allows
substitution patterns to be flexibly modelled by parameters, but such a flexible substitution
patterns may not be applicable in the present analysis. Flexible substitution patterns mean
that, for example, the tendency to be without ‚PC use and internet access‛ could possibly
substitute with the tendency to make Web response. If none of the respondents ‚buys a PC
and sets up internet just in order to respond via the Web in the survey‛, substitution
between absence of ‚PC use and internet access present‛ with ‚Web response‛ has no basis
in reality.
7

household income. By estimating these probit models, we are to answer to theaforementioned two questions.
The results are shown in Table 2. First, we discuss the result for the first model,
explaining the case of ‚PC use and home internet access available‛. At 5% significancelevel, the probability of ‚PC use and home internet access available‛ is significantly high inmales, young individuals, individuals with spouses, highly educated individuals,
individuals in clerical and information processing occupations (compared with the
unemployed), and those with high household income. Furthermore, compared with the
unemployed respondents, the tendency of home PC and internet availability is significantly
lower among workers in the agricultural, forestry, and fisheries industries. These effects
exist even when other factors are controlled.
[Table 2]
The second model, explaining ‚Web response‛, yields apparently different results.
Here, at 5% level, the attributes of sex, age, educational level, and household income do nothave significant impact on Web response (in comparison with base categories for dummy
variables)10.
In other words, the factors explaining the tendency of ‚Web response‛ differ
markedly from those explaining the tendency of ‚PC use and internet access available‛.
With regard to sex, the probabilities of both Web response and ‚PC use and internet accessavailable‛ are high in males. However, with regard to age, we cannot clearly say theprobability of Web response is higher among the younger respondents and low among the10

We note that the sample size in this second model differs from that in the previous
model. Thus, even for variables exerting the same effect, if the same significance level isused, the null hypothesis that no effect exists in the ‚Web response‛ model is harder to be
rejected for a smaller sample size. For this reason, in order to better compare the results of
two models, we also calculated the standard error and p value of each variable under the
hypothetical condition that, for the subsample in the second model, the sample size is3,349, and, the information it has is of the same nature as that in reality. More specifically,in this hypothetical scenario, the standard error is set as the square root of the square of theactually calculated standard error multiplied by 1,993/3,349. We omit the details of theresults, but in this case, at 5% level, the dummy variables such as female, ‚30–39 year old‛,
spouse present, and ‚Other Occupation‛ become significant. Even in this supplemental
analysis, the factors explaining the tendency of ‚Web response‛ differ from those
explaining the tendency of ‚PC use and internet access available‛ in multiple ways.
8

older respondents. Similarly, education level showed no significant effect on Web response,
nor is Web response significantly more likely among clerical and information processing
workers than among the unemployed. Furthermore, with regard to marital status, theprobability of ‚PC use and internet access available‛ is high when the individual ismarried, whereas that of ‚Web response‛ is higher among those unmarried11.
These results imply that, while Naoi, Yamamoto, and Miyauchi (2010: 54-59)
identified some characteristics with which individuals are more likely to make Web
response, namely ‚young‛, ‚single‛, ‚university graduate‛, ‚work in clerical orinformation processing‛, and ‚with higher income‛, these can explain who have home PC
and internet access, but we cannot say these characteristics encourage Web response among
respondents with access to home PC and the internet. In other words, these characteristics
stipulate the environment in which Web response is possible, and do not necessarily
explain inclination for Web response given such an environment.
In this section, we have examined what type of target individuals are likely to
become Web respondents and why. We next consider whether, as pointed out by Naoi,
Yamamoto, and Miyauchi (2010: 59), the Web respondents would have responded in theabsence of Web mode; i.e., if no Web mode was available, would the Web respondents have
responded via paper questionnaire? This issue is discussed below.

11

The results relating to marital status are worthy of further mention. The high probabilityof ‚Web response‛ among singles contrasts with the high probability of ‚PC use andinternet access available‛ among married individuals. About this, we can make thefollowing interpretation. Home PC and internet resources may be shared in a household.
Therefore, even if a married individual possesses no strong desire to use a PC or internet
access, they may be encouraged to use these resources if their spouse uses them. In
contrast, among individuals with ‚PC use and internet access present‛, Web responses may
be harder for those with spouses. JHPS2009 was designed such that, when a spouse waspresent, both the target individual and his/her spouse completed the retainedquestionnaire. When the target individual used the paper survey form, even if theindividual and his/her spouse were to complete it at different times, the paper survey form
was passed on and completion requested. However, if Web response was selected, and thetarget individual and his/her spouse wished to respond at different times, both individuals
were required to login to the response website. In this case, for example, if the target
individual easily completed the login process while the spouse experienced difficulty, Web
response completion might have been precluded. That is, when a spouse is present, unlessthe login process cost is low for both the target individual and the spouse, Web response isseen as difficult. This may explain why the probability of Web response is smaller than thatfor singles.
94. Illustrative Description of the Method
As mentioned above, in JHPS2009, the availability of Web mode was not assigned
to only a part of target individuals, but given to all; hence, to estimate the effect of Web
mode on the response rate requires a novel analysis method. In this section, we present a
method for evaluating the effect on the response rate of mixed-mode surveys with nocontrol group. In general, this method evaluates the effect of a type of treatments, where
we add alternative(s), with no control group.
Of the 4,022 valid respondents to the JHPS, 91 responded via Web. Therefore, thenumber of additional valid responses gained by adding Web mode is at most 91. We shouldnote that, part of these 91 respondents may have responded to the survey via paper, if Web
response option was not available. Therefore, presumably, the net increase in number of
respondents ‚generated‛ by Web mode is smaller than 91. To measure the increase in
number of respondents generated purely by Web mode, we must determine whether the
online responders would have responded in the absence of Web mode.
Even if Web mode could ‚generate‛ valid responses, it could not do that unless thetarget individual was notified that Web mode is available. Hence, the ‚non-contact‛ cases,in which no contact was made during the survey period, are excluded from the discussion
and analysis.
For the purpose here, we need to identify, which alternative (paper response or
non-cooperation) would be chosen by each individual, who in reality responded via Web,
under the counterfactual situation that Web mode did not exist12. In other words, we must
analyze the substitution pattern of the alternatives. To this end, we should adopt a multiple

12

Here all ‚Web response‛, ‚paper response‛, and ‚non-cooperation‛ are treated as‚alternatives to choose‛. Strictly speaking, ‚non-cooperation‛ includes cases in which theagents did not personally contact the target individual. Therefore, we cannot assume that
all the target individuals consciously ‚chose the alternative of non-cooperation‛; thus, theterm like these may not be appropriate. To avoid misunderstanding, the terms‚alternative‛ or ‚choice‛ might be better replaced with vocabulary such as ‚target
individuals are ‘classified’ into one of the ‘states’: ‘web response’, ‘paper response’, and‘non-cooperation’‛. In this paper, however, we retain the terms ‚alternative‛ and ‚choice‛for simplicity and consistency with existing literature.
10

choice model in which substitution patterns can be flexibly expressed. We note that, thetype of multinominal logit model and conditional logit model, often used as multiple
choice models, expresses only specific substitution patterns. Specifically, the substitution
patterns in these models are fixed by the independence from irrelevant alternatives (IIA)
assumption. This section introduces nested logit model in which substitution patterns are
flexibly expressed. We will estimate the model, explaining the choice among the threealternative modes, ‚paper response‛, ‚Web response‛, and ‚non-cooperation‛, and next,
we will evaluate the number of valid responses ‚generated‛ by Web mode, based on thepredictions using the estimates.
Below we explain the concept of the substitution pattern used to analyze ‚Web
response‛ with a pattern diagram.
First, we assume a state in which ‚Web response‛ and ‚non-cooperation‛ residewithin the same nest13, and the two alternatives can be substituted extremely easily. Figure
1 shows the correspondence between the tendency for an individual to conduct a ‚Web
response‛ 14 , indicated by X-axis, and the probabilities of ‚paper response‛, ‚web
response‛, and ‚non-cooperation‛, indicated by Y-axis. The tendencies for choosing ‚paper
response‛ and ‚non-cooperation‛ are assumed to be fixed. In the case of Figure 1, theprobability of ‚non-cooperation‛ is small among the target individuals with strong ‚Web
response‛ tendency compared with those with weak tendency. On the other hand, theprobability of conducting ‚paper response‛ is similar in both groups. Thus, Figure 1graphically represents the pattern where substitution occurs very easily between ‚Web
response‛ and ‚non-cooperation‛. The state of no Web mode can be regarded as a state inwhich the Web response tendency is extremely small (equal to negative infinity) among theentire target sample15. Therefore, given the substitution pattern in Figure 1 and provided
that the target sample contains members with strong tendencies to conduct ‚Web
response‛, the existence of the ‚Web response‛ alternative reduces the probability of ‚noncooperation‛, i.e., improves the response rate.
[Figure 1]

The nest structure in this case corresponds to ‚nest structure (a)‛ shown in Figure 4.
Strictly, the tendency to make ‚Web response‛ discussed here refers only to the part
expressible by observed variables.
15 The condition for this proposition is given in Formula *18+.
13
14

11

We now assume a different nest structure in which ‚Web response‛ and ‚paper
response‛ reside in the same nest16, and thus are easily substituted. The choice probabilitiesof this situation are shown in Figure 2. In this case, for the individuals strongly inclined torespond online, the probability of ‚non-cooperation‛ is similar to those less inclined,
whereas the probability of ‚paper response‛ is small. This expresses the ease with which‚Web response‛ and ‚paper response‛ are substituted. In the case of Figure 2, even if target
individuals with strong ‚Web response‛ tendency exist in the population, the alternative‚Web response‛ merely discourages the selection of ‚paper response‛ and does not signify
an improvement in the response rate.
[Figure 2]
Figure 2 shows a state in which ‚Web response‛ and ‚paper response‛ can be
extremely easily interchanged, whereas the substitution between ‚Web response‛ and‚non-cooperation‛ almost never occurs. In reality, even if ‚Web response‛ and ‚paper
response‛ reside in the same nest, the substitution pattern might be less extreme than thatof Figure 2. Nested logit models can express substitution patterns flexibly by usingparameters, as an example shown in Figure 3. In the case of this figure, ‚Web response‛and ‚paper response‛ remain nested together, but non-negligible substitutions occur
between ‚Web response‛ and ‚non-cooperation‛.
[Figure 3]
According to the substitution pattern of Figure 3, target individuals with strong
tendency for ‚Web response‛ have low probability of ‚paper response‛ compared withthose with weak tendency. In addition, the high-tendency group has a low probability of‚non-cooperation‛. Consequently, when Web mode is added in this scenario, ‚Web
response‛ replaces ‚paper response‛ and also ‚non-cooperation‛ to a certain extent. The
extents to which ‚Web response‛ replaces ‚paper response‛ and ‚non-cooperation‛ areexpressed by a parameter estimated in the nested logit model. Once these estimated results
specify the substitution pattern, we can evaluate the contribution of Web mode to the
response rate, by calculating the level of ‚Web response‛ replacing with ‚noncooperation‛.
A more specific description of our method is as follows. As described above, under16

The nest structure in this case corresponds to ‚nest structure (b)‛ shown in Figure 4.
12certain assumptions, the state of no Web mode is equivalent to that of no tendency (ortendency indicated by the value ‚negative-infinite‛) for Web response among the whole
population. Roughly speaking, when the tendency for ‚Web response‛ is extremely weak(corresponding to left-hand side endpoint in Figure 3), the probability of ‚Web response‛ isextremely small, and this state reflects a virtual situation where no Web mode exists. On
the other hand, when the tendency for ‚Web response‛ is strong (corresponding to righthand side endpoint in Figure 3), a significant probability of ‚Web response‛ exists. We caninterpret, comparing with the left side of the figure, at the right side the ‚Web‛ probability
comprises two parts, which replaced ‚paper response‛ and ‚non-cooperation‛, shown as γand δ, respectively. Thus, in this pattern, δ reflects the extent to which the probability tomake valid response is increased by letting the Web option available.
In the following analysis, first, the nested logit model is used to determine thechoice structure (substitution pattern) and the effect of individual attributes on selection
tendencies. Next, we simulate two states: Web mode available and Web mode absent. We
determine each individual’s probability of ‚non-cooperation‛ in each state, and obtain the
expected value of the increase in the number of respondents generated by Web mode, bysumming up the δ values across all target individuals. Thus the net effect of the optional
Web mode is measured.

5. The Model
We adopt a nested logit model with two levels. In this section, we introduce themodel formulation based on random utility maximization (RUM), and, demonstrate thatthe estimated probabilities in the case of no ‚Web response‛ can be calculated consistently
with RUM. We owe a considerable portion of the following discussion to Heiss (2002) andTrain (2003).
First, we present a general two-level nested logit model as follow. Consider asituation in which each individual i selects one option from J alternatives. The index for thealternatives is j (or k), and the alternative selected by individual i is yi. Each alternative
belongs to one of the M nests, expressed as Bm (m = 1, <, M). The nest to which alternative jbelongs is denoted by B(j). Consistent with preceding research, the tendency for theindividual i to select the alternative j is called ‚utility,‛ denoted by Uij.

13

The utility Uij when individual i selects the option j can be separated into acomponent Vij on the basis of the observed variables and an unobserved component(disturbance term) εij. Typically, Vij is expressed as a linear combination of observed
variables and parameters.

U ij  Vij  εij*1+

The disturbance term εij is a random variable that distributes a standard Type I
Extreme Value distribution whose variance is π2/6. εij is independent among individual
decision makers. Among the alternatives, εij is independent between the alternatives indifferent nests, but is correlated between the alternatives within the same nest. The
correlation coefficient between the εij’s in nest Bm is denoted by ρBm.

 ij ~ S tandard TypeI Extreme Value ,*2+and this means Pr(εij  x)  exp[ exp(x)]*3+

Corr(εij , εik )  ρB( j ) if B(j)  B(k) and j  k,*4+

εij  εik if B(j)  B(k).

For a nest Bm, an inclusive value coefficient (or dissimilarity parameter) τBm isdefined as follows.

τBm  1  ρBm*5+.

Furthermore, for individual i, the inclusive value IiBm of nest Bm is defined asfollows:*6+

I iBm  ln[

 exp(V / τ )] .
ik

kBm

Bm

On the other hand, RUM assumes the following:

14

*7+

Pr(y i  j)  Pr(U ij  U ik k  j).

Under the above assumptions, the probability that individual i selects alternative j
is expressed as below:*8+

Pr(yi  j)  Pr[yi  B( j)] Pr[yi  j | yi  B( j)]*9+exp[τ B( j ) I iB( j ) ]Pr[y i  B( j)]  M exp[τ I
m1

*10+

Pr[y i  j | y i  B( j)] 

Bm iBm ]exp[Vij / τ B( j ) ] exp[V / τik

B( k ) ].

kB( j )We next consider the case in which specific alternatives are excluded from themodel. Assuming J > 2, when the number of alternatives reduces by one (i.e., the number ofalternatives is J − 1), with other assumptions unchanged, the choice probability of eachalternative derived consistently with RUM takes a general form shown in Formulae *6+, *8+,
*9+, and *10+.
In a lead-up to the next section, we specifically show the formulation of thesituation where ‚Web response‛ option does not exist. As will become evident later, ‚Web
response‛ and ‚paper response‛ are estimated to be included in the same nest, so we
assume such a structure also here. We use the notations as follow: web, paper, nocoop, Bcoop,
and Bnocoop, which indicate, ‚Web response‛, ‚paper response‛, ‚non-cooperation‛, the nest
comprising ‚Web response‛ and ‚paper response,‛ and the nest comprising ‚noncooperation‛ alone, respectively. Then the utility from selecting each option is as follows:*11+

U i ,web  Vi ,web  εi ,web , U i ,paper  Vi ,paper  εi ,paper , U i ,nocoop  Vi ,nocoop  εi ,nocoop

We note that, according to RUM, the selection probability depends only on theutility differences between selection modes. Because of it, and also because we only use
individual-specific variables in estimation, Vi,nocoop is constrained to be 0. Here εi,web, εi,paper,and εi,nocoop each distributes the standard Type I Extreme Value, but because εi,web and εi,paper

15belong to the same nest Bcoop, they can be correlated with correlation coefficient ρBcoop. On theother hand, no correlation exists, between εi,web and εi,nocoop, nor, between εi,paper and εi,nocoop.
Assuming RUM, the selection probability of ‚Web response‛ is derived as follows. Here
τBnocoop cannot be uniquely determined, thus we arbitrarily fixed it as 1.

*12+

τBcoop  1  ρBcoop*13+

I iBcoop  ln[exp(Vi ,web / τBcoop )  exp(Vi , paper / τBcoop )]*14+

Pr(yi  web)  Pr[yi  Bcoop ]  Pr[yi  web| yi  Bcoop ]*15+Pr[yi  Bcoop ] *16+

Pr[yi  web| yi  Bcoop ] exp[τBcoop I iBcoop ]exp[τBcoop I iBcoop ]  exp[τBnocoop I iBnocoop ]

exp[τBcoop I iBcoop ]exp[τBcoop I iBcoop ]  1

exp[Vi ,web / τBcoop ]exp[Vi ,web / τBcoop ]  exp[Vi , paper / τBcoop ]In this model, when Vij is provided for each alternative, the selection probability
can be estimated for each individual assuming a virtual state with no ‚Web response.‛ In
the absence of ‚Web response‛, the individual would make a selection from choice setscomprising two modes, ‚paper response‛ and ‚non-cooperation‛, but each of the
disturbance terms for these modes (εi,paper and εi,nocoop) has an independent Type I Extreme
Value distribution. Hence, their difference distributes with the standard logistic
distribution, and the selection probability between ‚paper response‛ and ‚noncooperation‛ can be denoted by the binomial logit model. This conclusion is also derived
from Formulae *6+, *8+, *9+, and *10+. More specifically, under the constraint Vi,nocoop ≡ 0,Pr[y i  pa per| Pr(y i  web)  0]  Pr(U i ,paper  U i ,nocoop ) Pr(V i ,paper  εi ,paper  V i ,nocoop  εi ,nocoop ) Pr(V i ,paper  V i ,nocoop  εi ,nocoop  εi ,paper )*17+

 Pr(V i ,paper  κi )
 F(V i ,paper )

exp(V i ,paper )1  exp(V i ,paper ).

16

Here κi is the random variable following the standard logistic distribution, and F(·) is the
cumulative density function of the standard logistic distribution.
Even if the ‚Web response‛ alternative pre-existed, provided that the following
Formula *18+ is satisfied, as Vi,web becomes small enough to be negative infinity, theprobability that individual i selects ‚Web response‛ tends to 0. In this situation, thisindividual must select between ‚paper response‛ and ‚non-cooperation‛. This is theassumption of the formerly discussed proposition in terms of Figures 1–3, i.e., when thetendency for Web response is very small, the selection reduces to ‚paper response‛ or‚non-cooperation‛.
*18+

Pr(  εi ,web  r)  1 for any real number r.

We now specify Vij, i.e., the component of the utility, explained by the utility
explanatory variable. Ideally, if we aim to estimate a multiple choice model directly
consistent with a theory, we should use explanatory variables which represent what eachalternative means to the individual. In other words, the explanatory variables should bealternative specific. In the current analysis, for example, we could use the alternativespecific variable such as the perceived cost of each alternative for each target individual. In
this way, the general theory such as ‚the sense of cost sways the selection behavior‛ could
be more directly modelled and tested. If we had the variable such as ‚the sense of cost
relating to the input operation on the Web and PC operation‛, then we could use it as thealternative-specific variable for ‚Web response‛ consistently with the ‚cost‛ theory above.
However, because we have no such a variable in the dataset, the present analysis uses only
individual-specific variables.
To be more specific, letting the variable vector indicating the attributes ofindividual i represented by xi, the parameter vector applied to each variable when
explaining the utility of alternative j be represented by βj and the individual-specificconstant of alternative j be represented by αj, Vij can be expressed as:*19+

 x i , Vi ,nocoop  0
Vi ,paper  α paper  βpaper x i , Vi ,web  αweb  βweb

In this model formulation, the estimated parameters express the differences inutility of the selection behavior, caused by the individual’s attributes. This utility difference17

may arise from differences in, e.g., the value of payments received for survey cooperation,
the psychological sense of satisfaction gained by survey cooperation, or the sense of cost
incurred by survey cooperation. Because the survey method influences the sense of cost forsurvey cooperation, we use variables relating to the survey process in the following
estimates.

To simplify the calculations, the parameters in the next section are not directly
estimated from the aforementioned model but from the model formulated in Greene (2003:
725-727). Here because we do not use generic variables, i.e., variables whose coefficients arecommon across the nests, the model itself is still consistent with RUM (Heiss 2002). The~
estimated parameter vector and alternative-specific constant, denoted by β j and ~j ,respectively, are related to the aforementioned parameters as follows:

*20+

~
βj 

1
τ B( j )~ βj , αj1
τ B( j )αj~ˆ~ , which are
The coefficients we will show in the result table are not β j and α̂jestimated parameters themselves, but β̂ j and α̂ j , which are obtained from transformation
to let RUM-consistent interpretations, determined as follows:

*21+

~ˆ
~ˆ
βˆ j  τB( j ) β j , αˆ j  τB( j ) αjIn addition, hypothesis testing is conducted on β̂ j and α̂ j .
Having estimated the nested logit model as described above, we evaluate thenumber of additional responses generated by the Web mode, using the estimates. The
increase in respondent number due to the Web mode is the difference between the number
choosing ‚non-cooperation‛ in the absence and presence of Web mode. We can calculate
the expected value of it using the estimates of the model. Denoting the expected increase in
respondents generated by the Web mode as D, and the expected number of ‚noncooperation‛ responses in the absence and presence of Web mode by NC(T = 0) and NC(T =1), respectively, we obtain the following:*22+

D  NC(T  0)  NC(T  1) .

18

We intend (T = 1) shall indicate the state of Web mode availability. Here the expected
number of choosers of each alternative is the sum of the selection probabilities of eachindividual. Therefore, if the number of individuals is n, the following is obtained from
Formulae *17+, *19+, and *20+:

NC(T  0) n

 Pr(Ui , paper  U i ,nocoop )

i 1

*23+



n

 1  exp(V
1

i 1



n

 1  exp(τi 1i , paper )1~~
Bcoop α paper  τ Bcoop β paper x i ),whereas Formulae *13+, *15+, *19+, and *20+ yield

NC(T  1) n

 Pr(y  nocoop)
i

i 1



n

 [1  Pr(y  Bicoop )]

i 1

*24+


n

 exp(τi 1nexp(τ Bnocoop I iBnocoop )Bcoop I iBcoop )  exp(τ Bnocoop I iBnocoop ) 1  exp[τ
i 1

1
~
~
~
~
 x i )}]
Bcoop ln{exp( α paper  β paper x i )  exp(α web  β web.

D can be expressed as the sum of δ values (see Figure 3):

*25+

D

n

δ ,
i

i 1

where δi  Pr(U i ,paper U i ,nocoop )  Pr(yi  nocoop)

D can be obtained from Formulae *22+, *23+, and *24+ by using the estimatedparameters of nested logit model. Because it is based on the estimates of a statistical model,
it contains an error, whose magnitude is assessed using the delta method. The calculation19

of this error is discussed in Appendix.

6. Results(1) Estimated Results of the Statistical ModelIn this subsection, we apply the model to the survey confirmation form dataset, andreport estimated results17.
Prior to estimating the nested logit model, the nest structure must be specified.
Here we do not determine the nest structure a priori, and we let data determine whether‚paper response‛ or ‚non-cooperation‛ is more easily substituted by ‚Web response‛. The
nest structures formed from the three alternatives, ‚paper response‛, ‚Web response‛, and‚non-cooperation‛, are shown in Figure 4. Nest structure (c) is not an essential target of thisresearch because it assumes that ‚Web response‛ and ‚paper response‛ are substituted tothe same extent as ‚Web response‛ and ‚non-cooperation,‛ and thus, the effect of Webmode addition cannot be evaluated with this structure. Hereafter, this nest structure shallbe examined just for supplemental information.
[Figure 4]
Table 3 shows the inclusive value coefficients τ, resulting from the estimation of thenested logit model, with nest structures constructed in Figure 4. All explanatory variables
listed in Table 4 are included as individual-specific variables.
[Table 3]

17

The analysis in the previous section implies that the presence or absence of PC use andinternet access strongly stipulates the Web responses. Therefore, if none of the target
individuals would purchase PCs and setup internet access just in order to enable online
response, we can say that, among the target individuals with no PC use or internet access,‚Web response‛ is never likely to replace ‚non-cooperation‛. In this sense, the analysis in
this section might be more appropriate using the dataset with only those who use PCs andhave internet access. However, we cannot obtain information on PC and internet access of
non-respondents. Consequently, in the following analysis, the sample cannot be limited by
the presence of PC use and internet access.
20

The inclusive value coefficient represents the degree to which the alternatives
collated in a nest are dissimilar. The coefficients below 0 or greater than 1 are incompatible
with RUM18. In fact, Formula *5+ indicates that, when the correlation coefficient between
two disturbance terms lies in *0,1+, the inclusive value coefficient also lies in *0,1+. From
Table 3, we observe that the estimated inclusive value coefficient exceeds 1 in nest structure(a), suggesting that dissimilar alternatives are collated within a nest. In contrast, nest
structure (b) yields an inclusive value coefficient estimate within *0,1+. Therefore, nest
structure (b) is adopted throughout the remaining analysis, though we should note that the
95% confidence interval around the inclusive value coefficient includes the region below 0.
We can think it natural that the target individuals decide to respond online after deciding
to cooperate in the survey. If it is true, nested structure (b) is a valid construct.
Nest structure (b) suggests that the alternatives ‚paper response‛ and ‚Web
response‛ are similar and easily interchanged. As shown in the two rightmost columns ofTable 3, the null hypothesis that the inclusive value coefficient is 1 in nest structure (b) canbe rejected at the 0.1% level by Wald test. This means that, in our analysis, the substitution
pattern is not the one implied by IIA, i.e., the one assumed in the multinominal andconditional logit models. This result also confirms the appropriateness of our use of nested
logit model.
If the aim is not to evaluate the effect of Web-mode addition, nest structure (c) can
also be possibly adopted, because the estimated inclusive value coefficients fall within *0,1+.
However, we should note that, as shown by the 95% confidence intervals in Table 3 andWald test results, the null hypothesis that the inclusive value coefficient of nest structure (c)is 1 cannot be rejected.
The full estimated results based on nest structure (b) are shown in Table 4. The
explanatory variables are sex, age, housing status, inhabited area block, the scale of city,and method of survey. The survey method variables are fourfold, as follow: (i) whether thetarget individual partakes only in the retained survey or partakes also in the interview
survey; (ii) whether the target individual was a part of the initial target group or backup;and (iii) the completion reward for the collection agent surveying the target individual(whether a premium was earned by completing the initial target group), and the interaction

18

More strictly, Formulae *6+, *10+ etc. suggest that, even when the inclusive value
coefficient is 0, the probability cannot be calculated correctly.
21

between (ii) and (iii) 19 . Because this analysis mainly aims to acquire the estimates forpredictions, a substantive interpretation of Table 4 is not conducted.
[Table 4](2) Evaluating the Effect of Web-Mode Addition
In this subsection, the effect of Web-mode addition is evaluated from the estimatedresults of the above statistical model.
The predicted numbers of selectors for each alternative, based on the estimates
listed in Table 4, are shown in Table 5. The leftmost column of Table 5 lists the actual
number of selectors of each alternative and their percentages in the real dataset. Similarly
to the previous subsection, the result is based on the dataset including only those were
contacted in the survey; i.e., among the total targeted sample 2,928 individuals who could
not be contacted are excluded from this table.
[Table 5]
The column in the middle of Table 5 displays the expected number of individuals
selecting each alternative, calculated by summing the estimated choice probabilities foreach alternative for all target individuals in the dataset, assuming that Web mode exists.
The expected number of individuals selecting ‚Web response‛ is 91.9, close to the actual
number of Web selectors. The expected number of ‚non-cooperation‛ individuals is 5,598.8,
and it is equivalent to NC(T = 1).
The expected numbers of selectors in the absence of Web mode are shown in thesecond-right column of Table 5. Naturally, this state contains no Web respondents. The
expected number of ‚non-cooperation‛ individuals is 5,602.6, and it is equivalent to NC(T =0). The rightmost column of Table 5 displays the difference in the expected number of
selectors when Web mode is present and absent. The expected difference in the number of‚non-cooperation‛ selectors, D, is 3.8.
This result indicates that 88.1 of the 91.9 Web respondents with internet accesswould have selected paper response if Web response was not available, whereas the19

For details of the survey method, see Naoi, Yamamoto, and Miyauchi (2010).
22remaining 3.8 would not cooperate; i.e., among the online responders, approximately 3.8
would not have responded unless the Web option was available. Therefore, the number of
responses gained by adding Web alternative was approximately 3.8.
In Table 6, we show the estimated errors, for the D estimated above, and also, for
the increases in response rate and cooperation rate calculated based on it. The increase in
respondents (cooperators) generated by Web mode is estimated as 3.8, and the standard
error is 10.2. If a 95% confidence interval is setup around 3.8, the upper limit is 23.9. Even if
this confidence interval indicates an acceptable range of the standard error, it is implausiblethat we expect adding Web mode generated more than 24 responses. This 95% confidence
interval also covers negative values, suggesting that negative inclusive value coefficientsare permitted by the error margins. However, as mentioned earlier, negative inclusive
value coefficients are inconsistent with the model assumptions. Hence, this interpretationof confidence intervals requires careful consideration20.
[Table 6]
In assessing performances of surveys with regard to responses, response rate andcooperation rate are frequently used21. So, we also calculated the percentage increases in
the cooperation and response rates generated by Web mode, shown in the second and thirdrows of Table 6, respectively. The cooperation rate rises by 0.0397% points, whereas the
response rate increases by 0.0304% points. The 95% confidence interval for the increase in
the response rate is *−0.1294, 0.1902+. In other words, even if the 95% confidence interval
constitutes a usual error range, even considering errors, increase in the response rate due to
Web mode is not likely to exceed 0.1902% points. Note that, to show increases in response
rates and cooperation rates, we are not using a proportion whose upper limit is 1, butusing % points whose upper limit is 100.

20

To avoid such inconsistencies, one can think of a method whereby, after specifying thenested structure, τ is not estimated directly but equals (for example) 1/*1 + exp(−g)+. The
formula is rearranged to obtain g. Although, we might think negative inclusive value
coefficients reflect the reality, in the following sense. Negative inclusive value coefficients
may suggest that adding Web mode reduces the response rate. Israel (2009) found that, inthe case of a postal survey method, adding Web mode caused lower response rate. Like
Israel’s case, possibility of negative inclusive value coefficients in our analysis might imply,
even in the visitation survey method, Web-mode addition could reduce response rate.
21 For indices on performance of surveys on responses, see Yamamoto and Ishida (2010).
23

7. Conclusion
In this study, we showed, taking Web-mode addition for example, how we canevaluate the effect of additional options without a control group. The data from JHPS2009
were used in the analysis. The results are summarized below.
First, given the choice of answering via paper or online, the Web responders tended
to be young, university graduates, employed in the clerical and information processing
sectors, and in household with high incomes. However, these characteristics are largely
correlated with the availability of PC use and internet access.
Second, we evaluated the effect on the response rate when Web mode is added to
the retained paper-based survey. By adopting a multiple-choice model in which
substitution patterns are expressed flexibly, the increase in the response rate was shown to
be no more than 0.0304% points. The analyses assumed that individuals’ choice behavior
can be expressed by explanatory variables consistent with RUM, and that the disturbances
in utilities follow specified distributions.
The contribution we directly made is stated as follow. Ideally, sampling surveys
should produce minimum error at a fixed cost, and, improvements in the response rates
can contribute to minimizing errors, under certain assumptions. The effect of survey
methods on the response rate is best evaluated by randomly assigning the target sample todifferent survey methods. However, this design may incur large cost and high risks. If theentire target sample is exposed to the same survey method, the effect of a specific method
can be evaluated by a novel technique in analysis. Specifically, via flexible substitution
patterns, the effect of mixed-mode survey methods on the response rate can be assessed.
Such a technique is what we showed in this paper.
Finally, we discuss future research topics. In the present mixed-mode survey withpaper and Web options, Web mode was regarded as the additional mode. With equalvalidity, one could analyze the contribution made by paper responses to the response rate.
Thus, one could analyze the extent to which the response rate is reduced, assuming that theentire target sample has access to the Web and that paper response mode is absent. Suchinformation would interest survey organizers who aspire to minimize the error at a fixed
cost, provided that Web survey costs are low. Moreover, this method can assess not only

24

the effect of Web-mode addition on the obtained sample’s characteristics such as sample
bias, but also (in theory at least) its impact on sample characteristics in the virtual case oflowered Web-mode response cost and increased Web response rate22.
Furthermore, the proposed method can be used to verify the effects of othercombinations of mixed-mode surveys. For example, the method should be applicable toscenarios in which Web mode competes with postal method or visitation method competes
with postal method. The emerging substitution patterns would reveal how the alternative
options sway the response rates. Through thorough investigation of survey methods, theerrors in sampling survey can be minimized, and as a corollary to this, it is hoped thatfurther meaningful and accurate scientific knowledge can be accumulated.
In general, the proposed method can evaluate alternative-addition treatments in theabsence of a control group. For example, if a public secondary education school of new
type was established in the vicinity of a private high school, the method could evaluate theeffect of the new school on the private high school applicants, without bias caused by timetrends. Another example relates to marriages among the highly educated women. By
considering a virtual society in which highly educated women are extremely refused
marriage, one could determine the extent to which the proportion of married people isincreased by existence of marriages of women with men of lower education. Furthermore,
people would be interested in the degree to which ‚regular‛ work and unemployment aresubstituted by ‚non-regular‛ work 23 . Though the flexible expression of substitution
patterns in multiple-choice models is a classic tool, it is a very useful tool that yields
knowledge of great importance.

Appendix. Evaluation of Error Relating to the Estimated Effect
Below we show the details of the method to evaluate the error in the estimated22This virtual situation can be resulted from, for example, the conditions that procedures
for Web response are made further simplified and skills needed to answer via Web become
more widely spread. Such a virtual situation could be simulated by calculating theestimated probability while increasing the alternative-specific constant for the alternativeand observing the resulted sample characteristics.
23 We should note the following. If we analyze on this topic as a matter of labor market, solong as we take a simple approach, we could be neglecting the factors on demand side inlabor market.
25

increase in the number of respondents generated by Web mode (see Section 6). The error isevaluated by the delta method in the following steps. To simplify the formulae, the symbol^ (‚hat‛ indicating estimated values) is omitted.
The parameter vector θ is defined as follows:*A1+~θ  (αpaper~
βpaper

~
α
web

~

βweb

τBcoop )

.

The variance of D is calculated by below formulae:*A2+ D 
 D 
V ar(D)   V ar(θ) θ  θ 

*A3+

D NC(T  0) NC(T  1)


θ
θ
θ
.

The elements of

NC(T  0)
are calculated as shown below. The following
θ

~
~
~
~
~
~
 xi .
substitution is used: Vi ,paper  αpaper  βpaper x i , Vi ,web  α web  βweb

*A4+

~
n 
exp(τ Bcoop Vi ,paper ) NC(T  0) τBcoop
~
~
2


α
paper
i 1 {1  exp(τ Bcoop V i , paper )}



*A5+

~
n 

exp(τBcoop Vi ,paper )NC(T  0) τBcoop

x
i
~
~
2


{
1

exp(
τ
V
)}
β paperi 1
Bcoop i , paper



*A6+

NC(T  0)
0
~
α
web

*A7+

NC(T  0)
0
~
βweb

*A8+

~
n 

exp(τBcoop Vi ,paper )NC(T  0)
~




V
i
,
paper
~
2


τ Bcoop
{
1

exp(
τ
V
)}
i 1
Bcoop i , paper

.







26

The elements of

NC(T  1)
are calculated as shown below. From Formula *13+, weθhave the following:~~
IiBcoop  ln[exp(Vi , web /  Bcoop )  exp(Vi , paper /  Bcoop )]  ln[exp(Vi , web )  exp(Vi , paper)] .

*A9+

~
n 

exp(τBcoop I iBcoop )exp(Vi ,paper )NC(T  1)




τ

B
~
~
2
~
coop
α
{
1

exp(
τ
I
)}

exp(
V
)

exp(
V
)
paper
Bcoop iBcoopi 1 
i , paper
i ,web 


*A10+

~
n 

exp(τBcoop I iBcoop )exp(Vi ,paper )NC(T  1)




τ


x
Bcoop
i
~
~
~
{1  exp(τBcoop I iBcoop )}2 exp(Vi ,paper )  exp(Vi ,web )β paperi 1 



*A11+

~
n 

exp(τBcoop I iBcoop )exp(Vi ,web )NC(T  1)




τ

B
~
~
2
~
coop
α
{
1

exp(
τ
I
)}

exp(
V
)

exp(
V
)
web
Bcoop iBcoopi 1 
i , paper
i ,web 


*A12+

~
n 

exp(τBcoop I iBcoop )exp(Vi ,web )NC(T  1)




τ


x
Bcoop
i
~
~
~
{1  exp(τBcoop I iBcoop )}2 exp(Vi ,paper )  exp(Vi ,web )βwebi 1 



*A13+

n 

exp(τ Bcoop I iBcoop )NC(T  1)

 

I
iB
2
coop
τB
{
1

exp(
τ
I
)}


Bcoop iBcoopi 1 
coop












Inserting the estimated parameter and the variance-covariance matrix Var(θ), thevariance of D is obtained.

Supplementary Note
The JHPS2009 datasets used in this analysis were provided in August 2009 by the
Joint Research Center for Panel Studies at Keio University.

References[In English]
Couper, Mick P., and Peter V. Miller. ‚Web Survey Methods: Introduction.‛ Public Opinion

27

Quarterly 72 (5): 831-835.
Dillman, Don A. 1999. Mail and Internet Surveys: The Tailored Design Method (Second Edition).
New York: Wiley.
Dillman, Don A., Glenn Phelps, Robert Tortora, Karen Swift, Julie Kohrell, Jodi Berck, andBenjamin L. Messer. 2009. ‚Response Rate and Measurement Differences in Mixed-Mode
Surveys Using Mail, Telephone, Interactive Voice Response (IVR) and the Internet.‛
Social Science Research 38: 1-18.
Greene, William H. 2003. Econometric Analysis (Fifth Edition). Upper Saddle River, N.J.:
Prentice-Hall.
Groves, Robert M., Floyd J. Fowler Jr., Mick P. Couper, James M. Lepkowski, Eleanor
Singer, and Roger Tourangeau. 2004. Survey Methodology. Wiley.
Heiss, Florian. 2002. ‚Structural Choice Analysis with Nested Logit Models.‛ Stata Journal 2(3): 227-252.
Israel, Glenn D. 2009. ‚Obtaining Responsed by Mail or Web: Response Rates and Data
Consequences.‛ Paper Presented at the 64th Annual Conference of AAPOR.
Kaplowitz, Michael D., Timothy D. Hadlock, and Ralph Levine. 2004. ‚A Comparison of
Web and Mail Survey Response Rate.‛ Public Opinion Quarterly 68 (1): 94-101.
McCabe, Sean Esteban, Alison Diez, Carol J. Boyd, Toben F. Nelson, and Elissa R.
Weitzman. 2006. ‚Comparing Web and Mail Responses in a Mixed Mode Survey inCollege Alcohol Use Research.‛ Addictive Behavior 31: 1619-1627.
Naoi, Michio. 2009. ‚Attrition Bias in Longitudinal Survey: A Validation Study with
Interviwer’s Record of Respondent Mobility.‛ Joint Research Center for Panel Studies
Discussion Paper Series DP2009-003.
Train, Kenneth E. 2003. Discrete Choice Methods with Simulation. Cambridge: Cambridge
University Press.
[In Japanese]
Ministry of Internal Affairs and Communications. 2010. ‚2010 National Census Execution
Plan.‛ http://www.stat.go.jp/data/kokusei/2010/keikaku/pdf/sy02.pdf (Accessed March 24,2011).
Naoi, Michio, and Koji Yamamoto. 2010. ‚Sample Design and Representativeness of Japan
Household Panel Survey.‛ Yoshio Higuchi, Tamaki Miyauchi, and C. R. McKenzie, eds.
Dynamism of Poverty. Keio University Press: 3-27.
Naoi, Michio, Koji Yamamoto, and Tamaki Miyauchi. 2010. ‚Assessing the Performance of
Alternative Survey Methods on Unit-Response and Item-Response in JHPS.‛ Yoshio
Higuchi, Tamaki Miyauchi, and C. R. McKenzie, eds. Dynamism of Poverty. Keio

28

University Press: 29-63.
Yamamoto, Koji, and Hiroshi Ishida. 2010. ‚Final Disposition Codes and Survey Outcome
Rates for Japanese Life Course Panel Survey (JLPS): Measuring Cooperation of‚Targeted‛ Respondents and Success of Collection Agents.‛ The University of Tokyo
Institute of Social Science Panel Survey Project Discussion Paper Series, No. 34.

29

Figures and Tables

Table 1.

PC Use and Internet Access and Web ResponseHome PC Use and Internet Environment

Response via Web
Yes
No

Total

Uses Home PC; Has Internet Access
Uses Home PC; No Internet Access
Uses Home PC; Internet Access Unknown
No PC Use at Home
Unknown PC Use at Home

89
0
0
2
0

2,240
204
1
1,464
22

2,329
204
1
1,466
22

Total

91

3,931

4,022

Source: Table 1–6 from Naoi and Yamamoto (2010: 12)30

Table 2.

Probit Analysis of Internet Environment and Web Response
PC Use + Internet Access b
Coef.
S.E.
p

Web Responsec
Coef.
S.E.

p

Sex (Reference: Male)
Female-0.139 *

0.055

0.011

-0.228 +

0.128

0.075

Age (Reference:19–29 years old)
30–39
40–49
50–59
60+

0.091
0.079
-0.336 **-0.725 **

0.093
0.097
0.095
0.089

0.329
0.414
0.000
0.000

0.316 +
0.223
-0.061
0.158

0.174
0.190
0.229
0.208

0.070
0.242
0.791
0.449

Marital Status (Reference: Single)
Married0.142 *

0.064

0.025

-0.473 **

0.133

0.000

Education Level (Reference: Junior High/Senior School)
Junior College/Technical College
0.467 **
University
0.677 **
Graduate school
1.409 **
Other
0.469 **

0.078
0.065
0.293
0.108

0.000
0.000
0.000
0.000

0.189
0.176
0.174
0.190

0.179
0.140
0.308
0.218

0.290
0.208
0.573
0.385

Occupation (Reference: Unemployed)
Agriculture, forestry, and fisheries
Sales/Service
Management/Professional
Clerical/Information Processing
Transport/Manufacture/Security
Other-0.502 **
0.051
0.138
0.425 **
-0.157 +0.478 +

0.185
0.070
0.088
0.090
0.081
0.256

0.007
0.463
0.117
0.000
0.054
0.062

---d
-0.207-0.0620.185-0.1200.782 *

0.189
0.192
0.174
0.212
0.321

0.273
0.745
0.288
0.572
0.015

Household Income (Log)a0.355 **

0.039

0.000

-0.011

0.083

0.899

Constant-2.026 **

0.245

0.000

-1.585 **

0.540

0.003

n
Log-Likelihood

3349-1811.818

1993-292.840

Note: **: p < 0.01; *: p < 0.05; +: p < 0.10a"Household Income (Log)" is the logarithm of (household income (Unit: 10,000 Yen) + 1). This operation was
implemented because some households claimed 0 income.
b
Dependent variable was set to 1 when both of home PC and internet access were available and 0 otherwise.
When no response was given regarding home PC use or internet access, these cases were excluded from theanalysis.
c
Dependent variable was set to 1 when the participant responded online and 0 otherwise. The subsample for thismodel consists of only those with home PC and internet access.
dWhen this variable is 1, the dependent variable is always 0; i.e., this variable is a perfect predictor. Consequently,
this attribute is excluded from analysis. Thus, the 21 cases where this variable is 1 were excluded.
Source: JHPS200931

Choice Probability

Figure 2. Hypothetical Substitution Pattern (b-1)

Choice Probability

Figure 1. Hypothetical Substitution Pattern (a)Tendency of Web Response
Non Cooperation

Tendency of Web Response
Non Cooperation

Web Response

Paper Response

Paper Response

Note: Assuming "Web response" and "noncooperation" belong to the same nest B in a

Note: Assuming "Web response" and "paper
response" belong to the same nest B in a nested

nested logit model, with V i,paper = −0.331,
V i,nocoop = 0, and τ B = 0.01, the choice probability
in the range where V i,web is [−0.05, −0.006] isshown.

logit model, with V i,paper = −0.331, V i,nocoop = 0,and τ B = 0.01, the choice probability in the range
where V i,web is [−0.380, −0.331] is shown.

Choice Probability

Figure 3. Hypothetical Substitution Pattern (b-2)γδ

Tendency of Web Response
Non Cooperation

Web Response

Web Response

Paper Response

Note: Assuming "Web response" and "paper
response" belong to the same nest B in a nested
logit model, with V i,paper = −0.331, V i,nocoop = 0,and τ B = 0.5, the choice probability in the range
where V i,web is [−2.6, −0.331] is shown.

32

Figure 4. Nest Structures of Paper Response, Web Response, and Non-Cooperation

Nest Structure (a)Web Response

Non-cooperation

Paper Response

Nest Structure (b)Paper Response
Cooperation
Web Response

Non-cooperation

Nest Structure (c)Paper Response

Non-cooperation

Web Response33

Table 3. Inclusive Value Coefficient for Each Nest Structure

Estimate
Nest Structure (a)Nest Structure (b)Nest Structure (c)a

9.523
0.070
0.640

Inclusive Value Coefficient (τ ) for Nest including 2 Alternatives
Wald Test
Wald Test
95% C.I.
(H0: τ = 0)
(H0: τ = 1)
S.E.
5.776
0.188
0.698

Lower

Upper

χ2

p-valueχ2p-value-1.798-0.298-0.72820.844
0.439
2.008

2.720
0.140
0.840

0.099
0.709
0.359

2.180
24.470
0.270

0.140
0.000
0.606

Note: The Estimate of the inclusive value coefficient τ in each model corresponding to each nest
structure in Figure 4 are shown. The estimations were conducted only for those cases with whichcontact was made during the survey period. The formulation of the model is as per Greene (2003: 725727). The estimation algorithm is Newton–Raphson method. The explanatory variables used are thevariables shown in Table 4. For this estimation, no convergence condition relating to the first-order partial
derivative of the likelihood function was setup.
a

In the vicinity of the initially estimated parameters satisfying the convergence condition, the likelihood
function was not deemed to be a concave function, and hence, re-estimation was performed from theinitially obained set of parameters.
Source: JHPS200934

Table 4. Results of Nested Logit Model on Choice from Non-Cooperation/Paper Response/Web Response a
Base: Non-Cooperation
Paper Response
Web Response
p-valuep-value
Coef. bCoef. bSex (Reference: Males)
Female

0.028

0.498

-0.0230.871

Age (Reference: 19–29 years old)
30–39
40–49
50–59
60+

0.153 *
0.226 **
0.208 **
0.136 *

0.034
0.003
0.005
0.035

0.150 *
0.204 *
0.139
0.053

0.046
0.034
0.480
0.820

Housing Status (Reference: Detached and Wooden + N.A.)
Detached and Concrete
0.203 *
0.043
Multiple Dwelling and Wooden
0.363 **
0.000
Multiple Dwelling and Concrete
0.133 *
0.029

0.211 +
0.344 **
0.158 +

0.050
0.003
0.084

Area Block (Reference: Hokkaido)
Tohoku
Kanto
Chubu
Kinki
Chugoku
Shikoku
Kyushu

0.000
0.000
0.626
0.000
0.000
0.001
0.645

-0.940 **-0.908 **-0.004-0.353-0.729 **-0.760 **-0.040

0.005
0.000
0.984
0.135
0.001
0.003
0.782

Scale of City/Town/Village (Reference: Large City)
Other City
-0.009Town/Village
0.086

0.854
0.292

0.019
0.141

0.837
0.402

Survey Method (Reference: Retention only)Interview also Conducted

0.462-0.024

0.616

Initial Target/Back-up Target (Reference: Back-up Target)
Initial Target-0.143 *0.029-0.145 *0.038

Initial-Target Premium (Reference: No Premium)Premium for Initial Target Exists
0.521 **

0.000

0.490 **

0.003

0.293

0.218

-0.5260.506-0.843 **-0.968 **-0.059-0.429 **-0.757 **-0.717 **-0.057-0.031

Interactions between "Initial Target" and "Initial Target Premium"
Initial Target × Initial Target Premium
0.210 *
0.019
Alternative-Specific Constant
Inclusive Value Coefficientfor "Cooperation" Nest c
Inclusive Value Coefficientfor "Non-Cooperation" Nest dnLog-Likelihood-0.238

+

0.066

0.070
1.000

9621
-6873.731

Note: **: p < 0.01; *: p < 0.05; +: p < 0.10aEstimates of the model with nest structure (b) from Figure 4 are shown. The dataset used contain only the cases towhich contact was made during the survey period. The model formulation is as per Greene (2003: 725-727). The
estimation algorithm is Newton–Raphson method. For this estimation, no convergence condition relating to the firstorder partial derivative of the likelihood function was setup.
b
Coefficients shown are not the parameters in Greene's formulation itself, but they are the values of the parametersfrom Greene's formulation multiplied by the inclusive value coefficient. These values can be interpreted consistently with
RUM and are equivalent to β and α in formulae in the present paper. The standard errors for these modified coefficientswere estimated using the delta method, and p-values shown were also based on them.
c

For the standard error and result of Wald test on this parameter, see Table 3.
The inclusive value coefficient for the "non-cooperation" nest was constrained to 1 in estimation.
Source: JHPS2009d35

Table 5.

Estimate of the Effect of Adding Web-Mode based on a Nested Logit Model
Actual SelectorsnWith Web Mode(%)

Expected Number ofSelectors

Without Web Mode
Expected Numberof Selectors

Difference

Cooperation
Paper Response
Web Reponse

3,931
91

(40.86)(0.95)

3,930.3
91.9

4,018.4―

88.1―

Non-cooperation

5,599(58.20)5,598.8

5,602.6

3.8

Total

9,621 (100.00)

9,621.0

9,621.0

91.9

Note: Shown are: the expected number of selectors (sum of estimated probabilities to select thealternative for all cases in the dataset) when there exists Web mode, the expected number of
selectors when there is no Web mode, and the differences between them, all calculated based on
the estimates of the nested logit model in Table 4.
Source: JHPS2009

Table 6. Error Evaluation for Effect of Web-Mode Addition
Benchmark for the Effect of
Web-Mode Addition

Estimate

S.E.

95% C.I.
Lower
Upper

Increase in Number ofRespondents (Cooperators)

3.816

10.233-16.24223.873

Increase in Cooperation Rate(% points)

0.03970.1064-0.16880.2481

Increase in Response Rate(% points)

0.03040.0815-0.12940.1902

Note: Effect of Web-mode addition is shown for each index, based on the estimate of
the nested logit model in Table 4, and their errors are assessed. The errors areassessed by the delta method. The cooperation rate is defined as the ratio of thenumber cooperating in the survey to the number contacted in the survey. The response
rate is defined as the ratio of the number cooperating in the survey to the number
targeted by the survey. For JHPS2009, the number contacted in the survey is 9,621 andthe number targeted is 12,549.
Source: JHPS200936