----

##  Conducting Web-Based Surveys. ERIC Digest.                                   

The growth of the Internet has impacted on virtually every aspect of            society. Survey research is no exception. Two years ago in an                   informal search of Yahoo, Kay and Johnson (1999) identified over                2,000 Web-based surveys(1) in 59 areas. The interest in Web-based               surveying is not surprising as it offers a number of distinct                   advantages over more traditional mail and phone techniques. Examples            include reducing the time and cost of conducting a survey and                   avoiding the often error prone and tedious task of data                         entry (Medin, Roy & Ann, 1999). 
Email offers one option for                   distributing Internet surveys. Up until a few years ago email                   surveys were the predominate means of Internet surveying. As the                World Wide Web (WWW) has grown in popularity, the use of Hypertext              Markup Language (HTML) forms or Web-based surveys are becoming the              dominant method of gathering survey data. These forms streamline the            data collection process formatting and entering responses directly              into a database for analysis. Since HTML forms can be made                      programmable, it is also possible to have real time error checking              and correction increasing the accuracy of the data collection                   process. The formatting capabilities of HTML allow the creation of              easy-to-read and a attractive forms that may improve response rates.            In addition, the programmability of HTML forms makes it possible to             randomly order responses and tailor options based on information the            respondent supplies earlier in the survey. 
Combining an                       email "cover letter" as a means of contacting sampled people with               the use of an HTML form for data collection provides an especially              effective and efficient approach to Internet surveying. Modern email            packages automatically convert universal resource locators (URLs) or            web-addresses in the text of an email into a hyperlinks. Placing the            URL of the survey form in a cover letter email allows the respondent            to "click" their mouse on the URL to display the survey form and                subsequently fill it out. CONCERNS WITH WEB-BASED                             SURVEYINGAlthough Web-based surveying is very attractive, at                this point  it should be used with caution. Currently the biggest               concern in Internet surveying is coverage bias or bias due to                   sampled people not having or choosing not to access the                         Internet (Kay & Johnson, 1999; Crawford, Couper & Lamias, 2001).                Despite expediential growth of the Internet there are still large               numbers of people who do not have access and/or choose not to use               the Internet. It is also clear that there are wide disparities in               Internet access among ethnic and socioeconomic groups (Selwyn &                 Robson, 1998). 
There are specific populations where Internet                  access is extremely high and coverage bias is likely to be less of a            concern. College students and university faculty within the USA,                Canada and Western Europe are examples of such populations. Even                though coverage bias may be less of an issue in these groups,                   experience and comfort with Internet-based tools such as Web                    browsers is another serious potential source of bias both in                    response rates and the way people respond to the survey (Dillman,               Tortora & Bowker, 2001). 
Web-based surveying is still in the early            stages of development. The WWW is a unique media and it is not clear            to what extent the knowledge we have gained over years of experience            with more traditional surveying techniques fully applies to Internet            surveying (Dilman, Tortora & Bowker, 2001). Studies are just                    beginning to done to learn the optimal ways to structure and format             Internet surveys to limit biases and increase response rates. It is             also likely that the best way to design an Internet survey depends              in part on the familiarity and comfort of the respondents in using              Web browsers and email clients. It is also quite likely that the                type of Internet connection as well as the hardware and software                used in accessing the Internet will impact on response rates and                possibly how a person responds to an Internet-based survey. 
The               use of HTML forms for surveying poses a unique set of issues and                challenges that need to be addressed to ensure valid data. The Web              is a very public place and unless steps are taken to limit access to            a survey, it may be found and responded to by people who are not                among those sampled by the researcher. This can either happen by                accident or maliciously. Since one only has to "click" their mouse              pointer on the "submit" button to respond to a Web-based survey                 instrument once it is filled out, it is also quite possible for                 respondents to either mistakenly or purposefully submit multiple                copies of their responses. 
While Internet-based surveying                     techniques need to be used with caution, their benefits warrant                 continued exploration and the cautious use. It is also pretty clear             that coverage bias and familiarity with Internet tools will be less             and less of an issue over time. Additionally our knowledge about how            best to conduct Internet surveys will continue to improve with                  research and experience. RESEARCH ON INTERNET-BASED                           SURVEYINGAlthough the research on Internet-based surveying is               limited, findings are beginning to appear in the literature. Several            studies have found that response rates for Internet surveys are                 lower that equivalent mail surveys (Medin, Roy & Ann, 1999; Cooper,             Blair & Triplett, 1999). As noted by Crawford and colleagues (2001),            this may be due to our lack of knowledge on how to achieve high                 response rates using the Internet surveys. The lower response rates             for internet surveys may also reflect coverage bias, the lack of                familiarity with the media and/or lack of convenient access to the              Internet. In the author's experience, Web congestion can also be a              factor in lowering response rates for Web surveys particularly with             people who have relatively little experience with the Internet.Cook             and colleagues (2000) conducted a meta-analysis of factors                      influencing response rates in Internet-based surveys2. They found               that follow-up contacts with non-respondents, personalized contacts,            and contacting sampled people prior to sending out the survey were              the three dominant factors in higher response rates.                            Kittleson (1997) in a study of email-based surveying found it was               possible to double the response rate with follow-up memos though in             general this may be somewhat optimistic. As with mailed surveys,                repeated follow-ups have diminishing returns and at some point risk             irritating potential respondents without noticeably increasing                  response rates. Additionally, Dillman, Tortora, Conrad &                        Bowker (2001) found that relatively plain Web surveys that load                 quickly resulted in higher response rates than "fancier" surveys                that take longer to load. Jeavons (1998) analyzed detailed server               logs from three separate large-scale surveys. He found a relatively             high percentage of potential respondents stopped completing the                 surveys 1) when encountering the first question, 2) when                        encountering a complex question grid, and 3) when asked to supply               their email address. This suggests that some potential respondents              have difficulty with the media and give up early in the process of              completing the survey or when encountering complex questions. Others            may be reluctant to give out personal information such as an email              address. The logs were also merged with demographic data collected              via the surveys. Somewhat surprisingly no patterns in failure to                complete rates were found by gender, age or education level. In two             of the surveys, people with lower income were found to have a higher            rate of repeating screens of questions mainly due to improperly                 filling out questions. DEVELOPING WEB SURVEYSAs noted, due to             their inherent advantages, most Internet surveying is now being done            using HTML forms with potential respondents often contacted via                 email cover letters. While some developers still directly code these            forms in HTML, there are dozens of HTML editors available, and they             are becoming increasingly sophisticated and easy to use. There are              two general methods of capturing the data entered into an HTML form.            The form can be programmed to email the data back to a specified                email address or captured by a program on the server called a common            gateway interface (CGI) script. Using CGI scripts is more robust,               offers more flexibility and is the far more commonly used method of             capturing data. There are several HTML development packages that                both provide HTML editing capabilities and automate the process of              developing the CGI scripts necessary to capture data from HTML forms            developed with the package. Two widely used examples of these                   packages are Microsoft's FrontPage and Macromedia's ColdFusion.                 While these packages are general purpose Web development tools,                 there are also a growing number of software development systems                 designed specifically for Web-based surveying. Examples include                 Perseus's Survey Solutions for the Web, Creative Research System's              The Survey System, and Survey Said  Survey Software. These packages             tend to offer additional features specific to survey research.                  Examples include as managing the distribution of email cover                    letters, built-in statistical analysis and reporting capabilities,              and automatic tracking of people who have responded coupled with the            ability of sending out follow-up email reminders to those who have              yet to respond. Their HTML editors are also geared for survey form              development, allowing them to simplify and streamline the process of            developing and formatting the question response fields. Web Survey              Mailer System 
The author has developed a set of software tools                that provides many of the complex Web survey administration                     functions included in Web surveying packages(3). The software, Web              Survey Mailer System (WSMS), is an integrated survey administration             system that will send out personalized email cover letters, track               which of the sampled people have completed the survey while keeping             their responses anonymous and send out subsequent reminder emails to            only those sampled people who have not responded to the survey. WSMS            will block people who have not been sampled from accessing and                  responding to the survey and will keep respondents from submitting              more than one set of survey responses. The system includes a                    customizable CGI script to capture the survey responses and place               them in a tab-delimited ASCII database format that can easily be                downloaded from the server and imported into a standard PC data base            or statistical package. WSMS is written in PHP and uses the MySQL               relational database to store information on the sampled people. Both            PHP and MySQL are stable and powerful "open source" packages widely             available on university and commercial Web servers and can be                   obtained free of charge in a variety of versions that will run on               most common server operating systems. The WSMS scripts and                      documentation are available free-of-charge and can be downloaded                from                                                                            http://www.med-ed-online.org/rsoftware.htm#wsms SUMMARYInternet           surveys are clearly going to continue to grow in popularity as the              problems of coverage bias and unfamiliarity with the Internet                   subside. For the foreseeable future there will be people who will               lack Internet access either by choice or circumstance though this               will be less and less of an issue. Additionally the tools for                   conducting Web-based surveys will continue to grow in sophistication            and ease of use as will our knowledge on how best to employ this                survey methodology. At present researchers should use this technique            with caution in carefully chosen populations and with an eye to                 learning as much as possible about how to do it                                 better. NOTES1.In this paper we use the term "Internet survey"            for both email and HTML form-based surveying while the                          term "Web-based survey" is reserved for HTML form-based                         surveys. 
2.Cook, Heath, & Thompson (2000) included studies of both            Web- and email-based surveys. 
3.Detailed documentation for the Web            Survey Mailer System is provided. However, installing and using                 these tools requires a good working knowledge of HTML and some                  background and understanding of server-based programming. 
4.This              Digest is based on an article first appearing in Practical                      Assessment Research and Evaluation REFERENCESBest, S.,J.,                 Krueger, B., Hubbard, C. & Smith, A. (2001) An assessment of the                generalizability of internet surveys. Social Science Computer                   Review, 19, 131-145. 
Cook, C, Heath, F, & Thompson, R. (2000) A               meta-analysis of response rates in web or Internet-based surveys.               Educational and Psychological Measurement, 60, 821-836. 
Couper,               M.P., Blair, J. & Triplett T (1999) A comparison of mail and e-mail             for a survey of employees in federal statistical agencies. Journal              of Official Statistics, 15, 39-56. 
Crawford, S.D., Couper, M.P.&              Lamias, M.J.. (2001) Web Surveys: Perception of burden. Social                  Science Computer Review, 19,146-162. 
Dillman, D.A., Tortora,                  R.D. & Bowker, D. Principles for constructing  web surveys. Working             paper available from                                                            http://survey.sesrc.wsu.edu/dillman/papers.htm [Accessed                        6/01] 
Dillman, D.A, Tortora, R.D, Conrad, J. & Bowker D. Influence            of plan vs. fancy design on response rates of Web surveys. Working              paper available from                                                            http://survey.sesrc.wsu.edu/dillman/papers.htm [Accessed                        6/01] 
Jeavons A. Ethology and the Web: Observing respondent                   behavior in Web surveys. Proceedings of the Worldwide Internet                  Conference, Amsterdam: ESOMAR, 1998, available from                             http://w3.one.net/~andrewje/ethology.html [Accessed 7/01] 
Kaye                B.K. & Johnson T.J. (1999) .Research Methodology: Taming the Cyber              Frontier. Social Science Computer Review, 17, 323-337. 
Kittleson,             M. (1997) Determining effective follow-up of  e-mail surveys.                   American Journal of Health Behavior. 21, 193-196. 
Medin, C., Roy,             S. & Ann, T. (1999) World Wide Web versus mail surveys: A comparison            and report.. Paper presentation at ANZMAC99 Conference, Marketing in            the Third Millennium, Sydney, Australia, available from                         http://www.anzmac99.unsw.edu.au/anzmacfiles/papers.htm [Accessed                6/01] 
Selwyn, N., Robson, K. (1998) Using e-mail as a research                tool, Social Research Update, available from                                    http://www.soc.surrey.ac.uk/sru/SRU21.html  [Accessed                           6/01] 
----- 
This publication was prepared with funding from the             Office of Educational Research and Improvement, U.S. Department of              Education, under contract ED99CO0032.  The opinions expressed in                this report do not necessarily reflect the positions of policies of             OERI or the U.S. Department of Education.  Permission is granted to             copy and distribute this ERIC/AE Digest.                                     **Title:** Conducting Web-Based Surveys. ERIC Digest.                                   **Note:** Based on an article first appearing in "Practical Assessment                    Research and Evaluation."                                                    **Document Type:** Information Analyses---ERIC Information Analysis Products (IAPs) (071); Information Analyses---ERIC Digests (Selected) in Full Text (073); **Available From:** ERIC Clearinghouse on Assessment and Evaluation, 1129 Shriver                   Laboratory, University of Maryland, College Park, MD 20742. Tel:                800-464-3742 (Toll Free); Web site: http://ericae.net.                       **Descriptors:**  Research Methodology,  Response Rates (Questionnaires),  Surveys,               World Wide Web                                                              **Identifiers:** ERIC Digests                                                                 
###
[Return to ERIC Digest Search Page]