# Surveying the Surveyors: An Analysis of the Survey Response Rates of Librarians (Paper)

## Abstract or Résumé:

It is a well-established fact that the most widely employed research method by librarians is surveys. Given this fact, this ongoing study seeks to identify exactly how likely librarians are to respond to surveys and what, if any, circumstances will increase the likelihood they will respond. Using a quantitative content analysis, relevant literature from three separate LIS databases – Library Literature & Information Science Full Text (EBSCOhost), Library, Information Science & Technology Abstracts (EBSCOhost), and Library & Information Science Abstracts (ProQuest) – is currently being gathered and evaluated. Preliminary findings indicate trends regarding LIS research purposes, methodology, and subjects.

## 1. Introduction

According to several recent studies (Klem, Saleh, Devine, Gutzman, Knehans, Mills, ... Vardell, 2019; Luo & Mckinney, 2015; Tuomaala, Järvelin, & Vakkari, 2014; Ullah & Ameen, 2018), survey methodology has been the prevailing research method in multiple areas of library and information science (LIS) literature for more than 50 years. For instance, in a study on the evolution of LIS, in which the contents of 42 LIS journals were analysed, Tuomaala, Järvelin, and Vakkari (2014) found that, in the topic areas of “information-seeking” and “library and information-service activities”, surveys were the most popular research strategy in 1965, 1985, and 2005 (1455). Additionally, by 2005, surveys were the second most popular research strategy in “scientific and professional communication” and the third most popular in “information storage and retrieval” (1456). Hence, while surveys were shown to be the second most popular research strategy in 1965 and 1985 (just behind conceptual research), by 2005 they were unequivocally number one (1452, 1455). Similarly, a review of 35 articles¹ that reported on the use of methodologies and methods in LIS research found that surveys were used 33% of the time, with the next highest methods used being theoretical analysis (7.2%) and content analysis (7%) (Ullah & Ameen, 2018, 57). Thus, with librarians being so well-versed in survey methodology, one has to wonder, how often do librarians respond to surveys?

## 2. Survey Response Rate & Methodology Concerns

Researchers typically use surveys to collect three types of information: attitudinal, behavioural, and descriptive (Rea & Parker, 2005, 6). They do this by employing one or more modes of delivery, such as face-to-face, mail-out, telephone, and web-based (Cowles & Nelson, 2015; Groves, 2009; Johnston, 2008; Rea & Parker, 2005). Although all of these modes are still used, recent developments in communication and information technology have led to an increase in the
use of web-based surveys. However, prior research into the response rates for web-based surveys (Cobanoglu, Warde, & Moreo, 2001; Cole, 2005; Fricker & Schonlau, 2002; Greenlaw & Brown-Welty, 2009) indicates that, while they can vary greatly, the average is just over 30%. This is over 10% lower than the average response rate for mail-out surveys. Couper (2000) theorizes that this is because “the tried and tested motivating tools used in mail surveys... cannot be implemented in the same way in Web surveys”, adding that concerns over confidentiality or technical difficulties interacting with web-based surveys may be to blame (473-474). Yet, lower response rates are not the only issue arising with this shift to the digital. This change has also impacted sampling strategies, since web-based contact methods are far more likely to be non-probability-based as opposed to probability-based (Fricker, 2008, 212). Additionally, web-based surveys also face methodological challenges such as sample representativeness, data quality, dropout rates, and privacy and ethical concerns (Silva & Davis, 2015, 290). Thus, it is important to investigate the current state of survey methodology within LIS research and examine how the discipline that arguably uses surveys the most is tackling these issues.

**3. Research Questions**

RQ 1: What is the average response rate when surveying librarians?

RQ 2: How do response rates for web-based surveys compare with those of other modes?

RQ 3: What circumstances, if any, can increase librarian survey response rates?

RQ 4: What sampling strategies are being used when surveying librarians?

**4. Methodology**

Three LIS databases – Library Literature & Information Science Full Text (EBSCOhost), Library, Information Science & Technology Abstracts (EBSCOhost), and Library & Information Science Abstracts (ProQuest) – were searched using the following search terms: “survey or questionnaire” and “librarians” to obtain relevant research articles. Inclusion criteria consisted of peer-reviewed research published between 2010 and 2019 in which librarians were asked to fill out a survey and a response rate was obtained. Exclusion criteria consisted of full text articles in a language other than English and non-research article formats including books, book chapters, book reviews, column articles, dissertations, editorials, or theses. The analysis focused on the following variables: response rate, librarian type, survey type, survey purpose, and sampling method. However, other variables examined included country, journal, year published, year conducted, and length of survey availability. Although deductive coding was primarily used for this content analysis, inductive coding was necessary to create categories to code each survey's purpose.

**5. Findings**

Although data is still being gathered and evaluated, preliminary findings indicate:

* A response rate range of 8.9% to 96.8% with an average of 50.6%

* The majority of librarians being asked to fill out surveys are Academic librarians (72.7%) and Health librarians (13.6%)

* Email is the primary mode for seeking survey respondents
* Most librarians surveyed live in the United States (40.9%), presumably due to the higher number of librarians per capita and greater volume LIS research in the U.S.; however, response rates in the U.S. (37.8%) are typically lower than those in other countries (64.2%)

* Although surveys tend to be available on average for one month, this variable is not mentioned in almost 60% of articles

* Librarians are most often surveyed about knowledge related to their job (27.3%), the status of something related to their job (27.3%)², their opinion on something related to their job (22.7%), patron services related to their job (18.1%), and particular roles related to their job (4.5%)³.

* Non-probability-based sampling methods are primarily used for email-based surveys

**6. Discussion**

One trend that has already emerged in the data and is worth considering is the growing tendency among researchers to post survey invitations on listservs, discussion boards, and/or social media platforms. This is problematic as it often means that response rates cannot be calculated since the exact population number receiving the survey invitation is unknown. For instance, of the initial articles that met this study's criteria, almost 41% were eliminated because, due to this recruitment approach, a response rate could not be calculated.⁴ This is an issue because traditionally a survey's response rate has been viewed as an important indicator of survey quality.⁵ Thus, we must ask ourselves, if a survey's response rate is no longer an indicator of quality, what is?

**7. Limitations**

The results of this study are limited in that they do not include an exhaustive exploration of all available research articles that meet the inclusion criteria. The limitations of publication date, language, and databases used may have led to overlooking some relevant articles and/or data points.

**8. Conclusion**

Overall, this study has the potential to help LIS researchers to better understand current trends in their own survey methodology as well as what variables and circumstances could lead to higher response rates in future survey research.

**9. Links to Conference Theme**

This paper fits the conference's theme by exploring elements of survey methodology and data collection within LIS literature as well as questions regarding current survey practices and trends. As technology allows us to survey more people at a faster rate for a lower cost, it is worth asking ourselves, what are we losing in our pursuit of data?
Endnotes:

1. The time period covered by the selected reviews ranged from 1980 to 2016, with the majority falling between 2001 and 2016 (Ullah & Ameen, 2018, 56).

2. This includes purposes that could be considered a status assessment, such as the status of NGO librarians in a particular country, the acuteness of occupational burnout, and librarians' participation in systematic and scoping reviews.

3. This includes purposes that look at specific roles librarians may take on, such as promoting institutional repositories.

4. This is not to say that this recruitment approach always results in an inability to calculate a response rate, merely that it can and, according to this study's data, more often than not does.

5. Nevertheless, some studies have shown that lower response rates can be statistically indistinguishable from – or even more accurate than – higher response rates (Keller, 2014).
Reference List:

Cobanoglu, C., Warde, B., & Moreo, P. J. (2001). A comparison of mail, fax and web-based survey methods. *International Journal of Market Research, 43*(4), 441-15.
doi:10.1177/147078530104300401

Cole, S. T. (2005). Comparing mail and web-based survey distribution methods: Results of surveys to leisure travel retailers. *Journal of Travel Research, 43*(4), 422-430.
doi:10.1177/0047287505274655

Couper, M. P. (2000). Web surveys: A review of issues and approaches. *Public Opinion Quarterly, 64*(4), 464–494. Retrieved from
http://search.proquest.com/docview/214758072/

Cowles, E. L., & Nelson, E. (2015). *An introduction to survey research* (First ed.). New York, NY: Business Expert Press, LLC.

Fricker, R. D., & Schonlau, M. (2002). Advantages and disadvantages of internet research surveys: Evidence from the literature. *Field Methods, 14*(4): 347–67.
doi:10.1177/152582202237725

Fricker, R. D. (2008). Sampling methods for web and e-mail surveys. In N. Fielding, R. M. Lee, & G. Blank (Eds.), *The SAGE handbook of online research methods* (pp. 195-216). London: SAGE Publications. doi: 10.4135/9780857020055

Greenlaw, C., & Brown-Welty, S. (2009). A comparison of web-based and paper-based survey methods: Testing assumptions of survey mode and response cost. *Evaluation Review, 33*(5), 464-480. doi:10.1177/0193841X09340214

Groves, R. M. (2009). *Survey methodology* (2nd ed.). Hoboken, N.J: Wiley.

Johnston, R. (2008). Survey Methodology. In *The Oxford Handbook of Political Methodology*.
https://doi.org/10.1093/oxfordhb/9780199286546.003.0016

Keller, A. (2014, November 12). What is an acceptable survey response rate? *National Social Norms Center*. http://socialnorms.org/what-is-an-acceptable-survey-response-rate/

Klem, M., Saleh, A., Devine, P., Gutzman, K., Knehans, A., Mills, T., ... Vardell, E. (2019). Librarians and health literacy: A scoping review. *Library and Information Science Research, 41*(2), 102–108. https://doi.org/10.1016/j.lisr.2019.04.005

Luo, L., & Mckinney, M. (2015). JAL in the Past Decade: A Comprehensive Analysis of Academic Library Research. *The Journal of Academic Librarianship, 41*(2), 123–129.
https://doi.org/10.1016/j.acalib.2015.01.003

Rea, L. M., & Parker, R. A. (2005). *Designing and conducting survey research: A comprehensive guide* (Third ed.). San Francisco, CA: Jossey-Bass, a Wiley brand.
Silva, J., & Davis, M. (2015). Workshop Synthesis: Respondent/Survey Interaction in a World of Web and Smartphone Apps. *Transportation Research Procedia, 11*(C), 289–296.
https://doi.org/10.1016/j.trpro.2015.12.025

Tuomaala, O., Järvelin, K., & Vakkari, P. (2014). Evolution of library and information science, 1965–2005: Content analysis of journal articles. *Journal of the Association for Information Science and Technology, 65*(7), 1446–1462.
https://doi.org/10.1002/asi.23034

Ullah, A., & Ameen, K. (2018). Account of methodologies and methods applied in LIS research: A systematic review. *Library and Information Science Research, 40*(1), 53–60.
https://doi.org/10.1016/j.lisr.2018.03.002