AccuracyofAnswers
ProvidedbyDigital/Face-to-face Reference Services in Japanese
Public Libraries and Q&A Sites
Keita Tsuji1, Haruna To2, and Atsuyuki Hara1
12

Graduate School of Library, Information and Media Studies, University of Tsukuba
School of Library and Information Science, University of Tsukuba

Abstract: We asked the same questions using digital reference services (DRS) in
Japanese public libraries, face-to-face reference services, and Q&A sites. It was foundthat (1) the correct answer ratio of DRS was higher than that of Q&A sites, (2) DRS
takes longer to provide answers as compared to Q&A sites, and (3) the correct answer
ratio of face-to-face reference services in public libraries that provide (do not provide)
DRS was higher than (the same as) that of Q&A sites. Considering that a majority of
the Japanese public libraries do not provide DRS, the result indicates that Q&A sites
and face-to-face reference services are comparable in terms of their ability to answer
questions. However, the public libraries that earnestly provide DRS can outperform
Q&A sites.

1. Introduction
Reference services are evolving with the development of the Internet. A
significant change has been the rapid growth of Q&A sites. In this paper, we
define Q&A sites as those sites where anybody can ask questions for free,
and in many cases, receive answers from other users or specialists. In Japan,
Yahoo! Chiebukuro and Oshiete! goo are typical Q&A sites. Employing
unobtrusive testing, Tsuji et al. (2010) asked the same questions using
Oshiete! goo and face-to-face reference services (henceforth “face-to-face”)
in Japanese public libraries and found no difference between the accuracy ofanswers provided by both these sources. However, Tsuji et al. (2010)
investigated only face-to-face and municipal libraries. If they had
investigated e-mail digital reference services (henceforth “DRS”) andprefectural libraries, which possess more human and information resources,
the result may have varied. DRS are similar to Q&A sites in terms of theiravailability on the Internet. If the answers obtained by using DRS prove to be
more accurate than those from Q&A sites, people may begin using DRS
instead of Q&A sites. Against this background, we asked the same 60
questions to (a) DRS in prefectural and municipal libraries, (b) face-to-face,and (c) Q&A sites. Subsequently, we compared the accuracy of the answers
provided by each of these sources.
This paper may be criticized with respect to the following two problems,i.e., (1) unobtrusive testing and (2) focusing on the correct answer ratio. With

regard to (1), it is believed that unobtrusive testing may lower the morale oflibrarians who are investigated, and may waste library resources (Weech(1974)). However, providing feedback to the libraries based on findings may
aid them in improving their services (Kaske & Arnold (2002)). In addition, in
the U.S., where progressive reference services are observed, unobtrusive
testing is performed rather frequently, as mentioned subsequently. With
regard to (2), we understand that reference services consist of various
services and therefore, various evaluation measures have been proposed. A
popular measure is the degree of user satisfaction, which provides potential
for future research.

2. Related Studies
In this section, we describe the related studies and current status of Q&A sites
and reference services.

2.1 Q&A Sites
There are numerous Q&A sites in Japan, including Yahoo! Chiebukuro,
Oshiete! goo, OKWave, Hatena::Question, and livedoor knowledge.1 Amongthem, Yahoo! Chiebukuro is the largest and comprises over 38.4 million
Q&A and had 5.4 million registered users as of March 2010. Oshiete! goo
and OKWave are the second largest and both comprise 4.8 million Q&A each.
In the U.S., Yahoo! Answers, WikiAnswers, and AnswerBag are the mostpopular.2 According to Hitwise, the number of Q&A users in the U.S. is
rapidly increasing and number of visitors to Q&A sites increased by 889%
from February 2006 to February 2008. Coffman & Arret (2004) stated that alarge number of commercial reference services that worried librarians had
either died, or were so gravely wounded that they could no longer constitute a
threat to anybody. However, currently Q&A sites are gaining popularity and
we are uncertain whether or not their optimism will hold in the future.
There are no prior studies evaluating the accuracy of answers obtained
from Japanese Q&A sites except Tsuji et al. (2010), whose work has beenpreviously mentioned. As for the U.S. and U.K., Margariti & Chowdhury(2003), Lochore (2004), Roush (2006), and Bivens-Tatum (2001) evaluatedthe Q&A sites; however, their investigations were relatively limited.

2.2 Reference Services
Crews (1988) reviewed 39 papers, which employed unobtrusive testing for
1

Yahoo! Chiebukuro <http://chiebukuro.yahoo.co.jp/>，Oshiete! goo<http://oshiete.goo.ne.jp/>，OKWave <http://okwave.jp/>，Hatena::Question<http://q.hatena.ne.jp/>，livedoor knowledge <http://knowledge.livedoor.com/>.
2Yahoo! Answers <http://answers.yahoo.com/>，WikiAnswers<http://wiki.answers.com/>，AnswerBag <http://www.answerbag.com/>.

evaluating the accuracy of answers provided by reference services. Amongthem, Hernon & McClure (1986) and Crowley (1971) are the most popular.
Kaske & Arnold (2002)，Lochore (2004)，and Nilsen & Ross (2006)
investigated the accuracy of answers provided by DRS. Lochore (2004)
asked 10 questions using DRS (a Librarian in the U.K. and DRS by UCLA)
and a Q&A site (AllExperts) and found that the two DRS provided three andfive correct answers, respectively, and the Q&A site provided four. Studies
comparing the accuracy of answers obtained by using DRS and Q&A sites
are limited.

3. Method
In this section, we will describe (1) the public libraries and Q&A sites that weused for the investigation, (2) questions, and (3) answer categories.

3.1 Public Libraries and Q&A sites
We selected 6 prefectural libraries in Kanto district and 25 municipal libraries
around Tokyo and asked questions using their face-to-face. In addition, we
chose 6 prefectural libraries and 3 municipal libraries and asked questions
using their DRS. The number of samples of libraries for evaluating theaccuracy of answers provided by DRS is small because Japanese municipal
libraries do not provide DRS, and even if they do, the users are restricted to
the residents of their service area for a majority of such libraries. With respect
to Q&A sites, we selected Yahoo! Chiebukuro and Oshiete! goo.

3.2 Questions
For the purpose of our study, questions must be answerable and easy for us toevaluate the accuracy of their answers. Based on this, we employed 30
questions that have been used in the classes of “Reference service practice” atour university and 30 questions that have been asked by users in public
libraries and stored in their reference records. It must be noted that
confidential questions asked by users were not included. In 2009, we asked
the 60 abovementioned questions using (1) face-to-face (mainly from August
28 to October 6), (2) DRS (from November 17 to December 2), and (3) Q&A
sites (from December 9 to December 11).
With respect to face-to-face, on an average, we asked two questions toeach library. With respect to DRS, we asked about 8 and 6 questions to eachprefectural and municipal library, respectively. It was ensured that the samequestions were not asked using face-to-face and DRS in the same libraries.
With regard to Q&A sites, we asked a different set of 30 questions each using
Yahoo! Chiebukuro and Oshiete! goo.

3.3 Answer Categories
We classified the answers based on (a) amount of correct answers, (b) amountof incorrect answers, and (c) whether or not the correct answer was provided
directly (i.e., by introducing certain books, etc.). A few of the questions that
we employed consisted of multiple sub-questions. For example, a questionthat enquires the title and publisher of a book consists of two sub-questions,
one each for enquiring about the title and publisher of the book. Therefore,
the answer to one question can be partially correct (for example, the title
could be correct and name of the publisher could be incorrect or vice versa).
On the basis of these classifications, the answers provided by the threeservices were classified into 9 categories in the following manner: (A)Correct answer was provided directly and completely. Incorrect answer was
not provided. (B) Books, Web pages, or other materials that included thecorrect answer were indicated; the answer could be found easily. Incorrect
answer was not provided. (C) Books, Web pages, or other materials that
included the correct answer were indicated; however, it was not easy to findthe answer (i.e., it required additional time, skill, or knowledge). Incorrect
answer was not provided. (D) Books, Web pages, or other materials that
included the correct answer were indicated; however, it was not easy to findthe answer. Incorrect answer was provided. (E) Part of the correct answer wasprovided directly. Incorrect answer was not provided. (F) Part of the correct
answer was provided directly. Incorrect answer was provided. (G) Neither thecorrect answer nor reference materials were provided (this refers to thosecases where librarians were unable to find the answer or Q&A sites provided
no answer). Incorrect answer was not provided. (H) Neither the correct
answer nor reference materials were provided. Incorrect answer was provided.
(I) Difficult to evaluate the accuracy of the answer (includes cases where theanswer advises visiting the national library or provides names of a large
number of books and advises referring to all of them).
Henceforth, we term the ratio of answer categories A and B among all theanswers as “correct answer ratio.” In addition, we group answer categories A
and B to represent “Good”; C and D, to represent “Not Good”; and E, F, G,
and H, to represent “Bad” in the interest of brevity.

4. Results and Discussions
In this section, we initially indicate the basic results, followed by the results
according to question topics, time required, answers sources, and relationshipbetween DRS and face-to-face.

4.1 Basic Results
The correct answer ratios of DRS, face-to-face and Q&A sites are indicatedin Tables 1, 2, and 3, respectively. As indicated in Tables 1 and 3, 20 out of
46 (i.e., 43.5%) answers provided by DRS were classified into category A
while only 8 out of 30 answers (i.e., 26.7%) provided by Yahoo! Chiebukuro
were classified into category A.

Based on these tables, the following can be established: (1) The correct
answer ratio of DRS in municipal libraries is 85.7% (i.e., 57.1% + 28.6%),
which is higher than that of prefectural libraries (67.4% (i.e., 43.5% +23.9%)), as well as the correct answer ratio of face-to-face, and Q&A sites.
(2) The correct answer ratio of DRS in a prefectural library is higher than thatfrom a Q&A site. (3) The number of incorrect answers provided by DRS (i.e.,
answers categories D, F, and H) are fewer than those by face-to-face and
Q&A sites. (4) The correct answer ratios of face-to-face in prefectural andmunicipal libraries do not differ significantly. (5) The correct answer ratio offace-to-face in municipal libraries is 62.5%, which is slightly higher than thatof Oshiete! goo (56.6%). Tsuji et al. (2010) had described that the correct
answer ratios of face-to-face in municipal libraries and that of Oshiete! goo
do not differ significantly. Similar results have been obtained in this study. (6)The correct answer ratio of Oshiete! goo (56.6%) is a slightly higher than thatof Yahoo! Chiebukuro (50.0%), although it is not statistically significant.
A
B
C
D
E
F
G
H
I
Total

Prefectural
20 ( 43.5 )
11 ( 23.9 )9 ( 19.6 )0 ( 0.0 )2 ( 4.3 )1 ( 2.2 )1 ( 2.2 )0 ( 0.0 )2 ( 4.3 )
46 ( 100 )Municipal
8 ( 57.1 )4 ( 28.6 )1 ( 7.1 )0 ( 0.0 )0 ( 0.0 )0 ( 0.0 )1 ( 7.1 )0 ( 0.0 )0 ( 0.0 )
14 ( 100 )Table 1: Answer Categories
by DRS
A
B
C
D
E
F
G
H
I
Total

Yahoo! Chiebukuro
8 ( 26.7 )7 ( 23.3 )
3 ( 10.0 )0 ( 0.0 )3 ( 10.0 )0 ( 0.0 )5 ( 16.7 )2 ( 6.7 )2 ( 6.7 )
30 ( 100 )Table 2: Answer Categories
by face-to-faceOshiete! goo
13 ( 43.3 )4 ( 13.3 )2 ( 6.7 )1 ( 3.3 )
3 ( 10.0 )0 ( 0.0 )6 ( 20.0 )1 ( 3.3 )0 ( 0.0 )
30 ( 100 )Table 3: Answer Categories
by Q&A sites

4.2 Question Topics
We classified the questions into 8 topics (Persons/Organizations, Geography,

History,
Events,Languages,
Books,
Referencebooks,
andJournals/Newspapers) and investigated the correct answer ratio for each topic.
Tsuji et al. (2010) reported that the correct answer ratio of Q&A sites forquestions with regard to “Books” was higher than that of face-to-face.
However, such tendencies were not observed in our study. Correct answer
ratios of face-to-face for questions regarding “Books” were 60.0% and 55.6%
for prefectural and municipal libraries, respectively. On the other hand, thecorrect answer ratios for questions regarding “Books” were 50.0% and 46.7%
of Yahoo! Chiebukuro and Oshiete! goo, respectively. With respect to DRS,
the correct answer ratios for questions regarding “Books” were 75.0% and85.7% for prefectural and municipal libraries, respectively.

4.3 Time Required
We investigated the time required to obtain answers from DRS, face-to-face,and Q&A sites and their respective correct answer ratios. In this paper, “timerequired to obtain answers” is defined as the sum of “the time taken by theservices to reply” and “time taken by us to find answers by referring
materials such as books and Web pages, which were suggested in the replies.”
Undoubtedly, the time taken in the latter case is nil when the replies directly
indicated the answers (i.e., in case of category A answers).
We cannot present the complete result due to a limitation of space. In
brief, we obtained answers for only 2 questions within an hour using DRS.
On the other hand, we obtained answers for 53 and 22 questions within anhour using face-to-face and Q&A sites, respectively. It may be the future task
for DRS librarians to reduce the time to answer questions. However, we must
keep in mind that the correct answer ratio of DRS was the highest among
these services.
Answers were obtained for 29 and 11 questions within 30 minutes using
face-to-face in municipal libraries and Yahoo! Chiebukuro, respectively.
Concerning these, the correct answer ratios of face-to-face in municipal
libraries and Yahoo! Chiebukuro are 55.2% and 81.8%, respectively. The
face-to-face is the fastest source for obtaining answers, and Q&A sites arecomparatively faster than DRS.

4.4 Sources
We investigated the sources that were referred in the answers obtained from
DRS, face-to-face, and Q&A sites. We cannot indicate all the sources referredby the other services owing to the limitation of space. In brief, a majority of
the referred sources in DRS were dictionaries (for 20 questions) followed by
NDL-OPAC.3 DRS in prefectural and municipal libraries often did not refersources (for 6 and 4 questions in prefectural and municipal libraries,
respectively. Five of these answers were for questions that investigated titles
3

NDL-OPAC is an OPAC of national library in Japan.

or publishers of books.4) If the source (such as NDL-OPAC, Webcat Plus,5 ortheir own OPAC) is indicated, the users could learn of its existence and may
use it subsequently.6
Q&A sites often referred no source as well (for 17 questions) and referred
Wikipedia (for 9 questions). However, they referred NDL-OPAC or Webcat
Plus (for 9 questions) as sources including DRS and reliable government
homepages (for 5 questions). Furthermore, Q&A sites referred the image ofthe body of a book provided by Google Books (for 1 question). Tsuji et al.
(2010) also found that a few of the answers on Q&A sites referred such
images of books provided by Google Books. If we regard printed books asreliable sources, it may be established that Q&A sites as well as reference
services in libraries refer reliable sources for their answers. In addition,reference librarians must learn to utilize such sources for their DRS.

4.5 Face-to-face where DRS is being provided
Table 4 indicates the correct answer ratios of face-to-face by three categories
of public libraries including (1) libraries that provide DRS, (2) libraries
whose central library’s provide DRS (“C-Provide” in the Table), and (3)libraries that do not provide DRS. Table 4 indicates the correct answer ratioof face-to-face in public libraries that provide DRS is 87.5% (i.e., 7/(7+0+1))
and is significantly better than those that do not provide DRS (57.7%). Table
3 indicates that the correct answer ratios of face-to-face by public libraries
that provide DRS outperform those of Q&A sites. However, those public
libraries that do not provide DRS do not differ significantly from the correct
answer ratios of Q&A sites.
ProvideC-Provide
Not ProvideGood
7 ( 87.5 )8 ( 57.1 )
15 ( 57.7 )Not Good0 ( 0.0 )1 ( 7.1 )
3 ( 11.5 )Bad
1 ( 12.5 )5 ( 35.7 )8 ( 30.8 )Table 4: Correct Answer Ratio of face-to-face by three categories of public libraries

5. Conclusions
We asked the same 60 questions using DRS in Japanese public libraries,
face-to-face, and Q&A sites. It was found that (1) the correct answer ratio of
DRS in prefectural libraries is higher than that of Q&A sites, (2) the correct
answer ratio of DRS in municipal libraries is rather high, (3) DRS takes
longer to provide answers to questions as compared to Q&A sites, (4) there isno significant difference between the correct answer ratio of face-to-face in
4

5 other answers were for those questions that the librarians could not find answers
to.
5
Webcat Plus is a nation-wide OPAC of Japanese university libraries.
6
With regard to face-to-face, no source was referred for only 4 questions.

public libraries that do not provide DRS and that of Q&A sites, and (5) the
correct answer ratio of face-to-face in public libraries that provide DRS was
higher than that of Q&A sites.
Considering that most of the Japanese public libraries do not provide
DRS, the result indicates that Q&A sites and ordinary face-to-face reference
are comparable in terms of their ability to answer questions; however, if the
public libraries earnestly provide DRS, they could outperform Q&A sites. We
should take these results into account while considering the future ofreference services.
Subsequently, we will investigate DRS and face-to-face in locations otherthan Kanto district. In addition, we will interview DRS librarians in order toexamine the relationship between the process of answering questions inlibraries and correct answer ratio. Finally, we would like to evaluate thesatisfaction levels of users employing DRS.

References
Bivens-Tatum, W., (2001). Expert Services on the Web: The Commercial Competitionfor
Libraries.
C&RL
News,
Vol.62,
No.7,<http://www.ala.org/ala/mgrps/divs/acrl/publications/crlnews/2001/jul/expertservi
ces.cfm>.
Coffman, S. and Arret, L., (2004). To Chat Or Not to Chat: Taking Another Look at
VirtualReference,Part
1.
Searcher,Vol.12,
No.7,<http://www.vrd2003.org/proceedings/presentation.cfm?PID=196>.
Crews, K. D., (1988). The Accuracy of Reference Service: Variables for Research andImplementation. Library and Information Science Research, Vol.10, No.3,
331-355.
Crowley, T., (1971). The Effectiveness of Information Service in Medium Size Public
Libraries. [Crowley, T. and Childers, T., Information Service in Public Libraries:
Two Studies. Metuchen, N.J., Scarecrow Press], 1-71.
Hernon, P. and McClure, C. R., (1986). Unobtrusive Reference Testing: The 55
Percent Rule. Library Journal, Vol.111, No.7, 37-41.
Hitwise, (2008). U.S. Visits to Question and Answer Websites Increased 118 Percent
Year-over-Year.
<http://www.hitwise.com/press-center/hitwiseHS2004/question-and-answer-websi
tes.php>.
Kaske, N. and Arnold, J., (2002). An Unobtrusive Evaluation of Online Real Time
Library Reference Services. Library Research Round Table, Annual Conference ofAmericanLibrary
Association,<http://www.lib.umd.edu/groups/digref/kaskearnoldunobtrusive.html>.
Lochore, S., (2004). How Good are the Free Digital Reference Services? A
Comparison of Library-based and Expert Services. Library Review, Vol.53, No.1,
24-29.
Margariti, A. and Chowdhury, G. G., (2003). Digital Reference Services: Do We Still
Need Libraries. Proceedings of the 6th International Conference on Asian Digital
Libraries (ICADL 2003), 158-167.
Nilsen, K. and Ross, C. S., (2005). Evaluating Virtual Reference from the Users'
Perspective. The Reference Librarian, Vol.11, No.2, 175-182.
Roush, W., (2006). What's the Best Q&A Site? Technology Review,<http://www.technologyreview.com/InfoTech/17932/>.
Tsuji, K., Umehara, K., Kikawada, A. and Hara, A., (2010). Correctness of Answers
Provided by Q&A Sites and Reference Services of Public Libraries. The Library
World, Vol.61, No.6, 594-608. (Text in Japanese).
Weech, T. L., (1974). Evaluation of Adult Reference Service. Library Trends, Vol.22,
No.3, 315-335.