NIST Research Library 

Benchmark Study, 2001

Summary Report

Paula Deutsch 

Diane Cunningham

 Lisa Greenhouse

Susan Makar 

Warren Overman

 Susan Sanders

Barbara Silcox

NISTIR 7008

NISTIR 7008

NIST Research Library 

Benchmark Study, 2001

Summary Report

Paula Deutsch 

Diane Cunningham

 Lisa Greenhouse

Susan Makar 

Warren Overman

 Susan Sanders

Barbara Silcox

Information Services Division

Technology Services

June 2003

U.S. DEPARTMENT OF COMMERCE

Donald L. Evans, Secretary

TECHNOLOGY ADMINISTRATION

Phillip J. Bond, Under Secretary of Commerce for Technology

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

Arden L. Bement, Jr., Director
ii 

Abstract: 

In 2002, the NIST Research Library of the Information Services Division, Technology Services, 

benchmarked itself against other science and technology libraries. These libraries had similar 

focus areas and were comparable in size. The study compared library budgets, collection sizes 

and costs, staffing, and library services. This report outlines the methodology used. Significant 

findings are documented and further implications for the Research Library are addressed. 

Lessons learned from the study are also discussed. 

Keywords: 

benchmark, budget, collection, document delivery, information center, Information Services 

Division, library, National Institute of Standards and Technology, NIST Research Library, 

Technology Services. 
iii 

Table of Contents 

List of Figures iv 

 

1. Introduction 1 

2. Method 2 

 2.1 Developing the Survey 2 

 2.2 Benchmark Partners 3 

 2.3 Understanding the Data 4 

 

3. Findings 5 

 3.1 Overall Budget 5 

 3.2 Materials 6 

 3.3 Document Delivery 8 

 

4. Conclusion 10 

 4.1 Best Practices 10 

 4.2 Opportunities for Improvement 10 

 4.3 Where the Research Library Excels 11 

 4.4 Lessons Learned 13 

 4.5 Next Steps 13 

 

Appendix A. Benchmark Questionnaire 14 

Appendix B. Talking Points for Contacting Libraries 25 

 
iv 

List of Figures 

Figure 1. Average total expenditures per customer 5

Figure 2. Distribution of all expenditures shown as percentage of total 6

Figure 3. Percent of total materials budget spent on books and journals 7

Figure 4. Resources purchased per customer 7

Figure 5. Average cost per journal 8

Figure 6. Average per-item library subsidies for document delivery services 8

Figure 7. Relationship of items loaned and items borrowed at no charge 9

Figure 8. Average turn-around time for document delivery services 10

Figure 9. Ratio of customers per staff member 12
1 

1. Introduction 

The NIST Research Library is one of three work units within the Information Services Division 

(ISD). ISD is in the organizational unit of NIST Technology Services. The Research Library has 

a staff of 16 and maintains a collection of about 300,000 volumes and 1,150 journal 

subscriptions. Its primary customers are the researchers in the laboratory programs at the 

Gaithersburg campus of the National Institute of Standards and Technology. 

The Research Library initiated a benchmark study in the fall of 2001. It grew out of concerns 

expressed by the NIST Research Advisory Committee1 (RAC) in April 2001 about the declining 

state of the Research Library’s collections. In their annual report to the NIST director, the RAC 

indicated that it considered the Research Library to be a critical element of the NIST 

infrastructure however, its stagnant and inadequate funding was undermining its ability to 

maintain its high standard of service. 

 In response to these concerns the NIST Director’s Office asked ISD to undertake several 

assessment activities to determine if additional funding was warranted. Management agreed with 

RAC that the Research Library is indeed a vital NIST resource, however, determining spending 

priorities to fund all vital NIST overhead activities involved making difficult choices, given the 

limited overhead dollars. While the benchmark study was one of the requested activities, the 

exercise of benchmarking supports the goals of ISD and Technology Services to better 

understand and improve how it provides services to its customers. When the benchmark study 

was initiated in 2001, the Research Library's base operating budget for the purchase of all 

collection materials had been static since 1995. In FY 2002, the library received an increase to 

base of 11 percent However, over the last six years the cost of scientific journals has increased 

by 24 percent2

. Using the research requirements and recommendations of the NIST scientific 

and research community, the Research Library canceled subscriptions to materials considered 

dispensable, and re-deployed expenditures to critical or core journal titles and databases during 

this time period. 

Examination of the data collected from this study has enabled the library to assess how it 

compares with peer institutions, and to identify some best practices to emulate. The Library has 

used the findings from this study to guide program improvement. It continues to provide the 

basis for making changes in practices, procedures, and services to improve responsiveness to 

customer needs and stewardship of resources. 

 

1

 The NIST Research Advisory Committee was established in May 1976 by the NIST Director. Its roles 

include providing advice and recommendations to the Director and Executive Board on scientific issues; assessing 

the climate and status of forefront research activities at NIST; and acting as a spokesperson for scientific concerns 

and opportunities. 

2

 Based on statistics supplied by the Association of Research Libraries at 

http://www.arl.org/stats/arlstat/graphs/2001/2001t2.html. 
2 

2. Method 

As an initial step, a report was prepared for NIST management in the summer of 2001. This 

report compared Research Library data with data compiled by the Association of Research 

Libraries (ARL) on library materials budget expenditures and interlibrary loan/document 

delivery statistics. Because ARL libraries are much larger institutions than the Research Library, 

percentage comparisons were made of the differences (in terms of collection size and budgets) 

between ARL libraries and the Research Library. The report also included a description of the 

approach to be used for conducting a benchmark study. It outlined the criteria that would be 

used for selecting benchmark partners, included a list of suggested partners and, delineated what 

would be measured in the study. 

After submitting this report, ISD management assembled a five-member Benchmark Team to 

conduct a study with libraries and information centers comparable to the Research Library. The 

team represented a cross section of the Information Services Division. Three people were 

members of the Research Library and Information Group, one person represented the Electronic 

Information and Publications Group, and one person came from the Museum and History 

Program. This provided representation and experience both within and external to the library 

field. The team met twice monthly for a one-year period. Meetings were used to develop a plan 

of action, design the survey instrument, coordinate communication with potential benchmark 

partners and then finally to evaluate and analyze the data received. 

2.1 Developing the Survey 

The Benchmark Team spent about four months developing a survey tool for this project. The 

team began by exploring survey instruments from several notable libraries and library 

organizations. Library staff focused on the survey instruments and data collected by the 

Association of Research Libraries (ARL)3; the U.S. National Commission on Library and 

Information Science (NCLIS); the National Library of Canada (NLC); and the U.S. National 

Center for Education Statistics (NCES). Staff also consulted existing standards including ISO 

2789: International Library Statistics; ISO 11620: Library Performance Indicators and; 

ANSI/NISO Z39.7-1995: Library Statistics. 

After careful examination, the team determined that the instruments developed by ARL were the 

most comprehensive, and provided clearly articulated standards for quantifying library resources. 

From these ARL models, the team formulated its own survey instrument. The final survey 

included 69 questions that assessed collections, services, staffing, budgets, and expenditures (see 

Appendix A). 

As a means for pre-testing the survey instrument, the team completed the survey using data for 

the Research Library. By doing this, the team was able to identify questions that were unclear or 

 

3

 ARL, a not-for-profit membership organization comprised of the leading research libraries in North America, has 

continuously collected library statistics for the past 40 years and is widely respected for its progressive work on 

developing methods of measurement in the library field. 
3 

vague. Based on what was learned from this exercise, the team then made revisions to the 

survey. 

2.2 Benchmark Partners 

Based on information from the American Library Directory and input from the NIST Director’s 

Office and the NIST Industrial Liaison Office, 15 potential benchmark partners were identified. 

These libraries and information centers appeared to be similar to the Research Library in the 

following ways: collection size, library materials budget, size of customer base, library staff 

size, and a target audience of researchers in the science/technology fields. 

ISD included government, academic and private sector libraries in its benchmark survey. 

However, it is important to note that private sector libraries may be fundamentally different from 

the Research Library. The information needs of the customers supported by these libraries are 

more likely to be tied to short-term business goals and less focused on the broad needs of generic 

research and development. 

Once potential partners were identified, each member of the team was assigned three 

libraries/information centers to contact by phone. The team agreed on a set of talking points that 

would be communicated to each potential partner (see Appendix B). If a library/information 

center agreed to participate, the team member who originally contacted it remained the contact 

point throughout the process. The team felt this was important for the comfort of the partners. 

The amount of interaction that occurred with benchmark partners while they were completing the 

survey varied from partner to partner. 

Of the 15 libraries contacted, seven libraries were willing to participate in this study. These 

include: Naval Research Laboratory - Ruth H. Hooker Research Library, Sandia National 

Laboratories Technical Library, Oak Ridge National Laboratory Research Libraries, IBM - 

Thomas J. Watson Research Center Library, Xerox Corporation Technical Information Center, 

and two libraries who asked not to be identified as participants. The total group represents three 

federal libraries, two corporate special libraries, and two non-profit research/academic libraries. 

Two libraries did not provide any financial information and one library withheld salary 

information only. All libraries were guaranteed that they would not be identified with any 

particular set of data. 

Each benchmark partner was asked to complete the survey in four weeks. Some partners 

returned the survey in less time and others needed more time. In cases where benchmark 

partners had not submitted a completed survey within the allotted time, they were contacted to 

verify that they were still planning to complete the survey and to see if they needed any help 

with the survey. 

Incoming data were stored in a spreadsheet. Once all partners had returned their completed 

surveys, the team compared data from the benchmark partners to identify any area where it 

appeared that a partner had interpreted a question differently than other partners. From this, the 

team created a list of areas where it felt it needed further clarification from the benchmark 

partners. Many benchmark partners were contacted with follow-up questions. The entire 
4 

process, from making initial contact with potential benchmark process to finalizing the data, 

lasted about five months. 

 2.3 Understanding the Data 

Once the team determined that it received all the data and clarification it was likely to get, it 

began looking at the data more closely. Each team member was assigned a process or function 

addressed by the survey to analyze in greater detail. This analysis was shared with all team 

members. In addition, spreadsheet software was used to create charts and graphs that helped 

present visual comparisons of the data. 

All of this accumulated information was displayed on the walls of a conference room. Each 

team member was asked to review it and identify three areas where significant observations 

could be made about the data. Again, each member reported back to the group. From this 

exercise, the team was able to develop a list of topics it felt should be addressed in the team’s 

report to management. 

Due to the length and breadth of the survey, the team amassed a large quantity of data. The 

decision was made to report only on data that met two primary criteria. The first was that all 

benchmark partners reporting data on a specific question reported the data in a similar manner. 

For example, the survey asked benchmark libraries to indicate the size of their existing 

collections. Some libraries reported the number of journal volumes and others reported the 

number of journal titles. Some libraries included items in microform while others did not. 

Therefore, this report makes no comparisons based on collection size. The second criterion was 

that the data showed significant gaps that allowed ISD to identify best practices. 

Each member of the team was assigned a section of the report to write. At this time, an editor 

was brought into the process in order to give the finished product a single voice and to ask 

probing questions about the details of what was being reported. In addition, feedback from ISD 

management as well as Technology Services management helped to formulate a well supported 

analysis of the data. 
5 

3. Findings 

 

The benchmark study was conducted during the third quarter of FY2002 and was based on 

information from FY2001. This section of the report reflects the compilation, organization, 

analysis, and interpretation of the collected data. 

3.1 Overall Budget 

When looking at the Research Library’s operations as a whole, the benchmark team noticed two 

aspects about the way resources are allocated. The Research Library has the lowest percustomer expenditures, and its budget distribution heavily favors its collections over all other 

expenses. 

Of the libraries disclosing complete financial information (including materials or collections, 

salary, equipment and other costs) the Research Library spends the least per customer (see figure 

1). 

Figure 1. Average total expenditures per 

customer

$834.26 $841.87

$1,242.36

$2,525.47

$0.00

$500.00

$1,000.00

$1,500.00

$2,000.00

$2,500.00

$3,000.00

 NIST 

3,140

customers*

LIBRARY 3

3,180

customers

LIBRARY 6

4,500

customers

LIBRARY 5

1,500

customers

Libraries 1, 2, and 4 did not provide complete financial information. Library 7 is excluded because it 

counted its employees nationwide. This is because it operates nationwide. However, Library 7 is also 

supported by branch libraries whose budget information was not included in the survey. 

*This number is based on data supplied by the NIST Human Resources Management Division and 

includes 1456 ZP (professional pay band) employees, 83 post-docs, and 1600 guest researchers in 2001. 

The Research Library is spending far above the average percentage of its total allocation on 

materials, pulling from other areas of its budget to cover the Research Library’s collections 

costs. (The term “materials” refers to the library collections and includes journals, books, 

electronic resources, and document delivery. In addition, it includes the costs of maintaining 

these collections, i.e., binding and bibliographic utilities used for cataloging. Salaries, training, 

equipment, and supplies are not included.) The Research Library works hard to control 

expenditures in other areas in order to put more funds into library collections and other areas that 
6 

directly impact customers. In comparison to its benchmark partners, its other operational 

expenditures are low (see figure 2). 

Figure 2. Distribution of all expenditures

shown as percentage of total

63.51%

35.24%

1.12% 0.13%

40.58%

5.95%

13.40%

46.83%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Materials Salaries Equipment Other

NIST

Average

3.2 Materials 

Two elements of the Research Library’s collection stood out. The Research Library spent the 

largest percentage of its entire budget on materials of all its benchmark partners, and it purchased 

the fewest titles per customer. 

The Research Library spent the equivalent of 102 percent of its collection allocation on journals 

and databases. (Journals and databases have been combined here because the increasing number 

of databases that include full text articles makes it difficult to separate these costs.) 2 percent of 

its allocation was spent on books. The difference between the collections allocation and the 

actual amount spent was taken from other areas of the division’s budget. The average 

distribution of materials budgets among the benchmark partners was 89 percent for journals and 

databases, and 8 percent for books (see figure 3). 

In FY2001 the Research Library purchased the fewest resources per customer; on an average, 

less than 0.77 items per customer. (For a comparison to other benchmark partners, see figure 4.) 

 The average among benchmark partners was 5.24. 
7 

Figure 3. Percent of total materials 

budget spent on books and journals

90% 81% 88% 90% 82%

102%

2%

6% 9% 18%

2% 11%

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

LIB 5 LIB 6 LIB 7 LIB 3 LIB 1 NIST

Books

Journals and Databases

Libraries 2 and 4 did not provide any financial information. 

Figure 4. Resources purchased per customer

0.2 0.7 0.4 0.7 1.7 0.5

18.9

3.5

4.9 2.0

1.0 0.6 0.4 1.2

0

5

10

15

20

25

NIST LIB 1 LIB 6 LIB 3 LIB 5 LIB 4 LIB 2

All Journals and Databases

New Books

Library 7 is excluded because it counted its employees nationwide. This is because it operates nationwide. 

However, Library 7 is also supported by branch libraries whose budget information was not included in the 

survey. Library 2 has access to its parent organizations electronic resources at no or a minimal cost. 
8 

The low resource purchase per customer is partly due to the Research Library’s higher per 

journal costs. Its average cost per journal title in FY2001 was $1,137 while the average among 

benchmark partners reporting these costs was $772 (see figure 5). 

Figure 5. Average cost per journal 

LIBRARY 3 LIBRARY 6 LIBRARY 1 LIBRARY 7 NIST LIBRARY 5 

Average Cost 

per Journal $130.03 $502.37 $581.25 $800.00 $1,137.44 $1,481.19 

Libraries 2 and 4 did not provide any financial information. 

3.3 Document Delivery 

An overall view of the Research Library’s document delivery service reveals a highly productive 

staff working diligently to find free information resources. However, the Research Library 

appears to have higher costs than other libraries when it is unable to obtain free resources. While 

the data obtained for this study does not supply an explanation for this, some potential influential 

factors are considered below. 

The Research Library placed a greater than average number of requests for borrowing. If 

Library 1 is excluded, which uses a contracted service for processing requests, the Research 

Library placed a much greater than average number of requests to obtain materials from outside 

sources, with only one library placing more. Measured in terms of number of items lent and 

borrowed per staff member, the Research Library had the second highest productivity of the 

benchmark partners for which these data were available.

Although the Research Library placed a much greater than average number of requests, it paid 

fees for the fewest number of requests. The Research Library received 72 percent of its items at 

no cost, which is 20 percentage points higher than the next closest library. Because of this, of 

the five libraries reporting these data, the Research Library had below average per-item costs for 

borrowed items. 

When libraries did accrue costs for borrowing items, the Research Library’s average costs for the 

service were the highest of the five libraries that provided information (see figure 6). This, in 

part, is because libraries that charge back all document delivery costs to their customers paint a 

distorted picture of document delivery costs. Libraries showing the lowest costs or subsidies are 

charging all or most of these costs back to the customer and so they do not show up as library 

expenditures. 

Figure 6. Average per-item library subsidies for document delivery services 

LIB 5 LIB 1 LIB 3 LIB 7 LIB 6 NIST 

Average 

library subsidy 

when fee is 

paid 

$0.00 $0.33 $2.54 $20.00 $20.87 $21.73 

Charge back 

to customer Yes Yes Yes No Yes No 

Libraries 2 and 4 did not provide any financial information. 
9 

There are at least two other variables that might affect the cost per item. These include (1) the 

types of journals from which articles are requested; and (2) the sources being tapped for 

obtaining these articles. These variables are likely to prove to be interrelated. For example, 

articles from less specialized journals can be obtained from less expensive sources. 

The Research Library made significantly less than the average number of photocopies for 

document delivery inside the organization. Further study would be required to determine why 

the Research Library is receiving a relatively low number of requests for this service in 

comparison to its benchmark partners. 

The Research Library filled more outside requests than any of the six benchmark partners who 

were able to provide these data; five times the next highest number of outside document delivery 

requests filled.

The limited data collected in this study appear to support the argument that a relationship does 

exist between the number of items loaned to other libraries and a willingness generated in other 

libraries to supply items at no cost to the borrowing library. This is evident in data provided by 

the benchmark partners in that, out of the five libraries giving complete data on this, there is a 

clear correlation between the number of items supplied to other libraries and the number of items 

obtained at no charge (see figure 7). 

Figure 7. Relationship of items loaned

and items borrowed at no charge

0

1000

2000

3000

4000

5000

6000

LIBRARY

3

LIBRARY

5

LIBRARY

7

LIBRARY

1

NIST

Number of Free Items

Number of Requests

Filled for Other

Libraries

Libraries 2, 4 and 6 did not provide this data. 

Moreover, the OCLC4 interlibrary loan system itself, which is the Research Library’s primary 

means for retrieving documents, charges a $.40 fee per item borrowed and grants a $.40 credit 

 

4 OCLC is an international, nonprofit, library cooperative that currently serves 41,000 libraries.
10 

per item loaned. In addition, outside lending supports NIST’s role in working with U.S. 

industry. 
11 

4. Conclusion 

 An analysis of the benchmark data reveals best practices among the benchmark partners, areas 

that offer opportunities and challenges for improvement, and areas where the Research Library 

excels. 

4.1 Best Practices 

Within the data collected, two areas of best practices stood out. These relate to the recording and 

processing of customer feedback and the speed of document delivery services. 

Of the tracking mechanisms reported, one software product in particular stood out. This product 

is used for the following activities: capturing, routing, and tracking all incoming email 

questions; recording all "off-desk" interactions with customers including customer comments; 

and recording and tracking all questions received at the reference desk. This is a practice that 

the Research Library is currently considering emulating. 

The data revealed that one of the benchmark partners provides document delivery services with a 

very short turn-around time, i.e., in 11⁄2 days while the Research Library averages 5 days. The 

Research Library may benefit from a discussion with this library on how they are able to provide 

this efficient and cost effective service. (For a comparison of turn-around times for all 

benchmark partners, see figure 8.) 

Figure 8. Average turn-around time for document delivery services 

NIST LIB 1 LIB 2 LIB 3 LIB 5 LIB 6 LIB7 

Average turnaround time for 

books 

1 1/2 - 2 

weeks 1 week 2-3 weeks < 1 week 9.4 days 38 days 3-10 days 

Average turnaround time for 

photocopies 

3 - 7 days 1 week 5 days < 36 hrs 2.8 days 12 days 1-10 days 

Average turnaround time for 

requests filled 

electronically 

2 -3 days <1 day 2-3 days N/A 2.8 days N/A 1-2 days 

Data shown here appear exactly as reported by benchmark partners. “Turn-around time” is defined as the amount of 

time from date of request until the customer receives item. 

4.2 Opportunities for Improvement 

The Benchmark Team identified two areas where the Research Library could try to reduce its 

operational costs: journal costs and document delivery costs. 

The Research Library has among the highest costs per journal title (see figure 5), even though 

the subject focus of all the libraries studied is fairly homogeneous. Based on these data, the 

Research Library has explored opportunities for decreasing its costs per journal. 
12 

There are two ways to approach this issue. One is to look at the cost of individual titles. 

Recently, the Research Library asked its customers for input on titles published by one of the 

most expensive scientific publishers. Several title cuts resulted from that study. The Research 

Library has always considered the cost of a journal, among other things, when determining 

whether to start or stop a subscription5

. In support of this approach, the Research Library 

recently asked all NIST laboratories to compile a list of core journals in their fields. In this way, 

the Research Library continues to make studied decisions on keeping costs within budget while 

meeting the needs of its customers. 

A second place to look is purchasing titles as part of a consortium. The Research Library 

recently joined a consortium purchase agreement for titles published by Wiley InterScience. In 

FY2002, the Research Library gained access to 345 e-journals at an average cost of $442 per 

title. As a member of the National Research Library Alliance (NRLA), in FY2003 the Library 

reached similar agreements with the Institute of Physics and the American Chemical Society. 

The Research Library can benefit by finding ways to reduce its document delivery costs. While 

the Library paid for many fewer documents, when it did incur costs for document delivery, data 

indicate that its costs were higher than the other benchmark partners (see figure 6). 

The Research Library is currently collecting data which will allow it to conduct analysis in two 

areas: a cost benefit analysis of subscribing to a journal versus repeatedly requesting individual 

articles and; analysis of costs and service patterns of document suppliers currently being used. 

The Research Library is finding it considerably easier to do this with its recent purchase of 

document delivery management software, which greatly facilitates the tracking of these kinds of 

statistics. 

In addition, the Library has begun using a fee management service through OCLC. This service 

allows lenders to debit the Research Library’s account when they supply materials. Because 

suppliers prefer this method of payment, they often provide discounts for this arrangement. 

4.3 Where the Research Library Excels 

The Research Library performs well in areas of customer focus. This shows up in its 

communications with its customers and in the services it provides. The Library uses a variety of 

methods to solicit feedback from its customers, including customer surveys/questionnaires and 

focus groups. 

The Research Library is the only library of the eight benchmark partners that is using a Book 

Approval Plan. With this plan, the Research Library receives a bi-weekly shipment of recently 

published books and electronic lists of new titles that fall within its subject profile. These are 

made available to subject specialists within each NIST laboratory to review and make 

recommendations for purchase. The subject specialists appreciate this opportunity to be 

consistently involved in the Research Library’s development of its collection. 

 

5 In the Research Library, journal cancellation decisions are made based on usage statistics, the frequency with 

which a journal’s articles are cited, and formalized customer and library staff review. 
13 

The Research Library is adept at providing mechanisms for communicating information on 

library products and services. The Research Library communicates with its customers in the 

following ways: library news included in its Web site, training for specific resources, library 

tours, new employee orientation, announcements in agency publications, brochures, information 

sheets, and special events. The Research Library is one of three benchmark partners that publish 

a newsletter. This vehicle is used to educate customers about the Research Library’s products 

and services. 

The Research Library continues to fine-tune its interactions with its customers. The recently 

formed Research Library Advisory Board works with the Research Library staff to recommend 

directions for its collections and services consistent with current state and future trends in NIST 

research. Recent changes have also been made in the way the reference staff conducts business. 

Greater emphasis has been placed upon going out to individual laboratories instead of waiting at 

a reference desk for laboratory staff to come to the Research Library. 

 

The Research Library provides a broad range of services. Those services include: reference 

services (a staffed reference desk and an e-mail based reference service), online searching, a web 

site with research tools, interlibrary loan and document delivery services, customer training, 

historical archive maintenance for the NIST organization, and publishing support through its 

Research Consultant service. With a staff size of 16, the Research Library is able to achieve 

these services with the highest ratio of customers per staff member (see figure 9). 

Figure 9. Ratio of customers per staff member

48.39

100.00 102.04

122.35 132.35

196.19

0.00

50.00

100.00

150.00

200.00

250.00

LIBRARY

2

LIBRARY

4

LIBRARY

5

LIBRARY

3

LIBRARY

6

NIST

...

Library 1 does not appear here because its document delivery service is contracted out and is not 

included in the staff count. Library 7 is excluded because its staff count does not include branch 

libraries, but its customer base is counted nationwide. 

One of NIST’s stated organization goals is to “transform NIST into a learning organization that 

enables all people to do their best and aligns human resource management policies with mission 

and strategic intent.” The Research Library is already playing a critical role in facilitating the 

transformation of NIST into a learning organization. After benchmarking with comparable 

organizations, the Research Library now knows that, relative to its benchmarking partners, it 

excels in providing a broad range of learning resources. 
14 

4.4 Lessons Learned 

The team was advised not to use such a lengthy survey instrument because nobody would be 

willing to complete it. In fact, one library indicated it would not participate in part because of 

the length of the survey, and one library dropped out after receiving a copy of the survey. 

However, enough libraries were still willing to complete the survey. One library stated that it 

was being asked by its own management to benchmark and NIST was affording them that 

opportunity. 

The real issues with the length of the survey became apparent when the team began analyzing 

the data. It was difficult to determine just where the team should focus its attention because 

there was so much data. In hindsight, the benchmark team might have benefited from a more 

focused approach. The team might have gained a greater understanding of some of its data by 

asking more detailed questions about a narrower range of functions or processes. 

The team also made some errors in its use of more subjective questions. Data obtained from 

questions related to the adequacy of an institution’s budget were discarded because the team 

realized that these questions were not asked in a way that would produce quantifiable results. 

4.5 Next Steps 

The Benchmark Team collected data on a wide variety of library activities. Members of the 

Research Library have already begun to take a more in-depth look at the processes where the 

benchmark partners outperform the NIST Research Library. Some benchmark partners who 

seem to be setting the standard for providing superior information services to their customers 

have already been contacted in order to learn more about their processes. The goal is to emulate 

one or more of the best practices measured and make strides in offering improved research and 

information services to its customers. 

The Research Library strives to support the NIST organization in providing efficient and 

effective use of scientific and technological information, and in creating, managing and 

disseminating NIST's knowledge. Moreover, the Research Library expects to align its goals with 

NIST's 2010 Strategic objectives and begin to track customer data in order to define and 

understand customer needs, requirements, and expectations. The Information Services Division 

is a NIST asset which can contribute in valuable ways to NIST’s efforts to implement the 

elements of its 2010 Strategic Plan. 
15 

 APPENDIX A: BENCHMARK QUESTIONNAIRE 

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY RESEARCH LIBRARY

Benchmark Questionnaire 2002 

Answer the questions based on the most recent fiscal year that you have completed. 

For what year will you be completing this questionnaire? 

 to 

mm yy mm yy 

COLLECTIONS

Size and Scope 

1. Number of volumes1

 currently held 

2. Number of current journal subscriptions 

3. Number of other types of serial2 subscriptions 

4. Number of the above serials subscriptions that are received but not 

purchased (through gift or exchange programs) 

5. Gross number of books (monographs)3 added last year 

6. Gross number of the above monographs that were received but not 

purchased (e.g. through gift or exchange programs) 

7. Number of full-text electronic journal4 subscriptions 

8. Number of electronic reference sources5

9. Number of electronic books6

10. 

In what specific areas of the science/technology field does your organization specialize? 

(Check all that apply) 

 Biosciences 

 Biotechnology 

 Chemistry 

 Electronics 

 Engineering 

 Information Technology 

 Manufacturing 

 Materials Science 

 Mathematics 

 Medicine 

 Metrology 

 Physics 

 Other (Please Specify 
16 

Collection Development

11. Do you solicit input about your collection from your customers through the following ways? 

(Check all that apply)

 Regular review of books from a book approval plan 

 Submission of requests for purchases 

 A committee or board that regularly evaluates the collection 

 Other (Please Specify) 

 

12. On what do you base decisions to cut serial titles? (Check all that apply)

 Usage statistics 

 Formalized review process by library staff 

 Formalized review process by customers 

 Citation patterns or “impact factor”7

 Other (Please Specify) 

 

13. Does your library participate in any consortia or other types of partnerships? If so, please identify 

the reason(s) for these relationships. (Check all that apply) 

 To share resources 

 To procure resources at decreased costs 

 To benefit from the experience of others 

 Other (Please Specify) 

 

14. How would you characterize your partners? (Check all that apply)

 Academic institutions 

 Research institutions 

 Industry associations 

 Private companies 

 Government organizations) 

BUDGETS/EXPENDITURES

Budget

15. What was your total base budget8

 for library materials for your most recent 

fiscal year? 

16. What is the amount of any one-time funds9

 in your most recent fiscal year? 

17. What is the amount of endowed or special funds10 you received? 

 Yes No 

 Yes No 

18. Do you have a single source of funding for library materials, labor, and 

operations? If no, is the library director able to move funds from one source 

to another? 

19. Do you provide any services for which you charge the customer? If so, Yes No

• What is the dollar amount for expenses you recovered in your most 

recent fiscal year? 
17 

• Please complete the following table: 

Service Charges apply 

to internal 

customers11

Charges apply 

to external 

customers11

Fee structure or rate 

Interlibrary Loan 

Document delivery 

On-line searching 

Research consulting 

Other reference 

services 

 

Training 

Other (please specify) 

 

Adequacy Of Budget To Meet Customer Needs

 How much did your base budget8 20. increase/decrease from the previous year? 

 increase decrease 

unchanged 

21. Did you make cuts in your resources or services in any of the last three years? Yes No 

If so, please specify the general areas in which you made cuts (e.g. journal subscriptions, staff, etc.) 

Journals, Books 

 Are you continuing to offer new products and services? Yes No 

If yes, please specify which products or services and indicate how you paid for these new services 

(e.g. special, or one-time funding; budget increase; cuts in other areas, etc.) 

22. 

23. Do you have sufficient funding to enable exploiting new technologies? Yes No 

 

24. Do you feel you are able to acquire the most significant resources in your fields Yes No 

of focus? 
18 

Expenditures

Please indicate the approximate amount spent for each category below for the most recent fiscal year: 

25. Monographs (one-time purchases) 

• Print 

• Electronic12

26. Serials (continuing expenses) 

• Electronic journals13

• Accessed through direct licenses 

• Accessed through consortia agreements 

• Print journals 

• Other serial2

 subscriptions 

• Print 

• Electronic 

27. Electronic reference resources14

28. Binding 

29. Bibliographic utilities (e.g. OCLC, RLIN, etc.) 

30. Fees paid to interlibrary loan/document delivery suppliers (include copyright 

fees) 

31. Employee salaries (do not include contractors) 

 Does any of your computer support come from within your library or 

information center? 

32. 

 If so, have you included their salaries in your answer to question 30? 

33. What are your annual costs (or percentage of total budget) associated with 

the purchase, lease, and maintenance of your library’s hardware including, 

computers, copiers, fax machines, microfilm/fiche reader/printers, VCRs, CDRom, DVD players, TVs? (Estimates are acceptable.) 

34. All other expenses 

INTERLIBRARY LOAN/DOCUMENT DELIVERY

 Do you use any software products (other than a bibliographic utility, such as Yes No 

OCLC) for tracking ILL requests 

35. 

 If so, please specify which product or products 

36. Do you use any software products for maintaining statistics Yes No 

 If so, please specify which product or products 
19 

 Do you use any other software products for interlibrary loan/document delivery Yes No 

processes? 

37. 

 If so, please specify which product or products 

 Do you provide a photocopy service in which your library staff copies materials Yes No 

held within your collection for members of your organization?

38. 

 If so, how many requests were filled during the past fiscal year? 

39. What is the size of your interlibrary loan/document delivery staff?15

Receiving From Outside Sources16

40. Number of requests your center made for loans or photocopies during your 

most recent fiscal year. 

41. What percentage of your requests were for photocopies (as opposed to 

loans)? Estimates are acceptable. 

42. 4 Number of requests for which a fee was paid to the supplier during your 

most recent fiscal year. 

43. 4 Please list the top five of your most often used document delivery vendors. 

44. Number of requests filled electronically (e.g., Ariel Fax Service, e-mail). 

45. 4

4

 What is the average turnaround time (from date of request until customer 

receives item) for borrowed items? 

46. 4 What is the average turnaround time (from date of request until customer 

receives item) for photocopied items? 

47. 4 What is the average turnaround time (from date of request until customer 

receives item) for requests filled electronically? 

Lending To Other Libraries

48. Number of requests for interlibrary loans received from other libraries during 

your most recent fiscal year (include loans and photocopies). 

49. Number of requests for interlibrary loans that were actually filled during your 

most recent fiscal year (include loans and photocopies). 

50. What percentage of these requests were for photocopies? Estimates are 

acceptable. 

51. What percentage of requests were filled electronically (e.g., Ariel Fax 

Service, e-mail)? Estimates are acceptable. 
20 

STAFFING15 

52. Number of professional librarians17

53. Number of other professional staff18

54. Number of support staff (e.g. library technicians, secretaries, library aides) 

55. What is the number of contractors included in the above staff count? 

56. How many FTE’s15 are committed to the administration and/or maintenance 

of computerized resources19? 

CUSTOMERS

57. How many employees work at your institution or organization?

58. What is the size of your target audience or customer base?

 Who do you consider your primary customers/target audience? (Check all that apply to your 

primary customers): 

 Researchers 

 Educators 

 Administrators 

 Support Staff 

59. 

 External Customers11

Do you provide services to people outside of your institution? Yes No 

If so, are your external customers from (please check all that apply): 

 Educational institutions 

 Private companies 

 Industry associations 

 Research institutions 

 Government organizations 

60. 

 General public 

61. Is your library open to the walk-in public? Yes No

SERVICES

62. How many hours per week is your library accessible? 

63. How many hours per week is your library staffed? 
21 

 Please check all services that your library offers: 

Service

Offered 

internally 

Offered 

externally 

Reference service 

Staffed reference desk 

Email-based reference service 

On-line Searching 

Alerting service 

Research Consulting20 

On-line catalog 

Library maintained Web site with research tools 

Interlibrary Loan/Document Delivery 

Photocopying Service 

Training 

Maintaining an historical archive for your organization or institution 

Other (please specify) 

 

 

 

 

 

 

64.

 

65. How many staff hours per week are dedicated to reference services21? 

Which of the following mechanisms do you use for communicating services to your customers 

(Check all that apply): 

 Training for specific resources

 Library Tours

 New employee orientation programs

 Newsletter

 Announcements in company or agency publications

 Brochures 

 Information sheets 

 Special events (Open House, speakers, etc.)

 Others (please specify) 

 

 66. 

 
22 

67.Do you use any of the following tools to evaluate your services: 

 Written customer surveys/questionnaires 

 Focus groups 

 Advisory boards 

 Other (please specify) 

 

 

 Do you have a formal system for tracking of customer comments Yes No

 If so, please describe briefly. 

68.

69. What methods do you use to evaluate customer satisfaction? 

 

1

 A single physical unit of any printed, typewritten, handwritten, mimeographed, or processed work, 

distinguished from other units by a separate binding, encasement, portfolio, or other clear distinction, 

which has been cataloged, classified, and made ready for use, and which is typically the unit used to 

charge circulation transactions. (source: ANSI/NISO Z39.7-1995) 

2

 A publication in any medium issued in successive parts bearing numerical or chronological designations 

and intended to be continued indefinitely (source: ANSI/NISO Z39.7-1995) 

3

 A nonserial printed publication of any length bound in hard or soft covers or in looseleaf format (source: 

ANSI/NISO Z39.7-1995) 

4 This includes electronic full-text journals offered by established scholarly journal publishing houses (e.g., 

Elsevier’s ScienceDirect and Academic Press’s IDEAL), scholarly societies (e.g., American Chemical 

Society journals and American Institute of Physics Online), and services which aggregate content from 

smaller publishers or from those publishers that prefer to use an external delivery platform (Highwire, 

OCLC ECO, and EbscoOnline). 

Special considerations: Include journal titles that come with print subscriptions or print plus online 

subscriptions since the focus of the statistic has to do with how many scholarly electronic journal titles 

users can access. Do not include free government publications and free electronic journals to which the 

library provides links. Also exclude general-purpose periodicals such as magazines and newspapers. 
23 

 

(source: adapted from Measures And Statistics For Research Library Networked Services: Procedures 

And Issues. Arl E-Metrics Phase Ii Report, issued October 2001. 

<http://www.arl.org/stats/newmeas/emetrics/phasetwo.pdf>) 

 

5

 This includes citation indexes and abstracts; full-text reference sources (e.g. encyclopedias, almanacs, 

biographical and statistical sources, and other quick fact-finding sources); full-text journal and periodical 

article collection services (e.g., EBSCOhost, ProQuest, Academic Universe, and INFOTRAC OneFile); 

dissertation and conference proceedings databases; and general-purpose magazines and newspapers. 

Licensed electronic resources also include those databases that institutions mount locally. 

Special considerations: The unit of measurement here is the database not the whole service provided by 

a vendor. For example, if the library subscribes to OVID and the company provides five databases 

(ABI/Inform, Books in Print, CINAHL, INSPEC, and PsycINFO), then the count is 5, not 1. By the same 

token, if the library subscribes to three database packages (Academic Universe, Congressional Universe, 

and Statistical Universe) from Lexis-Nexis, the count is 3. This count should not include freely available 

databases to which the library provides links or library-created finding aids. 

(source: Measures And Statistics For Research Library Networked Services: Procedures And Issues. 

Arl E-Metrics Phase Ii Report, issued October 2001. 

<http://www.arl.org/stats/newmeas/emetrics/phasetwo.pdf>) 

6

 Number of electronic full-text monographs that the library offers to its users either through an individual 

licensing contract with the content providers or through other arrangements (e.g., regional or state 

consortium) where the library pays a reduced or no fee for access. This includes electronic books 

purchased through vendors, such as netLibrary and Books24x7, and electronic books that come as part of 

aggregate services. It excludes internally digitized electronic books, electronic theses and dissertations, 

digitally created archival collections (e.g., Early English Books Online), and other special collections. This 

also excludes publicly available electronic books to which the library provides web links. It does not 

include machine-readable books distributed on CD-ROM, or accompanied by print books. 

Special considerations: Do not include book collections that are a part of aggregate services and 

function more as a reference collection (e.g., MD Consult reference books, ProQuest’s Early English 

Books Online, and books@OVID). They should be reported in the electronic reference databases. Do not 

include freely available electronic books such as titles available from the National Academy Press.

(source: Measures And Statistics For Research Library Networked Services: Procedures And Issues. 

Arl E-Metrics Phase Ii Report, issued October 2001. 

<http://www.arl.org/stats/newmeas/emetrics/phasetwo.pdf>) 

7 Impact Factor is a formula developed by the Institute for Scientific Information to measure the frequency 

with which the "average article" in a journal has been cited in a particular year or period. The impact factor

discounts the advantage of large journals over small ones or frequently issued journals over less 

frequently issued ones, and of older journals over newer ones. 

8

 This is also referred to as permanent budget. It is usually the annual (sometimes biannual) but ongoing 

allocation for library materials. In government institutions, this is normally the primary fund allocation, 

often a line-item in the budget. In private institutions, the base budget may come from permanent 

endowments that are designated to the library materials budget or other stable revenue resources. The 

base budget represents an agreed upon figure between an administration and the library that is the basis 

for long-range budget planning. (source: adapted from CCDO Library Materials Budget Survey, 

“Definitions,” Prepared by Robert G. Sewell < http://www.arl.org/scomm/lmbs/lmbsdef.html>) 

9

 This refers to money that supplements the base budget and is available normally only for one year. 

These temporary funds are usually designated for a specific purpose such as to build up an area of the 
24 

 

collection or for the purchase of an expensive item. (source: CCDO Library Materials Budget Survey, 

“Definitions,” Prepared by Robert G. Sewell < http://www.arl.org/scomm/lmbs/lmbsdef.html>) 

10 Include funds provided for a specific purpose which are considered supplementary to your base budget 

and are not included in your one-time funds. 

11 Consider internal customers any patrons doing work for your institution, including staff, contractors, 

guest researchers, etc. External customers are patrons using your resources that do not work for your 

institution. 

12 Expenditures for electronic full-text monographs that the library offers to its users. Include both initial 

purchase costs and membership fees as well as annual access and service fees paid directly or through 

consortia arrangements. 

Special considerations: Traditionally books are purchased on a one-time payment in exchange for 

permanent ownership by the library. However, with regard to electronic books, it appears that some 

arrangements allow libraries to subscribe to an e-book collection at a predetermined fee and for a 

predetermined interval of time. We are concerned with the format of the material, not the subscription or 

payment arrangement. These materials should be counted as books, not serial publications. 

(source: Measures And Statistics For Research Library Networked Services: Procedures And Issues. 

Arl E-Metrics Phase Ii Report, issued October 2001. 

<http://www.arl.org/stats/newmeas/emetrics/phasetwo.pdf>) 

13 Expenditures for electronic full-text journal subscriptions that the library provides to its users. Include 

both initial purchase cost, membership fees (such as JSTOR) as well as annual access and service fees 

paid directly or through consortia arrangements. 

Special considerations: Some electronic full-text journals come either as a free service with a print 

subscription or as part of a print-plus-online-access subscription (the library pays extra for electronic 

access). Only include costs which are above and beyond costs paid for print copy. In cases where the 

cost breakdown is not clear, provide the best approximation. 

(source: adapted from Measures And Statistics For Research Library Networked Services: Procedures 

And Issues. Arl E-Metrics Phase Ii Report, issued October 2001. 

<http://www.arl.org/stats/newmeas/emetrics/phasetwo.pdf>) 

14 Expenditures for electronic reference sources and aggregate services that the library provides to users 

either through individual licensing contracts with content providers or through consortia or other 

arrangements where the library pays some fees. These fees include both annual access fees and other 

service costs paid to the vendor directly or through consortial arrangements. 

(source: Measures And Statistics For Research Library Networked Services: Procedures And Issues. 

Arl E-Metrics Phase Ii Report, issued October 2001. 

<http://www.arl.org/stats/newmeas/emetrics/phasetwo.pdf>) 

15 Report the number of staff in filled positions (including contract employees), or positions that are only 

temporarily vacant. Please count FTE’s (or Full Time Equivalents). To compute full-time equivalents of 

part-time employees, take the total number of hours per week (or year) worked by part-time employees 

and divide it by the number of hours considered by the reporting library to be a full-time work week (or 

year). Round figures to the nearest whole numbers. 

16 In this section, include only the number of requests you were actually able to fill, if this number is 

available. If you only have numbers for the amount of requests placed, please indicate that with a note. 
25 

 17 Professional librarian is defined as a person having a Master's in Library and Information Science from a 

graduate program accredited by the American Library Association. 

18 The criteria for determining professional status may vary among libraries. Each library should report 

those staff members it considers professional, including, when appropriate, staff who are not librarians in 

the strict sense of the term, for example computer experts, systems analysts, or budget officers. 

19 Include system administrators, Web masters, etc. Do not include staff responsible for content only. 

20 Research consultation is in-depth reference and research assistance that goes beyond ready-reference 

service and on-line literature searching. It suggests some further processing of the information collected. 

21 If some of the reference services staff divide their time with other duties, please approximate the 

number of FTE’s committed solely to reference duties. 
26 

APPENDIX B: TALKING POINTS FOR CONTACTING LIBRARIES 

Introduction

We are calling from the National Institute of Standards and Technology (NIST) library. NIST is 

an agency in the U.S. Department of Commerce that works with industry to develop and apply 

technology, measurement, and standards. 

The NIST library has collections in the physical sciences and engineering. In order to 

justify a request for a budget increase, the NIST library has developed a benchmarking survey to 

compare its services with those of other comparable libraries in government and the private 

sector. Because of your facility’s size and the types of collections that you hold, your library is 

on a list that we have compiled of about 15 potential benchmarking partners. Make it clear that 

this has nothing to do with A76 studies.

Results will be made available to study participants

Any reporting of results will be anonymous 

We will identify which libraries participated in the study, but we won't attribute any 

answers/data to a library. We will share the report with those libraries that participate in the 

study. We plan to share the report with NIST management. Only ISD staff members working on 

the benchmark study will see an individual library's data. 

We may report on the study at conferences, etc., but we won't attribute any response to any of 

the participating libraries. If a library is willing to participate but doesn't want to be listed as a 

participant, that's ok, too. 

Why their library was selected

We looked for partners that were similar to us: similar research areas, size (of collection or staff). 

We will assist them any way we can

• If they would like, we'll step them through the questionnaire over the phone. 

• If they are unclear anytime along the way, encourage them to call us. 

• We will give them a reminder call or email one week before it's due. 

• They are welcome to talk with anybody on the team, the head of the Research Library, or the 

director of ISD.