Evaluating Equity and Inclusion in
Cultural Heritage Grantmaking:
CLIR’s Amplifying Unheard
Voices Program
Jesse A. Johnston
Ricardo L. Punzalan
February 2023

Council on Library and Information Resources

Evaluating Equity and Inclusion in
Cultural Heritage Grantmaking:
CLIR’s Amplifying Unheard
Voices Program
Jesse A. Johnston
Ricardo L. Punzalan

February 2023

Copyright 2023 by Council on Library and Information Resources

This work is licensed under a Creative Commons
Attribution-NonCommercial-ShareAlike 4.0 International License

Published byCouncil on Library and Information Resources
1800 Diagonal Rd., Suite 600
Alexandria, VA 22314

Contents
Foreword ....................................................................................................
Executive Summary ...................................................................................
1. Introduction ..........................................................................................

1
4
7

1.1 Background ...........................................................................................................

7

1.2 Scope of This Report..............................................................................................

8

2. Methods .................................................................................................

9

2.1 Data Sources .......................................................................................................... 10
2.2 Semi-Structured Interviews ............................................................................... 11
2.2.1 Interviewee Selection ........................................................................................................................ 12
2.3 Focus Groups ........................................................................................................ 13
2.4 Limitations ............................................................................................................ 14

3. Findings and Discussion ....................................................................... 15
3.1 Applications Received and Reviewed ................................................................
3.1.1 Invitation, Funding, and Award Rates .............................................................................
3.1.2 DHC Program Rates over Time .............................................................................................
3.1.3 Assessing Broad Representation ............................................................................................
3.1.4 Material and Institution Types ................................................................................................
3.1.5 Unheard Voice Groups ....................................................................................................................
3.1.6 Potential Representation Gaps in Applications Received ...........................

15
15
16
17
18
19
20

3.2 Program Appeal ................................................................................................... 22
3.2.1 Appeal for Canadian Applicants ........................................................................................... 24
3.2.2 Appeal to Reviewers ........................................................................................................................... 25
3.3 Perspectives of Potential Applicants .................................................................
3.3.1 Non-Applicant Survey .....................................................................................................................
3.3.2 Notable Non-Applicant Survey Findings .....................................................................
3.3.3 Withdrawn Applications ...............................................................................................................

26
27
27
33

3.4 Perspectives of Applicants ..................................................................................
3.4.1 Program Values ........................................................................................................................................
3.4.2 Program Resources for Applicants .....................................................................................
3.4.3 Application System ..............................................................................................................................
3.4.4 Application Structure ........................................................................................................................
3.4.5 Feedback to Applicants ...................................................................................................................

34
34
36
37
38
39

3.5 Perspectives of Reviewers ...................................................................................
3.5.1 Interpretation of Program Values ........................................................................................
3.5.2 Review Processes and Support for Reviewers .........................................................
3.5.3 Scoring the Rubric ...............................................................................................................................

42
43
44
45

3.6 Direct Applicant Support ................................................................................... 46

4. Areas for Attention and Recommendations ........................................ 49
4.1 Allowed Activities ................................................................................................ 49
4.1.1 Support for Reparative Description .................................................................................. 49
4.1.2 Identifying and Evaluating Collection Strengths ................................................. 50

4.2 Applicant Support ................................................................................................ 51
4.2.1 Increase Direct Applicant Support ..................................................................................... 51
4.2.2 Make Information Available Earlier .................................................................................. 52
4.2.3 Information for Less-Frequent Grant Seekers ........................................................ 53
4.3 Application Process ............................................................................................. 54
4.3.1 Shorten Initial Application .......................................................................................................... 55
4.4 Review Process .....................................................................................................
4.4.1 Panel Process ..............................................................................................................................................
4.4.2 Panel Membership ...............................................................................................................................
4.4.3 Specific Feedback Areas .................................................................................................................

56
56
59
59

4.5 Award Process ......................................................................................................
4.5.1 Intellectual Property and Ethical Access ......................................................................

60
60

4.6 Program Values and Voice Groups ................................................................... 62
4.7 Program Administration .................................................................................... 63

5. Conclusion: Successes and Challenges ................................................ 65
References ................................................................................................... 68
Appendix A: Implicit Bias Orientation ....................................................
Appendix B: Review Rubic for Full Applications ....................................
Appendix C: Principles for Useful Feedback ...........................................
Appendix D: Comparison of Initial and Final Application Elements .........

72
75
78
80

Acknowledgments ...................................................................................... 83
About the Authors ..................................................................................... 84

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Foreword
Through the generous support of the Mellon Foundation, CLIR has
had the privilege of working consistently on regranting initiatives
focused on creating access to rare and unique materials since 2008.
With the launch of Cataloging Hidden Special Collections and Archives that year, followed by Digitizing Hidden Special Collections
and Archives in 2015, and Recordings at Risk in 2017, the organization took the first steps on a journey that so far has led to 23 calls for
proposals resulting in 391 funded projects. In total, these projects
reflect an investment of nearly $60 million.
Creating grant opportunities for collecting organizations anddepartments—especially for those who safeguard the world’s historical, cultural, and scientific heritage while under severe financial
constraints—can be deeply satisfying. Unfortunately, for every congratulatory email CLIR has sent to the recipients our reviewers havenominated for funding, CLIR has had to send many messages ofregret. CLIR has received more than 2,100 requests for funding since2008. The time, energy, and vision applicants have dedicated to making these requests is almost unfathomable. CLIR’s responsibility to its
communities of applicants is something that weighs heavily on theminds of program officers.
This long-held responsibility is what makes this report and theresearch it describes so vitally important to CLIR as an organizationand to any future regranting initiatives it pursues. In 2021, when
CLIR launched a new iteration of Digitizing Hidden Collections,
Amplifying Unheard Voices, researchers Jesse Johnston and Ricardo“Ricky” Punzalan joined CLIR’s team to offer their expertise and a
new perspective on how staff was adapting previous practices to train
the program’s focus on creating access to resources that document
historically marginalized people. This central aspiration serves CLIR’s
longstanding mission to advance knowledge and the public good byempowering grant recipients to contribute to a more equitable andsocially just digital learning environment for everyone.
Input from Johnston and Punzalan has been vital to our staff
throughout the first review cycle for Amplifying Unheard Voices, andit will be foundational for shaping CLIR’s future work in regranting.
The details of their analysis are specific to one program at a particular
moment in time. However, for this public report, they have endeavored to organize their observations and reflections in a way that canbe broadly useful to a variety of readers:• Funding officers who are interested in supporting collecting
organizations through philanthropy. The overview and discussion of program operations in section 3 will be helpful forunderstanding the needs of libraries, archives, museums, andother organizations that collect and share unique cultural heritage. This information will also be useful for anticipating potential challenges and obstacles that may surface when these1Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

•

•

•
•

organizations apply for and manage project-based grants. The
recommendations outlined in sections 4 and 5 suggest ways
for strengthening programmatic transparency and support
strategies for reviewers, applicants, and grantees. Appendixes
A–C can be adapted and repurposed for use in similar grantmaking initiatives.
Grant reviewers who are interested in offering constructive
feedback. Section 3 discusses the experiences of CLIR’s reviewers as well as applicants’ responses to reviewer feedback.
These discussions can help reviewers reflect upon the significance of their role in the grantmaking process and anticipate
questions and challenges that might arise. Recommendations
for improving the review process provided in section 4 will
help potential reviewers formulate questions about what afunding program expects from them and suggest ways program staff can offer guidance and support. Appendixes A–C
can also serve as useful tools for reflection and preparation,
especially for those newer to reviewer service.
Grant applicants who are interested in the mechanics ofopen-call programs. Many people who work with collectionsin the academic and cultural sectors find the experience of applying for grants confusing and frustrating. While this report
describes just one program, reading about the experiences of
CLIR’s applicants and reviewers in their own words should
help aspiring applicants recognize that they are not alone infinding the grant-seeking process challenging and that, as key
constituents in any program aligned with their needs, theirown suggestions and feedback can lead to better grant-seeking
experiences for others.
Researchers planning program evaluations. The overview ofthe project methodology provided in section 2 can be usefulto anyone pursuing similar assessment projects.
Any of CLIR’s applicants or recipients who are interested
in the wider context of their work. Participants in the first
iteration of Amplifying Unheard Voices will find this report
to be an opportunity to reflect upon their own experience andsee how their experience compares with that of others. The
authors’ suggestions for improvements to CLIR’s processes
provided in sections 4 and 5 are indicators of potential future
growth, particularly as CLIR seeks to strengthen support for
applicants from smaller and under-resourced organizations.

Additional information about Digitizing Hidden Collections:
Amplifying Unheard Voices can be found on CLIR’s website.1 The
program and its assessment were generously supported by the Mellon
Foundation. Former CLIR program officer Joy Banks offered guidance to the research team through most of the evaluation project.
Along with Banks, former program officer Becca Quon and current
program officers Sharon Burney and Alyson Pope worked tirelessly

1 https://www.clir.org/hiddencollections/2

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

throughout the study to make their visions for Amplifying Unheard
Voices a reality for the communities the program serves. Their sharedconviction that CLIR can always do more, and do better, continues tofuel planning for the organization’s future.
		—Christa Williford
Senior Director of Research and Assessment, CLIR

3

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Executive Summary
This report summarizes a yearlong program assessment of “Amplifying Unheard Voices,” a major revision of the Digitizing Hidden Collections grant program (DHC:AUV). The program is administered by
the Council on Library and Information Resources (CLIR). Throughthe 2021 program revision, CLIR aimed to expand the reach andappeal of the program to a broader range of institutions, including independent and community-based organizations and to emphasize thedigitization of historical materials that tell the stories of groups underrepresented in the digital historical record. The program revision
implemented significant changes to the application structure, created
new applicant support resources, expanded eligibility to Canadianapplicants, and added new thematic emphases and stated program
values.
The program assessment was based on a series of qualitative evaluation activities. These activities were designed and implemented bya team of two external consultants who worked with CLIR staff fromMay 2021 until June 2022. In addition to CLIR program staff, theconsultants worked with DHC:AUV stakeholder groups, including
reviewers, applicants, recipients, and interested cultural heritage organizations. These combined perspectives offered a holistic view of the
program, including:• Content analysis of program and application materials• Forty-seven semi-structured interviews with initial-stage applicants, full-stage applicants, grant recipients, and program
reviewers• Survey responses from 56 non-applicants, who expressed interest in the program but did not apply as well as applicants
who were invited to submit full applications but did not• Observations from two program review meetings conducted
via videoconference• Additional materials shared by CLIR staff, including applicant
emails, feedback surveys, and applicant support materials
Highlighted Findings

Enthusiasm for the program is high. The changes in the 2021
DHC:AUV iteration were warmly received by many potential applicants, including organizations that are not frequent grant seekers forcollections-related activities as well as many organizations that havepreviously applied to DHC. The revised program was recognized asa critical funding resource unique in its newly articulated support
for collections digitization in conjunction with social justice priorities. These interests are clearly expressed in the program values andpositively benefit the preservation of and access to digital collections

4

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

that contribute to a more broadly representative historical record.
CLIR’s resource materials for applicants were praised highly for theirclarity, comprehensiveness, and approachability and for being readily usable and accessible. The expanded membership of the review
panel represented expertise well-suited to evaluate the new group of
applicants and proposals received. Overall, program accessibility, theappeal of the call for proposals that emphasized underrepresented
perspectives in collections, and the continuing support for digitization were welcomed and well received. Even among those interested
in the program but who elected not to submit applications, more thanhalf hoped to submit applications in future competitions if given theoption.
Areas for Attention and Recommendations

Alongside these positive elements, we identified areas in which theprogram would benefit from further attention as it moves ahead:• Allowed activities. While the current focus on digitization
was popular, stakeholders also noted that DHC:AUV should
consider designating support for reparative description or redescription of collections.
• Applicant support. The current applicant support mechanisms
rely on direct email support and webinars; these are appreciated, but more tailored, one-on-one direct support is needed.
• Review process. We identified specific areas of need, including public library expertise; guidance materials for reviewers;
and direct, specific items for actionable feedback.
• Award process. This part of the program was generally effective, but we raise notable concerns about DHC:AUV’s approach to intellectual property and collection ownership. We
recommend that the program move toward reciprocal notionsof ownership and access that respect community sovereignty
and expertise.
• Program “voice groups.” If DHC:AUV aims to serve underrepresented narratives in the digital historical record, we identify community narratives, or “voices,” that appear to be ofparticular value to the program as it is currently implemented.
• Program administration. To the extent possible, we recommend that DHC:AUV explore the option of creating an additional program officer or a program manager role—someone
who can steer the review process and offer increased direct
support to applicants.
We conclude the assessment with optimism about the program’s
possibilities but also with an awareness of the significant work required to maintain and improve such funding programs. We note thehigh enthusiasm for increased support of community-based memory
initiatives that will diversify the historical record and make that record more digitally available. At the same time, the assessment reveals
challenges of funding digitization projects in cultural heritage: thesignificant time required for design, implementation, and manage-5

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

ment of multiyear programs; the limitations of project grants; andthe challenges of making incremental yet responsive changes withina longstanding program. This assessment is a snapshot of the initial
implementation of DHC:AUV, and there may be further impacts of
the revisions that may not be apparent for a few years. This programassessment revealed enthusiasm for and potential of the future of
DHC:AUV, but more broadly, we see further potential to increase equity in funding programs and representation of community stories inthe digital historical record.

6Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

1.

Introduction
This report shares findings and recommendations from the assessment of “Amplifying Unheard Voices,” the first iteration of a majorrevision of the Digitizing Hidden Special Collections and Archives(DHC) grant program. The assessment’s primary goal was to help theCouncil on Library and Information Resources (CLIR) program staff,
reviewers, and other stakeholders to understand what is working in
the revised program and to consider areas of program improvement.
The report is designed to be useful to multiple audiences. Beyond
CLIR, this report will be of interest to funders working in the cultural
heritage space, program managers working to construct equitable
review processes, those seeking grant funding, and anyone designinga qualitative program assessment. For grant seekers, we suspect that
this will be of particular interest to those seeking support for workwith archival, library, and museum collections, but also those looking
for support to work with community-based collections. Finally, we
hope the report benefits:• Other funders who are providing financial support for cultural
heritage activities• Those interested in applying for grants, whether working in libraries, archives, and museums or with community-based organizations, nonprofits, or others who support memory work
• Research administrators and development professionals, consultants, or volunteers at cultural heritage organizations

1.1

BackgroundIn early 2021, CLIR announced that DHC would continue to offer
grants supporting the digitization of rare and unique content in cultural heritage institutions, with financial support from the Mellon
Foundation. The new iteration of the program, Digitizing Hidden
Special Collections and Archives: Amplifying Unheard Voices (DHC:AUV), emphasized support for the digitization of collections that“deepen public understanding of the histories of people of color andother communities and populations whose work, experiences, andperspectives have been insufficiently recognized or unattended”(CLIR 2021a).
Through DHC:AUV, CLIR aimed to “fund a cohort of academic, independent, and community-based organizations in the United States and Canada to digitize now-unavailable or underutilized
collections with the potential to broaden the range of racial, ethnic,
and cultural representation in digital libraries and archives” (CLIR2021b). Notable changes from the prior DHC program included:• An expressed thematic emphasis on “unheard voices” through“collections documenting the hidden histories of people whohave previously been under-examined or unknown to broader
audiences”

7

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking• Expanded eligibility to Canadian nonprofit institutions• A shortened initial application (CLIR 2021a)
Concurrent with the program revision, CLIR commissioned an
external assessment to evaluate the program implementation andassess program clarity, transparency of operations, and applicant support. This report presents the findings and recommendations of the
program assessment.

1.2

Scope of This Report

This report assesses DHC:AUV program activities from May 2021
through June 2022. Although our analysis included some materials
documenting pre-application support, our formal activities did not
begin until after the initial applications were received by CLIR. Therefore, the bulk of data and findings for this report are based on information gathered after initial applications were received. We evaluatedactivity in three main program phases:• Initial phase, which included webinars for interested applicantsprior to application submission; applicant support; submission,
processing, and review of initial proposals; and invitations to submit full applications• Full application phase, which included the communication of
the initial panel review to applicants; a series of webinars for invited applicants; and the submission, processing, and review offull proposals• Award phase, which included the notification of awards, processing of final award documents, and beginning of funded projects
Throughout these phases, we analyzed the initial and full proposals received, queried selected applicants and reviewers to learn about
their experiences and received their input, observed panel review
meetings, and met with CLIR staff.
We focused our activities on assessment of the DHC:AUV program implementation and recommendations for future programmodifications. We specifically examined the program’s language andguidelines, interpretation of these program materials by applicantsand reviewers, applicant support services, and reviewer engagement.
In our recommendations, we identified actions to increase the equitability and transparency of the program, modify program structures,and increase clarity1for participants in potential future grant cycles.
We gave particular attention to participants based outside large institutional libraries or archives who may lack significant experience in
grant seeking.

1

8

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

2. Methods
The assessment project undertook a series of qualitative evaluation
activities to analyze the DHC:AUV implementation, including review
of data received from CLIR as well as data gathered by the assessment team. These activities combined perspectives from program
stakeholders and represented staff, reviewers, applicants, and those
who expressed interest in the program. Data consulted or gathered
included program handbooks and guidelines, application documents,
the list of applications received, survey responses, semi-structured interviews with 59 program stakeholders, and feedback from four focus
groups.
We used a range of methods to gather program information,
including surveys and content analysis, but we emphasized qualitative interviews and focus groups in our assessment approach. Weemphasized qualitative methods because these offered the best way
to understand applicant experiences in the program. We preferred interviews over a broad survey approach because we were able to selectinterviewees according to specific DHC:AUV priorities, particularly
institution type and region.2 In addition, we created and distributed
a survey to “non-applicants,” that is, those who had voiced interest in
applying but did not ultimately submit an application. We also conducted a survey of applicants who withdrew from the full application
phase. While we developed each of these assessment activities in consultation with CLIR, we primarily worked independently and sharedinsights only at specified times.
Our guiding questions included:
1. What does the breadth of material formats and topics represented among the letters of interest indicate about the level ofdemand for this program?
2. Are there obvious gaps among the range of topics, material
types, geographic regions, or institution types represented
among the letters of interest?
3. How do the outcomes of the competition compare with previous iterations of CLIR’s program in terms of the overall funding rate; the breadth of topics, material formats, geographic
regions, and institution types represented in the pool; and thebreadth of topics, material formats, geographic regions, andinstitution types represented among awardees?
4. Are the values and criteria for assessment clear to potential
applicants, and do they receive enough support in developing
their proposals?
2 The DHC program has seen a plurality of applications and awards from libraries,
archives, and museums based at academic institutions (Banks 2019).

9

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

5. Are the values and criteria for assessment clear to program
reviewers, and do they receive enough support in evaluating
proposals?
6. What changes or improvements can CLIR make to the framing of the call, to the program guidelines, to allowable anddisallowed costs, to applicant communications and support,
to reviewer communications and support, or to the programwebsite to ensure a satisfactory experience for future participants?
We actively sought feedback from program stakeholders to answer these questions. To develop our findings (see section 3), we used
contemporaneous notes of observations, interview transcripts, andopen-ended answers provided in feedback surveys. We also reviewed
program application forms and the application system. We used thequalitative analysis tool Dedoose to tag, group, and annotate data
(Dedoose 2022). We also met regularly with CLIR staff to learn aboutprogram developments and share our findings.

2.1

Data Sources

The report’s findings and recommendations are based on multiple
data sources. Primary data sources included the following:• Materials available through the program website, including documentation such as the Applicant Handbook (two versions, one
for the initial phase and one for the full application phase), FAQs,
application samples and templates, and webinar materials• Information about the 166 applications received in April 2021
and reviewed by the panel in June 2021
• Semi-structured interviews with 8 members of the review panel,
17 initial-stage applicants, 15 applicants invited to submit full
proposals, 3 full applicants who were not funded, and 4 grant recipients• Survey responses from 56 “non-applicants” who expressed interest in the program but did not submit applications• Survey responses from five applicants who were invited to submit
a full proposal but withdrew or did not submit a proposal• Observations of pre-panel planning meetings and panel review
meetings during the initial phase (June 2021) and the full application phase (January 2022)• Three focus group discussions with review panel members (January 2022) and one with CLIR program staff (April 2022)• Additional materials shared by CLIR staff, including emails fromapplicants and feedback surveys from three informational webinars and six applicant support webinars
To identify “non-applicants” who expressed interest in the
program but did not submit applications, we identified contacts
from lists of registrants at webinars for prospective applicants andcross-referenced these with contact information from submitted applications (see section 3.3).

10

11

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

All interviews, focus groups, and meeting observations were conducted via videoconferences on the Zoom platform. Because interviewing program stakeholders was our main data gathering activity,
we discuss this in detail below.

2.2 Semi-Structured Interviews
Semi-structured interviews constituted the largest, most complex,
and most illuminating data source. We interviewed two main stakeholder groups: (1) applicants at the initial and full application phasesof DHC:AUV and (2) program reviewers. This section summarizes
our selection process and the perspectives represented by the interviews to contextualize the interview data provided later in this report.
We took a semi-structured approach to the interviews, providing
standard opening and closing information, and organically following
a set of questions developed with input from CLIR staff. Some interviews were conducted by both members of the assessment team, but
most were conducted by just one member; the standard interview
protocol was used to facilitate this sharing of duties.3 We shared theinterview protocol with interviewees prior to each interview (see accompanying data at clir.org for our interview protocols). Interviews
ranged from half an hour to one hour and were audio recorded withinterviewees’ consent. Interviews were transcribed by a third-party
service and reviewed for accuracy by the assessment team. The quotations presented in this report are excerpted from interview transcripts.
In total, we conducted 47 interviews, speaking with 59 individuals representing 31 unique organizations. Table 1 summarizes theinterviews.
Group
IdentifierGroup Description

Number ofInterviews

Number ofInterviewees

I

Initial-stage applicants, not invited to submit full applications

17

22

F

Applicants invited to submit full applications

15

18

U

Applicants who submitted full applications but were notfunded3

5

G

Recipient of a DHC:AUV award (“grantee”)

4

6

R

Review panelists, including members and chairpersons

88

Table 1: Overview of semi-structured interviews and interview groups

To maintain privacy, we anonymized interview excerpts in thisreport to the greatest extent possible. Interviews were conductedwith the aim of gathering information about the program rather than

3 We elected not to include the protocols in the report, but these materials are available at CLIR’s website as accompanying documentation.

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

about the proposed projects, and aside from a few general characteristics, the identity of individuals or proposal specifics did not influence our findings. In cases where a specific proposal aspect, such as
the institutional context or background detail, is critical for the reader
to understand a quotation, we provide that information in our discussion.
2.2.1 Interviewee Selection

We selected potential interviewees to represent a range of perspectives from each stakeholder group. For interviews with reviewers, weaimed to speak with individuals representing various perspectives on
the panel:• US and Canadian reviewers• Reviewers with varied expertise, including subject specialists,domain specialists (library, archives, museums, and digitization),and intellectual property (IP) experts• Reviewers who are new to CLIR’s programs and those who previously reviewed for DHC
From the group of 22 reviewers, we invited 14 and conducted 8
interviews in August and September 2021.
When inviting applicants, we similarly aimed to include a varietyof perspectives. We recruited interviewees by email, using data fromapplications and data shared by CLIR. We followed these principles inselecting applicants to invite:• A mix of academic and nonacademic applicants• More small organizations than large ones
• A balance of US and Canadian applicants• Representatives of applicants whose initial applications received
highly positive or negative feedback• Applicants representing groups or organizations that have lessfrequently applied to DHC, including Indigenous organizations
and public libraries.
We interviewed a higher number of applicants from the initial
phase, so we followed a more involved process than in the full application or award phases. We sent invitations via email to four groupsof 20 to 30 applicants, with the understanding that a lower number
from each group would likely respond.
During the initial-phase interviews, we used a purposive sampling approach intended to represent each of the desired perspectives
we wanted to include. While we identified representatives of eachperspective using applicant information shared by CLIR, we were
not able to guarantee representation of each perspective within the
responses. We were, moreover, not able to effectively gauge institution
size as a selection factor from the available data, but we did consult
people from self-identified community organizations, which generally represented smaller organizations. Ultimately, we conducted interviews with applicants representing most of the perspectives identified
in our selection principles (table 2). While applicants affiliated withacademic organizations were highly represented, these applicants fre-

12

13

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

quently overlapped with multiple communities of interest; for example, interviewees from at least two academic organizations worked on
applications in collaboration with multiple Indigenous communities.
We spoke with individuals from three public libraries (all from theUnited States), and three Indigenous-identified organizations (all representing First Nations groups in Canada that were working on full
applications).
Group Identifier

I

F

U

G

Community organization

0

5

0

1

Academic

7

1

1

2

Public library

2

0

2

0

Indigenous

0

2

0

1

Government

0

2

2

0

Public5500Previous applicant

3

0

1

1

US

15

9

1

3

Canada2

62

1

Total interviews

17

15

3

4

Table 2: Selected aggregate information about interviewees. Information based on
self-reported information provided by applicants.

Academic, government, and public organizations represented byinterview participants comprised a variety of collections that document multiple communities and histories. Although we had notintended to interview previous applicants, multiple interviewees had
participated in previous rounds of the DHC program, either as grantwriters, advisors, or as applicants with other projects or organizations. While we were able to speak with individuals from both Canadian- and US-based organizations, we spoke with fewer Canadianapplicants. (In most interview groups, US and Canadian interview
numbers were balanced given the number of interviewees in thegroup, but the “I” group notably underrepresented Canadian applicants.)

2.3 Focus Groups
Finally, we conducted four focus group discussions to gather perspectives from program reviewers and CLIR staff. These focus groups
were particularly useful in eliciting discussion and shared experiences from these two groups. During the full application phase, we
conducted three focus group discussions with the review panel, andduring the award phase, we conducted one focus group with CLIR
staff. In our focus groups with the review panel, we purposely excluded members of the panel who are also employed by CLIR. Of the14

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

remaining 20 members of the review panel, we heard from half (10of the 20 non-CLIR-staff members). Our focus group with CLIR staff
took place in the award phase and included perspectives from thefour primary staff members who worked with DHC:AUV. As withinterviews, we cite quotations from these sessions using an alphanumeric identifier (table 3).
IdentifierGroup

Number ofparticipants

FG01

Reviewers

4

FG02

Reviewers

3

FG03

Reviewers

3

FG04

Staff

4

Table 3: Overview of focus groups

2.4 Limitations
We see the qualitative approach as a major strength of this report, but
it is important to understand the limitations of this data. The qualitative approach does not lend itself to broad generalizations. One primary outcome has been to share and amplify the experiences of the
program applicants and reviewers. We do, however, isolate recurring
themes that arose from our conversations and subsequent review of
the transcripts. At the same time, it is important to remember that thedata were gathered during particular social moments beyond the program. The assessment data were collected during a time of upheaval—social unrest related to issues of race, police violence, and tension
regarding educational content and standards for educating students
about race-related topics in history and other fields. Likewise, the
COVID-19 pandemic required numerous quick changes to the application review process and resulted in many organizational struggles
and changes for potential applicants, which were not always visible
in the applications. Thus, the findings should be understood within
this framework as a reflection of many subjective responses to a new
funding program at a time of tension. Nonetheless, many issues identified in this report are cross-cutting. These broader issues included
the overall program structure; peer review; organizational equity; andchanging approaches and needs for digitization of historical materials
among cultural heritage repositories, community organizations, andIndigenous communities.

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

3. Findings and Discussion
This section summarizes the findings of our assessment activities. All
quotations in this section come from the semi-structured interviews
and focus groups; tables summarize the analysis undertaken by theassessment team using data provided by CLIR. In analyzing these
data, we were particularly interested to assess the following:• Level of demand for the program, as indicated by the breadth ofapplications and feedback from applicants• Gaps in initial applications, as reflected in the range of topics,
material types, geography, or institution types represented• How the applications compared with previous iterations of DHC• How CLIR may change or improve the framing of the call for
proposals, program guidelines, allowable costs, or applicant communications and support for proposal development• How reviewers and applicants understood the DHC:AUV program values

3.1 Applications Received and Reviewed
This section characterizes the pool of DHC:AUV applications received and reviewed, at both the initial and full phases, to consider
the overall representation and to compare that with previous iterations of DHC. We also present findings on the representativeness of
the applications and any gap areas that the program may better serve.
3.1.1

Invitation, Funding, and Award Rates

An assessment across the DHC:AUV phases (initial applications,full applications, and funded projects) is key to understanding the
program’s overall invitation and funding rates. In the multi-phaseprocess, we tracked how many initial applications were received, howmany were invited to submit full proposals, how many proposals were
submitted or withdrawn at the full phase, and finally, how many projects were funded (figure 1).
Because of the variation between the initial and full application
phases (see Appendix D), we were hesitant to state a single overall
fund rate. Instead, we chose to highlight three important funding
ratios corresponding to the three program phases: invitation rate,fund rate, and award rate. From the initial applications reviewed, 67
applicants were invited to submit full applications, an invitation rateof 40%. Of these, 56 full applications were received and reviewed ofwhich 15 projects were funded, a fund rate of 27%. When considering the funded projects in comparison with the total number of initial
applications reviewed, the program’s overall award rate was 9%.
The two application phases differ significantly, so a focus on
the award rate alone obscures some of the program’s work to attractand support a diverse group of applicant organizations. Moreover,15

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

applicants had different experiences between the program phases.
Although significant work is required to submit a proposal at either
phase, at least one withdrawn applicant reported that the increased
logistical and administrative work required to complete a full application were disincentives. For applicants, therefore, each phase required
a significant, and distinct, amount of work. We suggest the three rates
as a way to differentiate the program phases.

Fig. 1: Disposition of initial and full applications received for DHC:AUV

3.1.2 DHC Program Rates over Time

When viewed in comparison with DHC cycles since 2015, the invitation, fund, and award rates for DHC:AUV do not appear significantly
different. DHC:AUV has a lower overall fund rate than previous
years, but the 2018 and 2019 award rates appear as high outliers thatraise the average fund rate. In addition, DHC:AUV received a notably
higher number of initial applications than any previous round of the
program.
Overall, the program’s application and award numbers have remained relatively consistent over time (figure 2). Initial applications
reviewed have varied, from a low of 101 reviewed in 2018 and 2019 toa high of 166 in 2021 for DHC:AUV. The slight rise in fund rates andaward rates (2018 and 2019) appears to coincide with a multi-year
award from the Mellon Foundation for DHC, which offered increased
program stability. Although the number of full applications reviewed
has increased since 2018, the number of awards has gone down, andthe program’s overall fund rates and award rates have trended slightly
downward since 2015. During these award cycles, CLIR made intentional program changes to increase representativeness of the review
panel, support collaboration, engage communities of applicants, andaddress equity and diversity in the application process (see Banks andWilliford 2018, Ferraiolo 2019a and 2019b). As suggested in section
4, additional work to address program expectations regarding collec-

16

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

tion ownership, staff involvement, and intellectual property requirements is critical if DHC:AUV intends to change or increase funding
and award rates. Moreover, multi-year funding arrangements, which
would allow CLIR staff to offer more definite advice to potential applicants year to year, would likely increase application numbers, particularly resubmissions and contribute to program stability.

Fig. 2: DHC applications and awards since 2015. Applications received in 2021 represent DHC:AUV.

3.1.3 Assessing Broad Representation

To assess representation within the DHC:AUV application pool, weunderstood the representation rate to be the percentage of applications representing a given applicant category within the total applications at a given stage of the program. Thus, if 64 initial applications
were reviewed from applicants that identified as academic organizations, and the total of initial applications was 166, the representation
rate of academic institutions was 64% of the initial applications received. In the full application phase, if 29 applications from academicorganizations were reviewed and 56 full applications were considered,then academic institutions had a 52% representation rate within thefull application phase.
Since we also wanted to understand how many applications weresuccessful within a given group, we also considered the relative success of applications within the context of a given group. We described
these related measures as invitation rate (i.e., the percent of initial
applications reviewed within a category that were invited to submit
full applications) and funding rate (i.e., the percent of full applications reviewed within a category that received funding). These rates
indicate success within specific categories at each phase, and they
provide useful indicators of how specific groups fared in relation tosimilar applicants. So, for example, although Indigenous-identified

17

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

organizations show a low representation rate (a representation rate of
4% of initial applications received, and 7.5% of full applications invited), we also note a high invitation rate for this group of 71%. That is,five of the seven applications received were invited to submit full applications, resulting in a larger representation rate in the full phase. At
the full phase, 40% of applications reviewed from Indigenous-identified applicants received awards. Thus, while representing a small
portion of the overall pool of applications, as compared to similar
applications, these applicants saw a high rate of success.
While we found the invitation rate and funding rate to be useful measures, they should be used with some caution. In categories
where very few initial applications were received, small numbers ofinvitations would result in high invitation rates. For example, only
two applications representing multiple sectors (“joint” applications)were received in the initial phase; since one of these was invited to
the full phase, the invitation rate in this category was 50%. Given thesmall underlying numbers, this rate does not seem particularly informative.
3.1.4 Material and Institution Types

Initial applications were broadly representative of mixed cultural
heritage collections. We noted collections comprising a wide variety
of materials, from large paper and text collections to ethnographic
and audiovisual materials. A notable number of collections included
oral history materials (at least 13), and at least one-third of applicants(55) mentioned some audiovisual portion in their collection. Thissuggests a high level of multiformat collections, which would requiresignificant collection management resources, specialized care, andcomplex planning for digitization activities.
A variety of institution types were represented in the applicant
pool at each phase. A large portion of these were cultural heritage
collecting institutions, including about two-thirds of applicants
affiliated with some sort of library, archive, or museum (112 initial applications, or 67%). Notably, nearly 10% of initial applicants
identified themselves as representing a community organization (14initial applications, or 8%). While community organizations were asmall portion of the initial applications, over 40% of these were invited to submit full applications, and 75% of the full applications fromself-identified community organizations (3 out of 4 reviewed in the
category) were funded.
When analyzed by sector (table 4), the applications demonstrate
a broad diversity. Academic-identified organizations represented overa third of all applications (39%); of these, just under a third identified
themselves as “independent” organizations (31%), and about a fifth
said they were “public” organizations (19%). We would note that“Indigenous” applicants only comprised about four percent of theapplicant pool (7 applications), but a very high proportion of theseapplicants were invited to submit full applications (5 applications,or 71% of the initial applications). Thus, applications from Indigenous-identified organizations saw a notably high invitation rate,
and among the funded projects, a 40% fund rate. In other words, wewould suggest that Indigenous organizations were underrepresented
in the overall application pool, but this was balanced by a high invitation rate following the initial proposal review.

18

19

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Leadorganization
sectorInitialInvited toapplications full phase

Invitation
rate

Full
Fundedapplications projects
reviewed

Fundingrate

Academic

64 (39%)32 (48%)50%

29 (52%)5 (33%)

17%

Government

9 (5%)

5 (7%)56%

5 (9%)

1 (7%)

20%

Independent

52 (31%)14 (21%)27%

11 (20%)5 (33%)

45%

Indigenous

7 (4%)

5 (7%)71%

5 (9%)

5 (9%)

40%

Public32 (19%)10 (15%)31%

5 (9%)

1 (7%)20%

Joint

2 (1%)

1 (1%)50%

1 (2%)

1 (7%)

100%

Total

166

67

56

15

Table 4: Distribution of applications by sector of lead applicant

3.1.5 Unheard Voice Groups

To characterize the communities represented in the various collections nominated for digitization, we considered the groups of “unheard voices” mentioned in proposals. To assess this element, we used
the list of community histories identified on the program website
as a taxonomy to identify groups whose voices were documented incollections identified by applicants. Thus, we categorized each application as representing one of the following: Persons with disabilities;
LGBTQIA+ individuals; Hispanic or Latino; Black or African American; Asian/Asian-American or AAPI; Middle Eastern, Arab, orArab-American; Native Hawaiian or Pacific Islander; First Nations orAmerican Indian; Alaskan Native; or Women. In many cases, applications were not adequately represented by these categories. In thesecases, we identified three additional categories: “Multiple” specified
by applicant; “Another group” specified by applicant; and “Not specified” (table 5).
Applicants invited to submit full proposals reflected greater percentages of many groups, particularly those groups represented byrelatively small numbers of initial applications. This suggests that as
proposals in the initial pool are evaluated, the program’s reviewers
and review processes tend to favor applications perceived as broadening the range of underrepresented social groups and communities
included in the pool.

20

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Voicegroups

Initialapps

InvitedInvitation
rate

Full
apps

Fundedprojects

Fundingrate

Persons withdisabilities

2 (1%)

2 (3%)100%

2 (4%)

1 (7%)

50%

LGBTQIA+
individuals

10 (6%)

3 (4%)30%

3 (5%)

1 (7%)

33%

Hispanic or Latino

12 (7%)

2 (3%)17%

1 (2%)

1 (7%)

100%

Black or African
American48 (29%)27 (40%)56%

23 (41%)

4 (27%)

17%

Asian/AsianAmerican or AAPI

9 (5%)

4 (6%)44%

3 (5%)

1 (7%)

33%

Middle Eastern,
Arab, or ArabAmerican

1 (1%)

1 (1%)100%

1 (2%)

0 (0%)

0%Native Hawaiian orPacific Islander

3 (2%)

2 (3%)67%

1 (2%)

0 (0%)

0%First Nations orAmerican Indian

22 (13%)

11 (16%)50%

9 (16%

3 (20%)

33%

Alaskan Native

1 (1%)

1 (1%)100%

1 (2%)

1 (7%)

100%

Women

10 (6%)

2 (3%)20%

2 (4%)

0 (0%)

0%Multiple specified
by applicant

20 (12%)5 (7%)25%

3 (5%)

1 (7%)

33%

Another groupspecified byapplicant24 (14%)7 (10%)

29%

7 (13%)

2 (13%)

29%

Not specified

4 (2%)

0 (0%)0%0 (0%)

0 (0%)

0%

Total

166

67

56

15

Table 5: Distribution of voice groups by proposed taxonomy of community voices identified in the program call for proposals

3.1.6 Potential Representation Gaps in Applications
Received

When speaking with program staff and reviewers, we asked whether
they perceived any notable gaps in the applications or voice groups
that appeared to be underrepresented. In general, responses indicatedthat reviewers saw the applicant pool as quite broadly representative.
As one person stated, “I was pretty impressed with the array of voices
and originating communities … that the collections represented.”
Some reviewers even noted that the array of collections nominatedfor digitization exceeded their expectations: “There were some pleasantly unexpected voices that were included—some expected ones aswell—that … I found refreshing to see.”Some reviewers also noted that the program saw a good response
from applicants with collections representing Indigenous communities. As one reviewer noted:
There was a very broad gamut of unheard voices or underrepresented communities, and … there wasn’t any particular[time when] I thought, “Oh, I wish they would have had

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

more of this.” Even in terms of my real area of expertise,
which is Indigenous people, … I felt like we had Inuit, we had
First Nations, we had Native Americans, we had all different sort of Indigenous peoples, and so I thought it was quite
strong, actually, in terms of a broad cross-representation ofsome of these voices. I can’t think of anything specific that I
felt was a glaring omission.
Meanwhile, other reviewers named specific groups for which they
hoped to see greater representation. Multiple reviewers pointed out
an underrepresentation of differently abled communities. Said one, “A
couple of voices that I wish I was able to see [in the pool] were communities that were differently abled, … [especially voices that would
help researchers] get more deeply into some mental health conversations.” Another noted, “I was surprised that there weren’t more archives around people with disabilities. … To be honest, I don’t know
what’s out there, but it was not something that I thought had a tonof visibility in terms of the applications.” A third reviewer observed,“There was maybe one application that dealt with incarcerated individuals. … Going forward, it would be good to see more representation from that segment of our communities.”
Beyond these voice groups, some reviewers mentioned types ofapplicants that were underrepresented. For example, one reviewer
said, “I was heartened to see that there were several [applications that
had to do with Native American collections]. ... They tended to befrom universities and from organizations. … I would like to see morefrom actual tribes themselves.”
Likewise, various areas of the United States may be underrepresented. One reviewer observed, “There were some proposals that had
to do with some other US territories, Pacific Islanders, and it would
be good to continue to see proposals from there and perhaps evenincrease. … [I was glad to see] two or three proposals from PuertoRico.” Another reviewer noted that some cultures were not highly
represented: “In terms of the United States, … we saw South, we saw
Midwest, and then within Native populations. … [But] there’s somany more stories from Asian cultures and Latino-Latina culturesthat could have really gotten a little bit more engagement.”
Overall, it was difficult for individual reviewers to assess thedegree of broad representation. This is due to multiple factors. The
primarily factor is that reviewers only read closely a subset of thepool, and they tend to have preferences that focus their interests into
specific technical or social areas. This focus limits the group of applications that they review in depth. One reviewer noted this challenge
from their perspective:
I find that hard to answer because I know I was assigned files
that reflect my specialization, which is also on categories ofidentity. And I had a sense from that hearing the discussionthat other people were assigned … files [matching their interests]. So I think if you probably put everything together,
there’s a broad range, but I don’t know that I can evaluatethat based on the files that I read. But within the world of the
fields I specialize in, I thought there was really good reflection of really, really different kinds of collections.

21

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Since the applications that each individual reviewer reads are
already narrowed from the whole, we think it may be prudent to discuss how each set of submissions reflects the program’s value of broad
representation separately from discussions of the individual applications. While this value should remain part of the evaluation rubric
and reviewers should be prompted to offer comments about how each
proposed project holds the potential to broaden the array of digitizedmaterials available to researchers and interested communities, it may
be useful to use the concept of “voice groups” as we have here to
ground a whole-panel discussion of broad representation.
While we cannot yet make conclusions about the representativeness of the projects to be funded by DHC:AUV over time, we foundthat the perception of broad representation of unheard voices within
the pool of applications received by the program in this first round isvery good. Proposals were received from each area identified by the
program and from other groups identified by applicants as “unheard
voices.”3.2 Program Appeal
Interviewees provided a lot of information about the level of interestin the revised program’s focus. In the spirit of assessing the level ofdemand for and interest in the program, this section addresses various aspects of the program’s appeal that applicants, and potential
applicants, shared. In general, the newly articulated DHC:AUV values
were highly appealing to applicants and appeared to be received withenthusiasm. As one initial applicant stated enthusiastically, “The prospect of amplifying unheard voices was really exciting!” Another interviewee noted that the emphasis on unheard voices made the program
more appealing:
I’m a queer person, and so being able to look at our history to
be able to bring [people] that are not White, male, cisgender
to the forefront to be able to say, “We’ve always been here”:… this was pretty amazing. … Digitizing Hidden Collections
is cool in the first place because there’s not a lot of different
places that are really supporting digitization work at thispoint in time, and then to be able to look at doing it at scale,where I might be able to get my entire collection [online],
was super amazing. And raising hidden voices made it evenbetter.
This applicant represented a large research university and was notinvited to submit a full application, but even looking back on theirapplication, they spoke enthusiastically about the values articulatedfor DHC:AUV. This response appeared to be heightened by theirpersonal identification with an historically underrepresented group.
Similarly, one community organization working on a full application
noted:
When I saw that this year, it was more for amplifying unheard voices, it really encouraged me to apply on behalf of
our institution because we are the only state-designated repository for African-American history and culture in [ourstate], and we have an amazing collection that is almost en-22

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

tirely donated by the community … it is a very small archive… that doesn’t have a lot of outside funding, and I kind ofworry that I would [not] be able to stack up against otherlarger intuitions who can afford to contract out [grant writingwork] because it was all just us—we were the ones writing.
So it was really nice to be able to see the increased focus on aparticular subject that very perfectly lined up with our mission statement [and] our history … even if we don’t have thebest infrastructure, if we don’t have all the pieces together,
I know that I can craft a story in a narrative that shows theamazing material that we have, the community passion that
we have, [and] the community involvement that we have.
As this applicant related, the DHC:AUV values were a key factor
in the organization’s decision to apply. In this case, it helped them tomake this decision since they understood that their collections answered the program’s new call, even though they did not think they
had all the necessary technical elements in place at the time of theapplication.
We also heard from multiple applicants for whom the program’svalues and emphases were not only appealing, but also encouraged
them to support new aspects of their organization’s work. A small
museum that submitted an initial application noted that the programaligned with their community values but also spurred local collaborations:
We are doing a cataloging and inventorying project rightnow, and so it really felt perfect because our project is about
uncovering untold stories in our collection, and we’ve beenworking towards a partnership with our local public library
who also has a collection, a special collection of photographsand documents. … A lot of us here are interested in why …
the marginalized peoples and communities … have beenleft out of the history of this town. So it really just seemed
perfect: a perfect alignment of our museum, the library, ourtown’s, efforts towards inclusivity and our attempt to really
understand our own collection better.
This applicant represented a small museum that had not yet undertaken a large digitization project, but they did see that the kind ofwork they were doing might be supported through the program in future. Another applicant noted that the emphasis on “unheard voices”
helped them not only to advance current initiatives but to see theirorganization’s activism in a more historical context:
I noticed that this is an opportunity for us to really dig into
[our archives] … taking a more holistic approach to theseissues of justice. We can be forward-looking … with our activism, or we can be trying to do things that are advancing
justice and mercy in society now, but by looking back andelevating a lot of our historical documents—or these resources that we have—allows us to say that [our organization] hasbeen doing this throughout our entire history. And it allows
us the opportunity to lift those up and to show those to the
public in ways that we haven’t been able to do before.

23

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

In this case, it appears that the DHC:AUV program values appealed to a community organization with a history of social justice
and activism, which was also building its archives to celebrate theorganization’s ongoing activism. They saw their socially engaged workand archival interests as harmonious with the new program goals.
Apart from digitization, applicants mentioned other activities
connected to ongoing conversations and initiatives that we havenoted in the archives field, including the growing attention toward
reparative description and community archives. For example, a Canadian-based research center aimed to advance decolonial descriptive
practices:
The program itself was very appealing because [of our] initiative to … develop critical cataloging within the institution.
So it was, in a way, a program that we thought could help …
move forward and develop further this initiative and help us
decolonize the collection more, so it was really the nature or
the fundamental objectives of the program that were appealing to us.
Similarly, a public library saw an application to the program as away to advance its engagement with community history:
Just by looking at the title of it is appealing enough. … It wasusing the terminology that we’ve been using when we weretalking about community engagement and amplifying otherstories that aren’t necessarily people who think [they] arehere in the archives. … So just the marketing on the title wasenough to make me go, “Okay, … it’s not just a leadershipgrant or a digitization grant. This is … a lot more active.”[The program] fits both our community engagement side as
well as our preservation side.
As these quotations suggest, numerous applicants saw the DHC:AUV values as highly responsive to their interests in critical cataloging and community engagement. Even those who were not invited
to submit full applications noted a high sense of alignment withpriorities for advancing social justice goals through collections work.
Although unsuccessful, this applicant’s feedback suggests ongoing interest in the program:
There’s so many community collections that you could be
seeing. Some of the Native American communities, other Indigenous populations, but just minorities and that. So I thinkit’s very timely. And I definitely think that should be the focus
going forward.
Although unsuccessful in the first phase, this applicant saw highimportance in the DHC:AUV program’s revised focus, even hintingthat interest in the program may grow in the future.
3.2.1 Appeal for Canadian Applicants

Canadian applicants noted that there are few comparable Canadiansources of funding to support digitization activity of the sort that they
are interested in. As one applicant from a university archive stated:

24

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

We have the possibility of applying for SSHRC grants up
here, … but this wasn’t so much a research project as an accessibility project, so I thought CLIR was really the best option for us, and … the funds available would support the kindof work that was going to be required to work with Northern
communities.
Likewise, an applicant from a government-supported Canadian
research center noted:
In Canada, a lot of the funding or digitization programs are
directed at community-based archives, and … we’re not eligible for a lot of the granting programs because we’re … agovernment body. … So, we fall into a gap.
These applicants indicated that DHC:AUV serves particularneeds and functions for quasi-governmental institutions in Canada,
which are similar to some US-based nonprofits but operate on different funding models than state-supported cultural organizations in
the US. This suggests an ongoing demand for grant opportunities like
DHC:AUV among Canadian applicants.
3.2.2 Appeal to Reviewers

Reviewers likewise communicated high enthusiasm for the program’s
goals and timeliness. One reviewer succinctly summarized that theattraction of participating in the program was the potential to effectpositive change in the preservation of collections for underrepresented people:
Archives have traditionally been not as open to amplifying
those voices, so I think that [this service as a reviewer] is very
important. … Being able to be involved with something thatis more proactive and contributing to the general body ofknowledge was rewarding.
Others noted that the work of reviewing applications was achance to support the sort of collaborative, community-focusedarchival work that they value, while others referenced the value ofserving communities and collections that they have worked with or
supported.
Reviewers expressed a sense of satisfaction in doing work thatserved collection needs, scholarly goals, and social needs. As one observed, their service satisfied both scholarly and archival goals:
The nature of the grant [program] combined a topic that ispart of both my scholarly interest but also my interests as acommunity member and the volunteer work I’ve done in thepast with community archives. So if this funding opportunity
only funded universities or large institutions, it wouldn’t haveinterested me. I was really interested in the idea of a grant for
centering the records of marginalized voices that meaningfully can partner with community groups, … and I liked thatthe process was framed as being also about bringing together
a community of reviewers in the adjudication process, andalso in a kind of celebratory way.

25

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Another reviewer pointed to the broad impact of informing thereview process and supporting communities and collections:
Everyone I work with is a perfect candidate for [this program] . … I’m excited about this grant, from a couple of different angles, so that’s why I was excited to review for it. I’dnever been a reviewer. … It’s a huge honor, and I know whatgoes into making those applications because I’ve had to dothat kind of work before, so I felt like it was just a huge gift tobe asked.
These two perspectives suggest that the DHC:AUV values motivated reviewers to accept the time-intensive work of reading, commenting on, and reviewing complex project proposals, since they feltthat this work was helping to move the archival work they value inpositive directions. Moreover, even though we know from program
staff that the work of recruiting reviewers is time-intensive, there
seems to be a dedicated corps of scholars and archivists who want tosupport the DHC:AUV program emphases.
While the program’s service to communities and collections wasappealing, so was the authentic consideration and development that
reviewers saw in the way that CLIR has assembled the program. Not
only did numerous reviewers note that it was an honor to be asked toreview, but they also noted the care that CLIR took in demonstrating
the program values. As one reviewer noted:
I thought for an organization like CLIR to be leading withthose themes at this time, with the way that it was framed,
it was intriguing rather than, “Here they go again. It’s
2021/2020 and it’s on brand.” But it felt deeper.
This reviewer’s example suggests that, as with applicants, reviewers regard CLIR’s efforts and priorities with the DHC:AUV program
as meaningful, not only for the individual projects it supports, butalso for its potential benefits across historical collections and the cultural heritage sector.
The sense of community among reviewers, which was nurtured by
CLIR staff and the panel process, was clearly attractive to reviewers as
well. Reviewers appreciated the opportunity to gather with others whoshared interests in archives and communities. As one put it, “I was interested to collaborate with other people who would have been invited
to review as well.” Another noted, “My interest is both in the topic and… in learning how it’s done, because I’m not really an archivist.” These
responses illustrate the need to continue cultivating a community ofreviewers who embrace the program’s values as much as they bringscholarly and technical expertise to the work of DHC:AUV.

3.3 Perspectives of Potential Applicants
We considered two types of “potential applicants”: (1) individuals
who may have considered an initial application but chose not to applyand (2) those who were invited to submit full applications but either
chose not to submit one or withdrew after submission. In both cases,
we aimed to identify reasons that led to these applicants choosing not
to proceed. For the first group, we surveyed a group of “non-appli-

26

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

cants”—people or organizations that showed interest in the program
but did not submit or complete an application—about their decisions
in order to identify barriers and disincentives. The second group,“withdrawn applicants,” consisted of a smaller, clearly defined groupof eight teams who chose not to proceed with full proposals.
3.3.1 Non-Applicant Survey

This survey was conducted in June 2021, following the receipt of initial applications. We created a list of these “non-applicants” by comparing the list of applications received with lists of attendees at three
informational webinars held prior to the deadline and a list of applications started but not completed. By eliminating matches to affiliatedinstitutions and email address domains cited in received applications,
we created a list of 445 potential non-applicants.
We circulated invitations to these non-applicants via email usinga Web-based survey tool (SurveyMonkey). Of the 445 invitations,nearly 20% were opened (85), and we received 54 responses to thesurvey (45 complete responses, and 9 responses to some but not allquestions). We estimate this as a 12% response rate, which is significant given that most of these individuals had not submitted applications.
The non-applicant survey aimed to identify barriers or motivations that led potential applicants not to apply. Survey questions covered two major areas: first, aspects of the program requirements andtimeline that influenced the respondents’ choice; second, open-ended
questions about what considerations went into the choice not to apply. (The non-applicant survey instrument and a question-byquestion summary are available in the documentation accompanying
this report at clir.org.)
3.3.2 Notable Non-Applicant Survey Findings

Overall, the responses from non-applicants indicate high enthusiasm for the program. When asked if they would “plan to apply in theFig. 3: Most non-applicant respondents indicates strong enthusiasm to apply to a similar program in a
future year. Non-applicant survey, Q6; N=46.

27

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

future” (Q6), two-thirds of respondents answered affirmatively, andnearly half strongly agreed with the statement (figure 3).
Lack of time to assemble an initial application was a significant
reason that potentially interested applicants did not submit applications in 2021. When asked specifically about the timeline of the application process (Q4), one-third of respondents indicated that they felt
strongly that they did not have enough time to complete the application, and half agreed with the statement to some degree (figure 4).
One respondent noted that this issue was particularly acute for tribal
applicants, who may have additional internal processes that requiresignificant lead time in grant seeking:
Many TALMs [Tribal Archives, Libraries, and Museums] areunderstaffed and have little support or time for research andwriting proposals. RFPs need to be released to underrepresented groups at least a year before the deadline so that thereis time to research, develop, write, and garner support.
Another respondent noted that the complexity of the applicationand the short timeframe prevented them from applying this year, butthey hope to apply in future:
Sometimes the notices arrive so late, and the deadlines are sotight that it is hard to apply because of the complexity of doing the application process. So, best to flesh out the concepts
and writing and have it set to go the next time the opportunity arises to submit.

Fig. 4: Half of the non-applicant respondents agreed or strongly agreed that they did not have enough time toprepare and complete an application. Non-applicant survey, Q4; N = 48.

We suspect that the newness of the RFP for this year’s DHC:AUV
program was welcome but that some applicants were not well-positioned to prepare a submission given the new guidelines. In that
respect, it seems likely that some of these applicants would consider
applying in a future round.

28

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

A few barriers were mentioned multiple times in survey responses. First, multiple comments pointed toward a perceived institutional
bias in the program. Second, some comments pointed out the inherent power dynamics that complicate the relationships between
community organizations and collecting institutions. Third, despite
the open and specific identification of numerous “unheard” groups inthe program guidelines, some groups felt alienated in the process; thiswas particularly notable for those who self-identified as Native American and/or First Nations. Fourth, numerous comments suggested
barriers regarding readiness for work that would focus on digitization
activities, from needs for greater processing to concerns about ownership and copyright of collections.

3.3.2.1 Perceived Bias toward Large Institutions
Multiple responses described a perceived bias toward large, established cultural heritage institutions and organizations. One respondent, for example, noted the challenge of serving non-collecting organizations through cultural heritage grants:
As someone who used to work inside a large cultural heritage org, who is used to working with funding like the CLIR
program, it is extremely enlightening to work outside thatsystem and think about what would be truly appropriate. It isan entirely different world. These folks deserve support, andtheir collections are worthy, but the whole process, from appraisal to use, is just really different and needs to be allowed
to be different. I feel like the DHC program’s heart is in theright place, but it is not a good fit. I’d characterize their needs
as being mainly a combination of capacity building and along and flexible timeline. The grant cycle in general is notconducive to their needs.
Another respondent, who identified themselves as representing
an independent religious organization, noted organizational infrastructure, specifically the lack of support to manage a grant, as a notable barrier:There was a lot of scary wording about how we would selectand pay our staff. As an all-volunteer organization, withvolunteers already stretched thin, we would need to hire a
professional [to manage the grant] and pay them market rate,with benefits. None of our regular staff get benefits or professional salaries. … That’s when the ship went down. … A
nice established organization like a college or museum does
not need you as much as we do! But they, unlike us, have theinfrastructure to allow help to come.
We would note that it is unlikely that DHC:AUV—or by extension any collections-focused, project grant—will provide a helpful
solution in situations where applicants are already significantly
under-resourced, substantially unstable, or facing major challenges
beyond collection needs. Nevertheless, this response does point out
unique staffing challenges that may be similar for other small organizations or less frequent grant seekers. Moreover, the high level of detail solicited for the initial application may contribute to the sense of29

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking“scary wording.” Another volunteer-led organization added that theemphasis on salaried positions put them at a disadvantage:
We are an all-volunteer organization. We were concernedthat the emphasis on paid, salaried employees would disqualify us from receiving funds, because our mission statement
prevents us from instituting that structure. It felt like there
was an implication that social justice can only be enacted
through salaried employment.
Multiple responses suggested that some potential applicants perceived the initial proposal requirements as prohibitively complex andtime-consuming to meet:
Time to complete the application would have been a barrier if
we had continued.
We could have completed the application with current
capacity, but based on the previous year’s application, it isa very heavy lift. The application has many more questions
than most federal grant applications. Some questions seemed
overly esoteric or theoretical. More focus on simple, practicaloutcomes and clear, direct questions would help.
The length of the application, the lack of assistance fromprogram directors, and the requirements that are a barrier forsmall budget institutions … without reassurances from theofficers or leaders of CLIR, we could not commit to anothertime-consuming process that had several barriers and unknowns for our team.
The application is too time-consuming for small budget
institutions.
These challenges may be addressed to some extent by reworking
language in the guidelines about “collecting organizations,” removingsome of the initial application elements such as itemized budget attachments, and communicating openness to contract positions (versus a perceived emphasis on permanent or salaried staff). We explore
these possibilities in more detail in our recommendations (see section
4.3.1).
One respondent suggested creating a tier of support for smaller-scale projects: “A lower tier with less programming requirements($15,000 with focus on finding aids and collections) maybe … forsmall museums that have less staff.” This kind of opportunity could
support greater capacity building for smaller organizations. Weresuch a possibility to be explored, however, it should be planned tocomplement existing funding or capacity building programs forsmaller institutions, such as the Preservation Assistance Grants for
Smaller Institutions from the National Endowment for the Humanities (NEH 2022) or the Museum Assessment Program supported by
the Institute for Museum and Library Services and American Allianceof Museums (AAM 2022). Despite the existence of other opportunities, none of the alternative programs available to US applicants is
explicitly designed to support the intellectual work of collections processing and analysis suggested by this respondent.

30

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

3.3.2.2 Organizational and Cultural Power Dynamics andDifferentials
In addition to perceived challenges for smaller organizations, we noted that additional power dynamics, including past extractive cultural
relationships and the “outsider” status of collecting organizations,were noted by individuals tied to Indigenous or Native organizations
as disincentives to their application plans. For example, one respondent who identified themselves as representing an Indigenous archive
wrote:
Funding parameters were really more focused on supporting large, well established entities to further their collectionof marginalized communities versus allowing communities
such as our tribal community to build and enhance its capacities for archival and documentary histories and tell our own
stories in our own voices.
This comment suggests that program biases, perceived or real,that favor “established” organizations can cause applicants from Indigenous communities to hesitate, perhaps due to a distrust of outsidefunding agencies. While the exact reasons were not clearly explainedin the comments, this hesitation may be linked to the program’s requirement to entrust intellectual property created through projects
to CLIR if a grant recipient cannot sustain access. While importantfor making materials available for research, this condition may beseen by Indigenous communities as a request by outsiders for control
of collections. The institutional bias concerns may also be related toapplicants’ worries about organizational capacity: time to researchand understand a grant program, plan and write a complete application, and manage an award if offered. Tied to this observation may
be concerns—again perceived or real—about the power dynamics
inherent in partnerships between smaller and larger, more establishedcollecting institutions. These comments point toward a perceptionthat larger organizations tend toward controlling not only the items
themselves (“collection of marginalized communities”) but also theirrepresentation (“tell our own stories in our own voices”).
In other words, we detect here a significant concern that organizations originating or stewarding collections of underrepresentedsocial groups may not only lose control of the materials by partneringwith larger organizations or funders, but might also cede control of
the narrative. Applicants representing Native American and First Nations groups, as well as some community organizations, mentioned
similar apprehensions in interviews.

3.3.2.3 Concerns about Representation within the Program
Possibly related to the perception of organizational power dynamics
in cultural heritage, some potential applicants perceived or assumed
they were underrepresented within the program:
It seemed like a lot of the webinar participants were fromnon-Native institutions, and that made me question our eligibility.
Who are the reviewers? Do they represent a tribal or ethnic
community?

31

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

These concerns were voiced by only a few respondents, but wethink they deserve attention. Whether based on perception, assumption, or some other factors, these hesitations shaped the applicant
pool and would continue to do so if not addressed and therefore
weaken the program’s ability to inspire broad representation in digital
cultural heritage. Other studies of peer review processes have suggested that diverse applicants within a group of proposals are “evaluatedand judged most fairly when they make up a critical mass” of thepool (Wigginton et al. 2021), so if potential applicants are self-selecting to not apply, this creates a feedback loop. Thus, if CLIR hopes toreach new applicant communities, it faces the challenge of breaking a
feedback loop wherein applicant perceptions reinforce the perceived
exclusion in the program, despite increased outreach to new applicant groups. If the program makes more awards to recipients that
ethically serve and support groups underrepresented in the historicalnarrative, including notable projects led by or partnering with community-based organizations, that may change the loop over time bycommunicating that Indigenous applicants and community-basedorganizations do indeed receive support.

3.3.2.4 Collection Readiness
Finally, numerous comments suggested that the significant work ofprocessing collections posed a barrier for some applicants. Others
noted the challenges posed by ownership and copyright:
We struggled with copyrights. We have collections that wewant to make available, but we don’t own those copyrights.
We would like to digitize those collections and make themaccessible within our library branch for public access.
There were some questions about whether or not thematerial would be considered “owned” by the group, since
they share ownership with the people who originally submit
the material and gladly return that material or withdraw it if
the original creator or submitter requests. The creator clearly
consents in the consent form to having their material digitized and shared online, but we explicitly want to ensure thatthe creator continues to have agency over their materials. It
was also difficult to [address] all of these questions during thesmall amount of time we had to prepare, especially as a smallorganization without any grant-writing staff.
Our partner organization has no interest in “owning”
the materials in that sense, because the creators are alreadymarginalized. … The materials seemed to fit beautifully with
the CLIR description of “hidden collections” but to us, theownership requirement is in real conflict with the ethicalconsiderations we have put into our consent form with thepartnering organization.
If the program continues to prioritize public access to digitizedmaterials, collection ownership expectations are advisable. If greater
involvement of community organizations is desired, however, thismay require additional attention.
Finally, some applicants were just not yet ready to proceed to amajor digitization project:32

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

The materials still need to be collected and archived first in alibrary.
The collection assessment and description was the trickiest part for us, especially right now when we don’t have easy
physical access to the materials.
Some of these issues may be intractable. A feasible digitization
project requires a certain level of planning on the part of applicants,
and the program aims not only to fund digitization but also to encourage responsible collections care and management. Additionally,
the pandemic has created challenges for collections managers that are
beyond the control of the grant program. Taken together, the above
comments, nonetheless, suggest a need for additional capacity-building resources for potential applicants who desire to prepare collections for digitization.
3.3.3 Withdrawn Applications

In the full application phase, the review panel considered 56 applications. This was slightly fewer than the invited number of applications
from the initial phase, when 67 initial applications (of the 166 initial
applications reviewed) were approved for advancement. CLIR staff
determined that three of the full applications submitted did not meetthe program’s eligibility requirements (5% of the invited applicants),
and eight applicants either withdrew or did not complete the full application (14% of the invited applicants). We sought the perspectivesof the eight applicants who withdrew through a survey circulated toprincipal investigators and collaborators listed in the initial roundapplications. Five applicants responded to the survey. (The withdrawn
applicant survey instrument is available in the data accompanying
this report at clir.org.)
Applicants withdrew for different reasons, and no single issue
stood out. We did, however, identify the following themes in decisions to withdraw or decide not to submit a full application:• Concerns about privacy (one respondent noted a “lack of clarity
on who would have access to all documents”)• Concerns about control of digital materials (for example,
non-Native people working with and making decisions about
what collections are digitized and how they are accessed)• Inability to get quotes from the vendor they wanted due to thepandemic• Tensions around collaboration, which took various forms, including asymmetries between collaborating partners such as:º organizational power (for example, tensions between an
academic institution working with a tribal organization, or
between a local public media station and a national-level
partner)º staffing (generally, staff from small organizations worried
they might be overshadowed by larger, or more well-known
institutions; similarly, organizations representing marginalized communities raised concerns of being outnumberedby staff at largely White, or historically White, institutions)33

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking• Concerns about readiness to pursue a large digitization project(applicants relayed decisions to prioritize work such as a collection inventory, digital asset survey, or preservation assessment
prior to pursuing DHC:AUV funding)
These responses indicate that some withdrawals were spurred byfactors that could be addressed by DHC:AUV program changes, butmany other circumstances are beyond what the program can accommodate. For example, lack of readiness, collaborative tensions, andthe difficulty of obtaining quotes during the pandemic are largely outside the program parameters. It may be possible for CLIR to addresselements of the feedback centered on privacy and control, throughprogram modifications in areas impacting the ownership of collections, open access, and intellectual property (see section 4.5).
Despite their choice to withdraw applications, many of these applicants noted their continuing interest in the program. At least one
respondent highly praised CLIR’s work to support them, writing, “We
had a great experience and the CLIR staff who assisted us with ourapplication were very helpful.” Of the five respondents, four voicedinterest in applying in a possible future round of DHC:AUV.

3.4 Perspectives of Applicants
In our interviews with program applicants, we aimed to learn more
about their experiences applying to the program. Specifically, we
aimed to gather information that helped to answer these questions:“Are the values and criteria for assessment clear to potential applicants, and do they receive enough support in developing their proposals?” We asked questions about what led them to apply, how useful the program resources were in preparing their application, how
clear and understandable the review process was to them, and how
useful they regarded the feedback they received concerning decisions
from CLIR. (See Appendix B for the question protocols that guided
our semi-structured interviews.) As explained in the methods section(2.2.1), we conducted 40 semi-structured interviews with applicants;
about half of the interviews were with applicants who had not been
invited for the final round, and about half were with those who had
been invited to complete full applications. Although this report includes views from both groups, the emphasis is on initial applicants.
3.4.1 Program Values

In general, applicants appeared to find the program values (figure 5)
appealing and in many cases, they noted the values as a significant
factor in their decision to apply to the revised DHC:AUV program.
At the same time, we often noted that applicants did not strongly
differentiate among the program’s five articulated values; in manycases during interviews, we were asked to reiterate the values orpaused while applicants reviewed the program resources or theirapplication. We expect that this was partially due to the amount oftime that had passed between applicants’ direct work on applications(often four or five months), as well as the wording of our questions(we did not specifically outline the program values in our question
protocol). It is nonetheless worth noting that applicants often voiced34

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

a general agreement with the program’s principles and values but did
not always directly articulate them. For example, this statement froma university-based archivist discusses community, partnership, andrepresentation within collections in complex ways while also stating
their alignment with the program:
Community partnership is really important to us, and as wework to decolonize and unsettle our collections, working
with communities to provide more appropriate access led by
the communities is really important. So, it was like a custom
fit for us; it seemed like the CLIR program values had been
written with us in mind.
This respondent seemed to elide at least three of the program
values including community-centered access, authentic partnerships,
and public knowledge, suggesting general alignment with the program but not clearly differentiating among the stated values.
When discussing the program values, applicants most often
mentioned approaches around community and partnerships, which
we assume to align with the program values of community-centeredaccess and authentic partnerships. For example, one public librarian
noted the intersection of community and partnership values in the
program:
It was very realistic, very finger-on-the-pulse of what people
are trying to do with community archives, and working andnot having transactional partnerships, but having authentic
partnerships and building off of those. So that’s where I reallythought—that’s where I really valued this grant and the vision
here.

Fig. 5: The DHC:AUV program values were listed and explained on the program webpage, as illustrated in this screenshot. See https://www.clir.org/hiddencollections/.

35

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Another applicant from a public library likewise noted that “eventhough we’re a public library, … we tend to operate more like a public history program.” This applicant also mentioned that sustainableinfrastructures was “something that we were focusing on, and again, I
think the reviewers felt that we didn’t have the infrastructure in place
for that. But we were trying to build the infrastructure to have it.”
We noted scant mentions of public knowledge and broad representation by applicants. One possible reason for this, as noted above,is that the conversations were conducted well after applicants had
been working on applications. We would also suggest, however, thatthese two values actually relate to the overall constellation of the applications received, invited, and ultimately, funded by the program.
So, while some questions in the application do solicit information
about these aspects, these two global values are perhaps of less immediate relevance to applicants than are their local collection and community needs.
3.4.2 Program Resources for Applicants

Program resources for applicants were primarily available via the
DHC:AUV program page on CLIR’s website. These resources included:• The Applicant Handbook (two versions: v.2 for the initial phaseand v.3 for the full application phase)• A Frequently Asked Questions (FAQ) page
• Application materials (including two sample intellectual property
agreements and templates for the list of collections proposed for
digitization; sample one-year budget, two-year budget, three-year
budget, budget narrative, and work plan)• Webinar transcripts and recordings
Applicants generally praised the materials for their high level ofcomprehensiveness, clarity, and usability. One applicant exuberantly
related high praise for DHC:AUV program resources:
Honestly, this is an absolute gold standard. This is the waythat I think that everybody should do it, the help that you
had in place, the structures that you had in place, the clarity,
the ease of access, y’all knocked it out of the park.
While others were less enthusiastic, we found that similar sentiments about the high responsiveness, clarity, and approachability ofthe program documentation were shared by many other applicants:
The handbook was awesome. We just used that, and it answered all our questions.
The handbook, I would say, is definitely the most useful,and the fact that it could be a collaborative tool through Google Docs, that was extremely helpful.
The fact that it also had a guidebook or a guideline …
was … so super helpful, and … it’s an easy read, too … It really helps when … the guidebook that’s supposed to help you
is an easy read.
CLIR resources are really good. … [The program] waswell documented.

36

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

All of the resources were really well organized, CLIR
was actually one of the most well-organized funders that I’ve
come across in a while. The information was, I think, … wellstructured. The webinars were clear. … The information waspretty concise.
As these quotations show, the applicant resources appear to behighly appreciated by applicants. There were, however, requests for
additional types of resources. Other requested support included:• Webinars earlier in advance of the application deadline• Review of drafts by program staff with comments prior to the program deadline• Information about funding ratios and numbers of applications
from previous rounds• Potential for conversation or direct contact with program officers
Moreover, multiple applicants noted the utility of sample applications from previous rounds. Such materials could obviously not be
included in 2021 since this was its first year, but this is worth noting
for future iterations of DHC:AUV.
3.4.3 Application System

Applicant experiences with the application system (SurveyMonkey
Apply, or SMApply) and the process of assembling and submitting
applications were overwhelmingly positive. As one applicant stateddirectly, “I thought it was really easy.” Another stated enthusiastically,“It was super easy to use, because the instructions were so clear, and
I understood what needed to be submitted in PDF form and what
the size of the documents were and things like that. It was brilliant,
it was a really easy-to-use system.” One recalled, “no problems, and I
remember it being easy, … and working just fine.” The complementarity between program handbook and application was frequently
noted: “I found [the application system] to be fairly easy to use, especially because we prepared most of the materials ahead of time in thehandbook template.”
Other applicants praised specific features. One appreciated that
it was possible to change and modify attachments without direct
permission from CLIR, which was not allowed in other systems they
had used. Another noted the helpful indicators from the system about
whether applications were complete or not: “It was great. It was actually great. … I believe it was green.”
Respondents noted some frustrations and additional features theywould like to see. One applicant encountered frustration when they
thought the handbook asked for a word count, but the application
system used a character count, which required changes at the lastminute. Another applicant requested more full-featured text formatting capability, noting that the lack of function hampered their preferred approach to grant applications: “There was no way to format
our narrative or anything like that. One of my grant writing strategies
is to underline things that are important, to bold them.”

37Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

3.4.4 Application Structure

Few applicants shared specific feedback or suggestions about thestructure of the application, but some did offer insights on theirexperience during the application process. In general, although theprogram application phases appeared to be clearly explained in program resources, applicants interpreted the phases in different ways.
For example, multiple applicants misjudged the amount of detail to
include in their initial applications, perhaps assuming that they could
supply it later should they be invited to submit a full application. One
applicant explained:
If I understand correctly, there is the first phase to, let’s say
to do this first application, and then if you were selected, you
had another period in which you had to give more detail
and we thought, … that the detail that we would give more
specifically would be part of that second phase and maybe wemisunderstood that.
We believe this sort of confusion may have been amplified
through the review process, when reviewer comments prompted
many applicants to provide more detail on budgets, workplans, orpreservation solutions. Some applicants had assumed they could
provide this later. One applicant from a public library system felt thatthey were questioned about technical capacity in their initial-phaseapplication even though they did not think they were asked to provide such detail in the first phase:
We have all of these other technology questions that we could
have answered for them. They didn’t ask those questions. …
When you’re asking touchy-feely sort of questions, you’re notgoing get very technically based formatting answers. … Partof the frustration, I think, is that we are very well situated to
do this. Also, I think part of it is we do look small. We’re notas small as we look.
The full application was notably more complex than the initial application, but the abbreviated application was nonetheless perceived
as a challenge. While this seemed to be expected by larger institutions, representatives from smaller organizations suggested that thecomplexity of the initial application heavily factored into decisions
about whether to apply. Said one archivist:
It’s really overwhelming to apply. … I would say we probably
put 40 or 50 hours into that. … And that’s paying people, because everybody is contracted here … grants are, they’re notfree money. They actually takes a lot of time to [manage] andto apply for. I do think though that getting all of this together
for us, gives us a proposal we can take to community funders.
So, I’m not regretful that we spent time on it, and in someways it maybe is like the kick in the butt to get you to getyour stuff together so you can go out to these other people.
But when you just don’t have lot of resources, any time you
take on anything, is [significant].
This applicant suggested that it would be useful to have additional
feedback from CLIR to indicate whether a proposed project would becompetitive within the program. As this applicant continued:

38

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

It might even be nice to have another review that’s maybe justlike a letter of interest or something. … Maybe having some
sort of first level like, “Hey, you guys are a good fit.” … With
all these grants like this, it’s just … a lot of time and effort,
and if you’re a small organization putting that in, there’s acost to it.
Another solution may be to reduce the amount of material required for the initial-round application. Either way, attracting more
applicants from smaller, community-based organizations may require
lessening the burden of this administrative tax; many larger organizations have significant administrative staff, whereas community-basedorganizations may or may not.
Speaking about the initial application structure, one applicant
working with a large, national-level museum application expressed
a concern that the application would be a heavy lift for many smallorganizations:This would be a very difficult application for a smaller or lesssophisticated institution, so if you really are talking about
amplifying unheard voices, you might want to try to find away to make this a little less complicated. … I’m thinking
about smaller organizations. Even some of our partners …
probably couldn’t be successful at this, but what they have isjust this enormous wealth of information that nobody knows
about or can get to. … So, if you really want to get this out, it
probably needs to be more user-friendly to smaller organizations.
To maintain information quality of the application but also reduce complexity, we would suggest reducing the number of attachments that applicants are asked to include during the initial round.
For example, rather than a separate timeline, the application might
include an abbreviated timeline that accounts for the requested funding period, staffing, and a general description of digital asset management resources for beginning the project. A two-page timeline at
this point also encourages reviewers to give more attention toward a
project’s technicalities rather than whether it fits the program’s values
and priorities.
Similarly, the budget section could be reduced from a standalone
attachment to a budget summary section—500 words or fewer—that
describes staffing and salaries (including consultants or training), any
outsourcing or vendor costs, and equipment. Additional costs could
be mentioned, and it would still offer CLIR and initial reviewers achance to spot any disallowed or ineligible costs. Reviewers could
still make general recommendations and evaluations, including items
such as staffing levels, reasonableness of salaries, or proposed service
provider estimates.
3.4.5 Feedback to Applicants

All the applicants we spoke with, regardless of whether they had received positive or negative comments, confirmed that it was useful toreceive written feedback from the panel regarding their applications.

39

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

3.4.5.1 Characteristics of Useful Feedback
Applicants generally appreciated comments in the written feedback
that were direct, clear, specific, and actionable. As one applicant at theinitial phase observed:
It was quite good feedback. … There were several points
throughout that were actionable, and that we can … incorporate … into another proposal, and possibly have a higher
chance of succeeding next time around.
Others observed that receiving the written comments made theirwork feel valuable, even if they did not receive an award:[The reviewer comments] were very, very useful. It’s like,“Wow, a human read this and took the time.” … That was
really gratifying.
Feedback quality from the panel varied, however. For others, thefeedback caused confusion or left them wondering how they shouldhave approached their application differently. As one applicant noted,
vague feedback left them with questions:
I think we were left … with more questions. … If we had to
do another application, … it wouldn’t be that clear. … What
do we have to work on more? How can we do it better?
Applicant experience influenced how feedback was received in
the initial round. Experienced grant applicants had more context that
aided their interpretation of feedback. As one observed, though, thatexperience would not necessarily be shared by each applicant:[The feedback] really was sort of like, “This is a great project.
This is wonderful, great.” Which … was totally fine with me.
There were a couple very specific comments, … to give moredetails, so we did. … I was like, “Okay, good. Let’s just makethese changes and go.” … [But] I think we’re at a … different… being experienced grant writers. … If I was inexperienced
and the project was new and I had only gotten the minimalcomments, I would have been a little nervous.
Applicants also related concerns about feedback that was not connected to stated program objectives, application elements, or content
of the application submitted. These concerns were amplified when
feedback was not accompanied with a clear statement about why anapplication was not advanced to the next phase. As one finalist stated:
The feedback that we received after the initial application was
helpful as we developed our final application. I will say … I
do remember one reviewer on the final version really making
some assumptions that I believe were inaccurate. And I haveto presume that it negatively impacted our application. … If
somebody has made a presumption about your project, andyou weren’t able to clarify it, I find that less helpful, otherthan thinking that probably the next time you present it, youneed to be a little more clear.
Beyond the frustration of receiving advice that was perceived
as careless or biased, the previous two responses suggest possible

40Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

program actions: guidance for reviewers to specifically note areas
where they do not have enough information to make a recommendation, and a channel of response for applicants to address critiquesor misperceptions from evaluators. We explore these further in later
sections of this report (section 4.4.3 and Appendix C).

3.4.5.2 Impacts of Feedback
In many cases, applicants relayed examples of how the feedback
had impacted their projects and organizations in ways that reachedbeyond the proposal. As one applicant observed, “Not everybody issitting there thinking they’re going to get the money . . . they’re using
this positive feedback as leverage for other things.” These broader
benefits were varied and included items such as improving and clarifying project plans, identifying areas where writing needed revision
or greater focus, gathering institutional support for digitization projects, raising awareness of collections preservation needs, building
relationships with allied organizations or community members, andarguing for the need for additional staff positions or salary.

3.4.5.3 Clarity of Feedback
Various aspects of reviewer feedback caused confusion for applicants,
including vaguely worded comments, comments perceived as reflecting a lack of comprehension of the proposed project, and apparent
contradictions between comments from different reviewers. Someapplicants experienced frustration due to apparent disagreement
between reviewers and the perception that reviewers misunderstoodtheir proposals based on reviewer comments. Although these situations appear to be rare in the process, they caused alienation and annoyance among applicants. One example shared with us displays both
reviewer disagreement and a misunderstanding. It is also useful inthe way that it suggests possibilities to address such split feedback:[The response] starts with rights and ethical review. Very
impressed with the extent. And it goes on to say, that this is
great. That’s the person who is an ethics expert. Okay. Thenwe get reviewer number one, who goes on and on about how
we’re not properly considering ethics, and brings up things
that of course we did consider, and we have addressed. Andso, if you’re asking about the process here, I would say thata gap in the process … somebody should be responsible forlooking at those reviews and being able to have the right tosay, “This one is out in left field by itself, and I’m gonna disregard it.”This finalist observed inconsistent comments between reviewers
regarding their approach to creator privacy, and they felt that they
were critiqued for deficiencies that had been well addressed in the
proposal. While this situation is difficult to avoid entirely, it is worth
noting the suggested remedy: a change in review process. As we discuss later (section 4.4), we propose that the program staff have a moredefined role in addressing concerns of this nature and that CLIR havea specific mechanism for addressing or settling these concerns.
In addition, applicants noted some alternative models for communicating feedback. In one example, particularly useful in cases

41

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

where applicants might be encouraged to resubmit a proposal at afuture time, one applicant noted that Canada’s Social Sciences andHumanities Research Council (SSHRC) has adopted a mechanism
for recognizing applications “recommended by the committee” butfor which there is not sufficient funding at the present time (SSHRC
2022). In a second example, an applicant pointed out an approachtaken by the US National Historical Publications & Records Commission (NHPRC), in which program staff share feedback with applicants and ask specific questions, after which “applicants have an
opportunity to answer these questions” (NHPRC 2022). Both of theseapproaches would require systematic changes, which have implications for program staffing as well as policy, which we discuss in therecommendations section.

3.5 Perspectives of Reviewers
In our conversations with reviewers, we attempted to gain insight
into their interest in the program, understanding of their task as reviewers, interpretation of the program’s values, and experience withthe process. The cross-section of perspectives represented in thesample came from both subject experts and technical experts as well
new and returning reviewers. Our goal was to gather data to addressthese questions: “Are the values and criteria for assessment clear toprogram reviewers, and do they receive enough support in evaluating
proposals?”
In general, as with applicants, we noted high enthusiasm among
reviewers for the program, but there was a range of approaches tothe review task. Multiple reviewers noted the positive and generative
experience of participating in the panel discussion (conducted via
Zoom in June 2021):[The panel meeting] was so positive for me. … I really felt that
it was one of the most thoughtful groups of people that I’ve
ever been part of, and I’m like, “Wow, I just want to be in conversation with these people my whole life.” … I was wondering,
how can this be sustained beyond this? It really felt like that,
like, does it have to be this sort of moment in time and theneverybody goes back to their lives? Or could we be in a network? And maybe that was part of the COVID, just longing fora connection and like-minded people, but I do think that CLIR
really was successful in communicating sort of a communal
feeling of like we’re in this together, and I think everyone took
it very seriously and was very thoughtful. … My overall experience was so overwhelmingly positive.
Although this comment largely addresses the panel meeting itself,
it also shows how engaging the service of reviewers can be. We heard
similar, positive sentiments from others, who generally found the review meeting to be an intense but evocative, intellectually rewarding
activity, and focused on communal and supportive dialogue. As theabove quotation also shows, there was even a longing to extend thisconversation beyond the scope of the review meeting; this suggestion is beyond the scope of this program assessment, but we thought
it may also be shared as something that might be taken up in other
CLIR programs, groups, or venues.

42

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Below we share more detail about reviewers’ understanding of the
program values and the review process.
3.5.1 Interpretation of Program Values

Reviewers were asked to evaluate each of the eligible applications andto explain how confident they were that the proposal exemplified theprogram values, followed the program’s rights and ethical access requirements, signaled readiness to conduct a digitization project, anddemonstrated the applicant’s need for external funding. The panel
seemed relatively comfortable discussing the technical aspects (readiness and need), but while there was much discussion of values, wenoted comments from multiple reviewers that suggest more guidancearound the program values would be useful:
The guidelines were really clear, and it was helpful to havethe handbook that applicants had. That was probably actuallymore helpful than any of the other documents. The one thingthat … [n]eeds a bit more work is the core values, I forget
how many there are, that we are told to evaluate on. I feel
like, for me, really following the spirit of those core values, it
became really hard to distinguish between the kind of middleof the pack applications. It was really easy to identify an applicant that did not have the capacity to plan for a long-term
stewardship. It was really easy to identify an applicant thatreally had that sorted out, but it was hard to score throughthose values and have distinction between people who had
good applications but not stellar applications.
As this reviewer notes, additional clarity about the program’s values, how they relate to each other, and how to identify features that exemplify an application’s strengths or drawbacks with respect to the values would be welcomed. In our conversations with other reviewers, wenoted additional differences in emphasis around the program values:
When I was reading applications, the thing that I was probably closest to, like sort of in the front of my mind, was thecommunity piece always, like, to what extent are you engaging? If you’re not in community, are you engaging community? I think that I always had the other values sort of aroundthat.
My second sort of run [through my set of proposals]
focused on how the work was being done, so what were thepartnerships? That was really, really important to me …, tothink about some of those voices and how people were working with folks. … I always like to see it pretty well thought
out about how people are gonna do this work because otherwise sometimes what you get back is not what you thought
you were gonna get back. So, that was my big concern.
I did tend to place greater weight on authentic partnerships than on representation, but especially in the partnerships just because especially with universities and other large
institutions, there might be lots of great ideas around. The
authentic partnerships, I think is the missing key element a
lot of times. And so, yes, I did tend to weigh that.
Sustainable infrastructures, I guess with this one, there

43Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

were definitely discrepancies in how I ranked more under-resourced community-based projects with institutional projects. A lot of folks applying for this grant from community
archives, they don’t know what they don’t know about how toensure the continuity of digital objects over time. And so that
tended to be a place where those community projects lost
points.
In terms of weighing them all against each other, I kindof weighed them all the same except that fundamental kindof broad representation question like are these histories hidden and will this grant help to make them unhidden?
In general, while we found that reviewers seemed enthusiastic
about the program’s values, their individual approaches to the review
process emphasized some values over others. There did not seem tobe a high level of agreement or cohesion between individuals. Only
one reviewer mentioned deliberately taking a “whole systems programming” approach to evaluating all values. While we would not
expect reviewers to emphasize values equally, it would be advisable
in the spirit of transparency and equity for applicants to arrive at aclearer idea of how reviewers evaluate the degrees to which proposals exemplify program values. It might be helpful to identify specificquestions about engagement with the program’s values that reviewers
want applicants to answer in initial applications or specific characteristics of initial applications that will be competitive for advancement.
In the final section of the report, we explore some ways to cultivate
consensus about values across the panel and ways that a consensus
view may be communicated to applicants.
3.5.2 Review Processes and Support for Reviewers

In general, the panelists seemed highly appreciative of the breadthof the reviewer backgrounds and attention to an open, supportive
discussion. As one panelist observed, “This was a … collegial groupof people who were there because they had real stakes in these materials, and it felt very supportive and everyone seemed to be there
for the right reasons. So, I think that’s ‘cause they were chosen reallywell.” Others applauded CLIR for assembling a representative and diverse group: “I thought that the review panel was quite, quite diverse
as well. … I would imagine it’s not always necessarily the easiest thing
in the world.” In other words, reviewers applauded the outcomes of
the conscientious work of program staff to create a responsive andbalanced panel.
We noted some areas in which reviewers may benefit from additional support. Some requested that information be provided in acentral place, not only by email. A few requested that staff might remind them about how much time the review could take (in terms ofreading applications, writing evaluations, and attending the meeting).
Nonetheless, the considerable time required to read applications waslargely seen as offset by the benefits of learning about new projects
and participating in the conversation: “It was an incredible experience both to read the applications and see how they might fit the corerequirements of the grant program, and then to also participate in theconversations with the other reviewers.” Thus, while the time commitments of reviewing should not be downplayed, many reviewers

44Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

expressed significant levels of satisfaction and reward from being partof the process.
As far as support resources were concerned, a process of norming seemed to occur throughout the two days of the panel meeting.
As one first-time panelist put it, “There was a feeling of like, ‘Wow,
day two felt so different,’ and really like if it was a conversation.” In
addition, at least one reviewer reported feeling that it took a while tobecome accustomed to the panel culture, a challenge that was compounded by the intense scheduling of the meeting: “Probably because
it was in my first year of doing this review work, I didn’t really feel
comfortable and wasn’t able to kind of quickly formulate a response… to address [a comment that I disagreed with] at that time. So I
think also the fast pace of it might prevent people from saying things.”
As we understand this, while reviewers had information about theprocess, the panel seemed to develop a shared sense of how it would
do business, conduct discussions, and interact, which settled in on
the second day of the review.
Given the short time available for the panel review meetings, itwould be advisable to address as much of the panel process and expectations as possible in advance of the meeting. Additionally, since
there was a high percentage of reviewers new to CLIR during theinaugural cycle of DHC:AUV, a greater amount of expertise will likely
transfer between different program iterations, reducing time requiredfor onboarding reviewers in future rounds.
3.5.3 Scoring the Rubric

Panelists offered comments on the initial phase scoring rubric. Aswith the program values, there were divergent approaches to how
the rubric was implemented. As one reviewer stated, “It seemed likewhen we started the discussion in the initial round that some of us
had pretty different philosophies of how we scored.” For first-phaseapplications, reviewers were requested to apply point values on individual aspects of each application according to a rubric. This was
a 50-point rubric, with reviewers asked to provide a value of 1 to 10
in four aspects: program values (10 x 2), rights and ethical considerations (10), applicant preparation and readiness (10), and apparent
need (10). (The value for program values was doubled, emphasizing
their importance and constituting up to 40% of an application’s total
score.) Some reviewers felt that the rubric was too broad:
There’s a lot more room for variation, and then there’s notreally a good way to express what the difference between a six
and a seven was. And I guess in keeping my comments earlier
on, when you’re onboarding people, [if] you say exactly what
those things mean, then that’s helpful, but it’s a lot of work infour or five different categories to elaborate what one means
and what 10 means and everything in between, had we provided or had we been provided with a bit of a more concrete
sort of a system for assigning value, I think that would havemaybe helped.
This reviewer seemed to appreciate the program’s values but expressed that the 10-point scale was unwieldy, not to mention onerous
to determine and apply a score in each of the four areas. Another45

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

reviewer suggested that they appreciated the granularity provided by
the ten-point ranges. The rubric presents a tool that communicates toboth reviewers and applicants a shared understanding of how applications will be evaluated, so we recommend retaining a rubric in future
iterations.

3.6 Direct Applicant Support
Given the new communities that CLIR aimed to reach in the DHC:AUV
program, applicant support was a critical component of managing
the solicitation and review of proposals. As noted above (section 3.4),
many applicants requested additional modes of support, including
phone consultations, draft proposal reviews, and sample applications.
For the initial round of DHC:AUV, staff offered three informational
webinars, posted transcripts of the Q&A sessions from the webinars,
and provided direct support to applicants via email. We used the email
messages with applicant inquiries as our major data source for evaluating current applicant support. CLIR shared with us a set of 476 email
messages from the initial open application period, which covered thetime between the public announcement of the program’s revision (February 2021) and the time when applications were submitted (late April
2021). For reference, we refer to specific messages based on the order inwhich they were received.
These messages show not only the range of inquiries that prospective applicants posed, but also offer insight into the amount of staff
time required to provide direct support. The themes of the messages
varied from mundane requests (such as permission to edit a shareddocument) to complex inquiries regarding applicant eligibility. Whilenot exhaustive, the following general themes were addressed in themessages:• Applicant requests for resources:º At least 10 incoming emails were automated messages sent
by Google Drive to request edit permissions for the Google
Doc that contained the program handbook.
º Staff received many requests for sample materials, including successful applications, budgets, and collection lists.
º Numerous requests for direct consultations with program
staff, including phone meetings, materials review, or videoconferences, were received.
• Webinar administration:º At least 34 incoming messages were generated in response
to automated confirmations of webinar registration or reminders. In some cases, automated webinar messages led tomore substantive inquiries.
• Application system administration:º These messages included answers to questions about theapplication system, confirming receipt of or replacing corrupted attachment files; confirming, changing, or updating
applicant information in the system; or confirming behavior of the system, such as how it calculated word counts,
eligible file types for attachments.

46

47

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking• Detailed applicant questions:º These varied and included questions confirming organizational eligibility; inquiries about the requirements of collection ownership; intellectual property questions; budget
questions or concerns, including questions about allowablecosts; and concerns about why certain types of work were
not allowed.
• Above and beyond assistance:º In one case, the staff identified that an applicant had submitted all the required application materials but under two
separate, incomplete applications. Staff helped to confirm
that these submissions were for the same application andassembled them into a single, complete application.
Just as these themes represent a broad range of inquiries, theamount of time required to respond to the messages was considerable. CLIR staff limited applicant inquiries to email, as they stated in
one response, “due to the volume of inquiries we receive, we are unable to take [phone] calls even for brief questions.” Additionally, CLIR
staff collaborated to reduce time spent on email by allowing multiple
staff to monitor a shared email inbox and reply directly to incoming
messages (at least three program staff were regular respondents).
They also created standard replies to frequent questions, which could
be reused.
Outgoing Support EmailResponse time required

No. of messages

Minimal

11

Small

62

Medium81

Significant

46

NA

3

Total messages

203

Table 6: Emails sent by CLIR staff (outgoing), categorized by response time.
Note: Three messages (“NA”) did not appear to be providing application support.

Despite these strategies to reduce time spent providing email
assistance, the time required was still significant. Using the log ofmessages provided by CLIR, we determined which emails were “outgoing” (that is, sent by CLIR staff) and tallied 203 messages. We thenestimated the staff time required to respond (table 6). We categorized
emails as follows:• A minimal response email was a very brief response of one line
or a few words. For example, staff might confirm receipt of amessage and promise to follow up later; we estimate such emails
would require 5 minutes or less of response time.

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking• A small response email had no more than one paragraph, often
contained a response similar to one in another reply or routed an
inquiry to another staff member. We estimated that such emails
would require 10 minutes or less.
• A medium response email required more than a brief reply. Suchemails might require staff to look up information in the program
policies; read and understand a basic question about a specific,
proposed project or application issue and respond; do researchon relevant resources to recommend to the prospective applicant;compose a multi-paragraph response; or respond to an issue
raised by an applicant that required action in another system(such as the application portal) and provide confirmation to theinquiry. We estimated such emails would require 30 minutes orless.
• Finally, a significant response email included a multi-paragraphresponse with original text (no elements copied or pasted). Examples included researching an organization beyond information
provided in the message; reading and understanding detailed information about a proposed project and providing a detailed response to an applicant about how well their project fit within the
program scope; making detailed recommendations about how toformulate a specific activity within a proposed budget; researching potential funding programs that an applicant might want toconsider in addition to CLIR. In many cases, a significant email
reply appeared to require additional time to coordinate with other staff. We estimated this kind of response would require up to
60 minutes.
Although a rough estimate, we suggest that staff devoted nearly
100 hours (97.8 hours) hours, or 2.5 weeks of one full-time staff
member, to managing email assistance for DHC:AUV over thecourse of three months. Given a program staff of three in 2021, thiswould account for nearly 7% of staff time to provide direct applicant
assistance (assuming total staff time of 1,440 hours over 12 weeks).
CLIR staff noted that the time required to respond to email inquiries
increased significantly as the application deadline approached. Given
the frequent requests for other time-intensive assistance from applicants (such as draft reviews and phone consultations), a significant
increase in staff time and resources is likely to be required if the program aims to expand applicant support.

48

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

4. Areas for Attention andRecommendations
We found that the first iteration of the DHC:AUV program elicited
a positive response and holds a durable interest among many stakeholders, including potential applicants, applicants, and reviewers.
Overall program accessibility, the appeal of the call for proposals emphasizing underrepresented perspectives in collections, and support
for digitization were enthusiastically welcomed. Nonetheless, there
are many program areas that could benefit from further attention andrefinement if the program is to serve its stakeholder communities
more effectively while also ethically supporting the expansion of adiverse, digital historical record. In this section, we focus on specificprogram areas that may require attention and make recommendations that address some of these areas.

4.1 Allowed Activities
While DHC:AUV does allow work beyond digitization, applicants
are advised that digitization should be the focus of their proposals.
In the course of our assessment, however, we noted additional activities that would be of interest as the program aims to broaden access.
Numerous applicants expressed concern that they had higher needs
for collection description and processing to fully engage with the program’s emphasis on underrepresented perspectives. Others noted theimportance of redescription, creation of new metadata, or “reparativedescription,” as a resource-intensive but also critical activity to amplify voices in collections that may have been previously described frommainstream perspectives. In addition, some described a need for additional work to select or identify materials within collections, especially if collections were held in the collections of large institutions orwere organized in fonds representing groups already well-represented
in historical collections.
4.1.1

Support for Reparative Description

Since many collections have been described from the point of viewof dominant groups, archivists and others have noted the importanceof updating descriptions to better represent perspectives and groups
that have not been mentioned or effectively made invisible in catalog
records, finding aids, or similar tools for discovery.
The Society of American Archivists (SAA) defines such “reparative description” as metadata creation that addresses “practices ordata that exclude, silence, harm, or mischaracterize marginalized
people in the data created or used by archivists to identify or characterize archival resources” (SAA 2022). Recognition of the importanceof such work has grown in the past few years, including the establishment of practices for initiating reparative projects and numerous
case studies (Hughes-Watkins 2018, A4BLiP 2019, Dean 2019, SSDN
2020, Frick and Proffitt 2022).

49

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

DHC:AUV should consider designating support for reparativedescription or redescription of collections. This kind of work is not
only a step toward reducing systemic bias in collection descriptions,but also an area of growing interest. Such support would be particularly apt, given the DHC program’s strong support for description andcataloging over time (Banks 2019), which could be programmaticallylinked to digitization activities. In other words, while the emphasis of
the program can remain on digitization, reparative description could
be explicitly requested or defined as an allowable cost.
One applicant pointed out how critical metadata work was fortheir process:You’ve got to have metadata before you can really digitize
things to be able to do it in the large batches that we’ve got.
… And quite frankly, people that don’t have folks that understand the metadata side of things enough to address social
justice concerns in that space could be at quite a disadvantage. … Particularly in a community-based setting, working
with community members and subject matter experts in thatspace to be able to bring in their expertise and help them
be a part of the process to make sure that any kind of harm
wouldn’t be perpetuated, is something that, at least, we’ve
started to talk about and think about in different ways.
A second applicant noted they could not separate the descriptive
work from their digitization projects:
It’s not just about digitizing hidden collections ‘cause it’s verymuch about the descriptions. If you don’t describe it in a certain way, you will not find it. … This is also why we … havestarted this project on reviewing our already existing descriptions because we know that they are, let’s say, to a certain extent colonial or they are done with a certain intention, whichleaves out narratives and voices. … We try to make a case forthe interpretation of the collection. [An object in a collectionis] not just hidden because it wasn’t digitized but also it maybe is digitized but it’s not described properly.
These comments illustrate our observation that organizations
taking deliberate steps to ethically recognize silenced communities
documented in records may require significantly more, and perhaps
qualitatively different, descriptive work than projects focused directlyon the creation of digital surrogates. Unless the program is only open
to applicants that already explicitly document the cultural heritageof marginalized peoples, the creation of digital collections without
support for related reparative work would likely reproduce systemic
exclusion of outsider perspectives, effectively continuing to mute unheard voices, rather than amplifying them.
4.1.2 Identifying and Evaluating Collection Strengths

Traditional collection management conventions, descriptive practices, and colonial descriptions hinder DHC:AUV’s aim to make thedigital historical record more broadly representative. Archival andmuseum collections in particular have been described and catalogedwith high attention toward provenance, which typically involves orga-

50

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

nizing materials according to the categories or organizations of dominant social groups. This has resulted in many materials being associated with dominant categories and organizational schemes without
attention to their relevance to other experiences or ways of thinking.
For this reason, underrepresented groups can be doubly “hidden”
in collections—that is, documented in collections that are minimallydescribed, difficult to access, and discoverable only through records
created from mainstream perspectives. Additional work to identify
and select content that documents underrepresented communities
may be required prior to digitization, to enable collections maintainers to focus on the histories of those communities. An archivist froma community-based archive described some of the challenges of amplifying unheard voices within collections created by dominant socialgroups:Most of our collections are donated from community members who identify as cis and white, and so a lot of the materials of BIPOC are in these collections and they are good. Someare unprocessed, some of them have been processed, but …
no one has been able to bring those materials out, to bringthat narrative out, to piece it together.
To begin preparing “unheard voices” collections for digitization,
then, CLIR may also consider directly supporting work to identify
collection strengths and evaluate representation within collections. Some level of collection work or processing could be allowed
for within the scope of the grant activities; for example, smaller award
amounts could support initial steps such as the creation of community advisory groups, locating materials within collections, supporting
community-based researchers, or identifying themes and confirming
representation in collections.

4.2 Applicant Support
For the initial round of DHC:AUV, CLIR provided considerable
information and resources to assist applicants in preparing submissions. This support included program information posted on the
CLIR website, a series of three information webinars prior to the initial application deadline, and a series of six webinars for those invited
to the full application phase. Below, we explore some areas in which
CLIR may consider increased support for applicants while also building capacity for less-frequent grant seekers.
4.2.1 Increase Direct Applicant Support

Several applicants pointed to the need for more time-intensive support mechanisms, such as phone or video consultations with program
staff (before and after deadlines), preliminary draft reviews, and unstructured “office hours” to speak with CLIR staff.
Similar ideas were also suggested by reviewers. As one put it:
Sometimes the way that we do things in terms of writtencomments is not particularly inclusive or sensitive. … Insome instances, panelists … were really clear that they wanted to invite these people to apply again, and in other cases
they weren’t, and my position was …, everybody should try

51

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

to apply again, but we should provide them with the support that they need to do it better. …Not surprisingly, thestrongest … technical applications were from big institutions—people that already have capacity—but that’s the rub.
If you’re from a community where you’ve got four volunteers
essentially trying to get this off the ground, and our comment
is, like, this should be a little bit more sophisticated or sleeker,
that’s not helpful, right?
[CLIR should think] about ways to better prepare people
to apply or to re-apply through guided sessions or virtual grant writing workshops. These are all really importantskills, and the reason that underrepresented groups are often
underrepresented is because they don’t have the capacity or
time--or money, really—to hire someone to do all of thatwork. … If the idea is to be more equitable in terms of including underrepresented groups, then we have to think
about … how we communicate…—like the medium—andthen also the message that we’re providing and how do weopen the door.
This reviewer noted that written comments may not always be themost inclusive or sufficient approach to communicating feedback toapplicants coming from smaller organizations or underrepresented
groups. Another reviewer elaborated, “Especially with Native American communities—it would probably apply to others as well—it
would be helpful to be able to have some conversations and not [for]everything [to] just be written feedback.” Since the program is now
open to Canadian applicants and groups that may not identify English as their first or primary language, it may also be important toconsider multilingual program support in at least French and Spanish. As a possible corrective, CLIR should endeavor to offer more
tailored, one-on-one feedback via phone, video chat, webinars, orworkshops. As discussed below, this will likely entail significant time
investments from program staff, so it is a recommendation that could
make the program more appealing to some applicants but one thatwould also require increased resources. As indicated in our analysis
of applicant support emails, provision of this level of support requires
both time and different kinds of expertise. If DHC:AUV aims to offer
this level support, this should be accompanied by an increase in program staffing, which we discuss further below.
4.2.2 Make Information Available Earlier

As the program moves ahead, CLIR should offer more information
about the funding opportunity farther in advance of the initialdeadline. This would not only promote effective planning for collaborative projects but would also assist less-frequent grant seekers in preparing submissions. In the non-applicant survey, multiple applicants
noted that the amount of time between the program’s announcement
and the initial application due date was a challenge. We would suggest making available more robust support and program materials atearlier points. This may include offering more preparatory webinars
to applicants prior to the initial deadline, specifically like sessions
hosted in 2021 for full-phase applicants covering the structure of theapplication, collection assessment as well as pointers on intellectual
property requirements, rights, and ethical use and access.

52Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

4.2.3 Information for Less-Frequent Grant Seekers

Because the program aims to better serve organizations that are
less-frequent grant seekers, it would be useful to offer additional
context for how CLIR structures the evaluation of applications andthe award selection processes. Although the review panel is listed on
the CLIR website, and it is referenced directly in the applicant handbooks, it was evident that many applicants did not have a clear understanding of the review process, who the reviewers were, or what thevarious stages of the process entail. We would suggest that CLIR may
offer additional explanations to support applicants, specifically aprocess diagram and a useful terms list.
A process diagram could illustrate the various steps that a submission would advance through in a typical cycle, serving as a visual
representation, or map, of the process. This would be helpful not only
to applicants but also to reviewers in helping them understand how
their feedback can be most useful (figure 6).

Fig. 6: Process diagram illustrating the application and award process

A terms list could clarify basic roles and concepts in a single
place. Many common terms employed by grant makers are not easy
to understand without additional context, and terminology can varyslightly among funders (for example, “partner organization” may havedifferent meanings for different kinds of applications). In addition, it
would be helpful to have these terms compiled in one location rather
than being dispersed throughout program documentation.

53

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

PROGRAM TERMINOLOGY
The following terms could be considered for the initial list of program terminology,
which would be especially useful for less-frequent grant seekers:• Principal investigator (PI) is an individual who takes direct responsibility forcompletion of the project, should funds be awarded. A maximum of three PIs isallowed, regardless of the number of collaborators on a project.
• Application contact is an individual who is not the PI but should receive communications regarding the application. In many cases, this is a local grants administrator, project manager, or finance officer. This is an optional position to
include in the application.
• Applicant organization is the organization applying for the grant. This is theorganization that would accept payment, should the proposal be selected forfunding, and would assume fiscal responsibility for the project. If multiple organizations are working together equally, one must be designated as the lead applicant organization.
• Collaborating organization is a partner in the project who also holds collection
materials and will be required to agree to the collaboration agreement and also
sign the intellectual property agreement, should the application be funded. Theseare explained and specified in the budget appendix of the Applicant Handbook.
• Allowable costs are expenses that are eligible to be requested in the project budget.
• Disallowed costs are expenses that may not be included in a project budget.
These are explained and specified in the budget appendix of the Applicant Handbook.
• Budget is a document that itemizes each area of cost in the project and tallies
the amounts to provide an explanation of how a proposal will use funds if it isawarded a grant. CLIR provides a template of cost categories that should be used
to structure the budget.
• Budget narrative is a document that accompanies the budget and justifies thecalculation of each item (e.g., how fringe benefits are calculated or what rates
used to calculate travel costs).
This list is not exhaustive but reflects some of the application terms and concepts thatwere frequently mentioned in our interviews with applicants.

4.3 Application Process
The feedback we collected about the application process was generally
positive. In particular, applicants praised the clarity of the program
documentation and usefulness of the Applicant Handbook and interactive documents provided as templates. The most frequently heard
feedback from potential and actual applicants was that the time between the announcement of the DHC:AUV call and the application
deadline was a challenge. As one museum applicant stated, “It would
be nice to have a little bit more time between the information session
and the due date, if that same level of detail was going to be required.
… That would be a very acceptable time frame if it was … a more
traditional kind of letter of intent, or a slightly more detailed letter of54

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

intent, but this was a rather intensive initial … proposal stage.” Wewould point out the following elements of the proposal process forwhich further refinements may benefit the program.
4.3.1 Shorten Initial Application

Additional steps to reduce time required to complete the initial application may make the program more accessible, particularly if thegoal is to increase responsiveness and effectiveness of the program forsmaller collecting organizations and community-based organizations.
Changes to this aspect would also follow recommendations from theTrust-Based Philanthropy project, which suggests that funders reduce
paperwork demands on applicants, when possible, to lower barriers
to less-frequent applicant organizations (Trust-Based Philanthropy
Project 2021; Wright 2021). Reviewers also suggested that a moreconcise application might encourage a broader array of applicants: “I
do think having a shorter application in the initial round does make ita lot easier, and hopefully attracts more of an array of applicants.”
The complexity of the initial application may have led to a greater
focus on technical aspects of proposals in the initial review, such as
budget, staffing, and project planning. As one reviewer noted, the request for a full budget with initial applications in 2021 tended to give
reviewers the impression that they should be evaluating the budget
details:
When a person is given a budget, they may be automaticallylooking for that budget to be 100% complete, but with the
new application process, it could just be a whole different setof why we are asking for this budget and what type of budget
we’re looking for at this phase.
If the initial application is shortened, CLIR may request less detailabout the technical elements of the application. An initial application,for example, should introduce the collection proposed for digitization, describe how the project would advance the program’s values,
and offer a high-level statement of how long the proposed work
would take, who would do the work, and what resources would be
required to complete the project.
To reduce focus on a proposal’s technical elements, CLIR may
consider eliminating attachments from the initial application altogether, instead requesting shorter, prose descriptions of the project
timeline and budget. Unlike other programs, such as NEH Humanities Collections and Reference Resources where the expectation isthat funded projects be “shovel ready” (that is, ready to begin work
as soon as an award is made and to conform to the highest levels of
broadly accepted, professionally endorsed standards), DHC:AUV
may place more emphasis on building an inclusive historical recordand building the capacity of cultural heritage organizations to promote that goal. Since these projects may not yet be fully planned, ahigh-level, narrative budget and project timeline may offer enoughdetail for reviewers to assess the reasonableness and potential feasibility of a project given the applicant’s grasp of their current capacity
and the resources they need. Such potentials would form the basis foran organization to develop a full budget and project plan—with guidance from applicant webinars and resources—if invited to submit in
the full application phase.

55

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

More radically, these “technical” sections (i.e., budget, timeline,
preservation plan) could be eliminated altogether from the initial
application, drawing more focus to an organization’s collections andpotential to help diversify the historical narrative. This may require
further clarification about the functions that the first phase of review
is intended to serve. Abbreviation of these elements could make sense
if CLIR and reviewers agreed that the goals of the first stage are tounderstand the voices amplified in the collections and the potentialof proposed actions to contribute to the program’s goals. If there is aperceived need to address planning, staffing, and budgets at this stage,
however, it may not be prudent to drop these sections altogether. Essentially, CLIR needs to answer the question of whether it matters at
the initial stage whether an organization is ready to undertake digitization or not.

4.4 Review Process
Overall, reviewers and applicants shared positive assessments of thereview process. The panel process worked smoothly and equitably,and it provided informative feedback for applicants at both the initialand full stages of the process. Below, we discuss numerous aspectsof the review process, which were mentioned by reviewers during
focus groups or developed from our analysis of the assessment data.
The panel discussion, currently emphasizing the relative merit of theentire pool of proposals, may be refined based on program themes or
priorities, such as “unheard voice” groups, regions, or organization
types.
4.4.1 Panel Process

Responses from reviewers and applicants suggest that the panel process has largely produced fair and equitable feedback. At the sametime, given the panel’s potential to assist in assuring broad representation across applications that are invited to the full stage and, ultimately, to receive funds, we would suggest exploring ways in which
the panel may also embed the program values in its process. Potentialchanges include:• Establish clear understanding of the phases of the entire review process for the panel, and with panel facilitators andpanelists, establish expectations for each stage of review. We
noted in a few conversations that the overall structure of theprocess and outcomes for the review meeting were not clear. As
one reviewer noted, “I actually wasn’t clear on how much they[the initial applicants] were expected to have a real detailed preservation plan. Or whether that would mostly be something thatwould come out in … the final round.” It is a challenge for reviewers, who generally have short, intense relationships with the
grant process (that is, only for a few weeks as a reader for applications and an additional few days as a participant in review panel
meetings) and are under other demands, to recall all details of the
program structure. In this spirit, panel co-chairs, program staff,
and reviewer orientation materials should emphasize even morestrongly the overall application, decision, and award processes
and the place of the review panel within it (see also section 4.2.3).

56

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Along with streamlining the application, removal of attachments
might also reduce the perceived expectations to comment on every detail of a proposal during the initial round.
• Ensure contextualizing or explanatory remarks are provided
as feedback for each proposal. This may help to convey a greater
sense of cohesion around program decisions, particularly when
individual written comments are limited or misaligned with one
another. To do this, the panel process could designate a single
reviewer as a primary reviewer for each application; this reviewer
would take notes and compose a short “panel recommendation”
or explainer of what happened during the panel deliberations.
Where possible, this statement should explain reasons for notadvancing to the next phase (or receiving an award). We notedsignificant variations of detail among different reviewers’ writtencomments, and sometimes between comments from the same
reviewer, which suggests all proposals do not receive comparable
levels of feedback.
One drawback of this recommendation is that this would necessitate significant further work on the part of each reviewer. If
such an approach is implemented, we would advise panel facilitators to integrate feedback writing or revising as part of the panel
meeting agenda since reviewers have already blocked off thistime. For example, many in-person review panels for proposals
to the National Science Foundation (NSF) assign a lead reviewer
for each application whose role is to guide discussion for assigned
applications, as well as a scribe for each application, whose role
is to record and summarize all discussion about assigned applications. The goal is to ensure comment-writing is shared among
the reviewers. The discussion leaders and scribes encourage aconsensus recommendation for each proposal, and each proposal
discussed is guaranteed to receive a summative panel comment(NSF 2022). Instituting this or a similar approach would require
additional staff support to organize the meeting in advance andto set up appropriate orientation and guidance to prepare reviewers to perform these roles.
• Address “split decisions.” Split decisions are cases in which
reviewers have antithetical opinions or conflicting evaluations.
Applicants reported confusion or frustration when receiving
comments that appeared to be contradictory, and it would be useful for the program to institute a mechanism to provide clearer
explanation of these situations to applicants. In the words of one
frustrated applicant who received conflicting responses about thepanel’s expectations: “It would seem that the reviewers did notconsult with each other. Or how else would they have put thosecomments in without reading those of the others that had said
that they approved? … I just don’t understand.” Beyond frustration, split decisions raised questions about the panel’s integrity:“Our whole team is convinced that reviewer one … sunk us.
And from my point of view, reviewer one should not have been
considered … because they clearly didn’t know what they were
doing.” In response, a “panel statement” might explain that the
proposal was carefully considered before reaching a decision.

57

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking• Consider alternative decision communications. The current
DHC:AUV structure offers binary results: an initial proposal iseither invited to make a full proposal or not invited; a full proposal is either funded or not funded. While we are not sure there arealternative outcomes in the current process, these could be communicated in ways that suggest additional options. For example,
feedback might include strong and clear encouragement from
CLIR staff and panelists to resubmit proposals in the future, or astatement characterizing an application was highly recommended
but not funded. Other decision processes and feedback processes
were described earlier by some applicants (see section 3.4.5.3).
• Structure the panel discussion to de-emphasize collection merit and focus more closely on program values. At the June 2021
initial panel meeting, applications were ordered in the discussion
starting with most highly ranked proposals, which guaranteedthat applications on which panelists were in highest agreement
received the most expansive, least-hurried consideration. This
start established panel rapport, but we would suggest ordering
proposal discussion in other ways, such as by key groups (e.g.,
Native American/First Nations applicants) or clustered by regionor according to applicant organization types. We think such ordering would help surface issues related to broad representationand possibly public knowledge, in ways that highlight these components. While there is not a direct correlation between orderof discussion and overall award profile, we believe that ordering
the discussion according to factors other than merit would leadto discussion of benefits of promoting proposals based on thosecharacteristics in addition to reviewer agreement.
• Hold an open discussion or orientation for reviewers regarding bias prior to review panel deliberations. Discussion ofand reminders about the potential for biases to enter the review
process, whether explicitly or implicitly, is a best practice thatwill help to raise awareness about possible bias areas and can be
repeated in guidance to reviewers and before panel discussions.
It is important to acknowledge that all reviewers carry biases. Reviewers may be reminded to avoid multitasking while reviewing(time and task pressure can amplify biases), and panelists may
be reminded of implicit or unconscious biases (Wigginton et al.
2021). (Note: This recommendation was implemented in June
2021, and we received positive feedback from reviewers regarding
this aspect; see Appendix A.)• Create guidance to reviewers for composing constructive
and actionable feedback for applicants. This guidance may bebased in part on findings from this assessment. As part of thisrecommendation, we would propose a feedback and focus groupsession to discuss and create additional resources that may assist
the panelists in arriving at clearer shared concepts, language, andevaluation criteria relating to the program values (see Appendix
B and Appendix C).
• Time management. The panel consistently felt time pressure.
As one panelist stated in a focus group, “I did feel that kind of
panic-anxiety of clock is ticking.” This poses risks to the panel’s

58

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

decisions since time pressure and limited information are likely
to result in situations where decisions are taken without the acknowledgement of implicit biases or mistaken assumptions.
• Number of reviewers assigned to each application. In focus
groups, some reviewers suggested that assigning three reviewers
per application was too few. With such a small number of perspectives, a negative evaluation from one reviewer could “sink”
an application, or conversely, if one reviewer is very topicallyaligned with or particularly passionate about a project, that could
disproportionately elevate an application. Assigning five reviewers per proposal would result in additional panel voices that areclosely familiar with a project and create a more coherent “panel
voice.” Challenges to this approach are that increasing reviewers
would extend the length of an already full panel discussion, andmoreover, recruiting, orienting, and compensating additional
reviewers would require increases in the program’s staffing andoperating budget.
• Assign reviewers according to expertise. Some reviewers reported that they felt they were assigned to applications somewhat
at random. In other words, while they might have a topical expertise, they did not always receive applications related to that
subject area. To the extent possible, we would suggest that CLIR
consider matching reviewers to proposals by expertise or topic
area or providing additional context to reviewers about their roleand the way that applications were assigned. As noted in the preceding recommendation, addressing this concern implies greateradministrative efforts from program staff in the management ofpanel relationships and assignment of applications.
4.4.2 Panel Membership

Given that notable applications were received from public libraries,
we would suggest that CLIR include at least one member on thereview panel who represents or works with public libraries. In
interviews, applicants affiliated or collaborating with public libraries
expressed that they felt equitably considered within the review, butthey also shared a sense that they had more capacity, technically aswell as in outreach to local communities and historical programming,
than was readily apparent to CLIR’s panelists. All mentioned robust
programs for digital collections and local history, which could be wellaligned with the goals of the DHC:AUV project.
4.4.3 Specific Feedback Areas

Responses from some applicants suggested that it would be helpful inthe initial round to structure feedback given to applicants so thatit addresses specific aspects of the proposed project as understoodand assessed by the review panel. This would be additionally important if the application is shortened as recommended above. See, for example, this feedback from an initial applicant who is an experienced
grant seeker:[I would like to see] the set of questions or the parametersthat the reviewer is looking at … in a condensed form: …
clarity of the mission, how close were you aligned with the59Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

amplifying unheard voices mission … and maybe just things
like that. How clear was the digital aspect or the core part ofit. … That … would help everybody.
This applicant suggests specific areas in which they would likefeedback from the panel, including how closely the proposed work
fits the program, to what extent the proposed project would advancethe program’s priorities, and what additional technical questions thepanel would like answered if the applicant is invited to proceed witha full application. To facilitate effective review and consistent comments, we developed a set of principles and additional resources forreviewers as well as a reviewer rubric to encourage consistent evaluation of applications (see Appendix B and Appendix C).

4.5 Award Process
If DHC:AUV is to serve less-frequent grant seekers and community-level applicants more effectively, the program’s approach to intellectual property and collection ownership requires further attention.
4.5.1 Intellectual Property and Ethical AccessMultiple applicants and recipients raised concerns regarding thecurrent DHC:AUV approach to intellectual property, rights, andethics. The review process appears quite sensitive to this area of concern, and the panel includes two ethics and copyright specialists.
The award process, nonetheless, continues to require all recipients tosign an intellectual property contract, which currently establishes alegal mechanism for CLIR to license any “digital copies” created with
grant funds if a grantee fails to store and provide access to digital files
created through funded projects. This was designed to enable CLIR
to rescue digital assets, should an organization dissolve or become
unable to meet digital preservation obligations. Our findings suggest,
however, that this agreement, as applied in DHC:AUV, is perceived asincongruent with the program values of authentic partnerships andcommunity-centered access.
The current agreement assumes that each DHC:AUV lead applicant, as well as any collaborating organizations, owns and controls allcollection items nominated for digitization. Multiple inquiries to theprogram included requests to confirm this prior to the initial application deadline. The concern underlying these questions was largely
motivated by the requirements of the model agreement to make stipulations concerning: (1) the “right, title, and interest” to any digitalcontent created through the program, which may not be possible ifcommunity members hold claims over the possession, licensing, oraccessibility of contents or items; and (2) that CLIR or its designees
be granted a “perpetual, irrevocable, [and] nonexclusive” license to all
digital copies created with award funds.
Multiple sources raised concerns about this requirement. For example, in an email inquiry, a potential applicant noted that they could
not apply if they had to require community participants to sign theagreement:
The IP agreement is an insurmountable barrier to our application. … We could not in good conscience lead communi-

60

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

ty-based organizations to sign an (eventual) IP agreementthat indicates they have acquired all permissions to avoid infringement of publicity, privacy, or copy rights. We … cannot
state as fact that we have permission from the rights holders
to digitize every single item.
The broader issue confronts many archives and is particularlypressing for community-based organizations: often the holding repository cannot in good faith assert that it holds legal rights to distribute or license digital versions of collections.
A second concern was raised by applicants collaborating with
Indigenous groups. They noted the program’s current requirements
may cause Indigenous communities to avoid participation while also
making it difficult to build collaborative projects:
The problem is that they’re looking to a legal structure thatis violent and coercive and implements hierarchy … [The IP
requirement] is going to damage relationships at the very
least with Native nations. … Those are sovereign nations[with] particular histories of collection that almost guaranteethat some kind of an intellectual property agreement where
they’re re-assigning ownership is going to seem like a re-colonization. … Taking it further, you’re going to marginalize communities who may want to [apply to DHC:AUV] but thenalso feel threatened. What if it’s immigrant communities?
… If we default to these things needing to be accessible andpublic, we don’t know in the future—some of those things—it
may be very harmful for the people for them to be open.
Such rigid structures of control, particularly when associated withthe history of extraction and surveillance faced by many historicallymarginalized groups, do not align with the DHC:AUV program values, nor would they be useful in forging authentic and ethical partnership between source communities and collecting institutions. As arecipient representing a First Nations museum put it: “The language
was quite unsuitable for our context.”
With the DHC:AUV emphasis on underrepresented voices, werecommended that CLIR and the program’s primary funder (the Mellon Foundation) revise the program’s approach to intellectual property. Currently, the program requires all recipients and collaborators
to sign a legal, preset contract governing intellectual property. While
recognizing the need for terms and conditions in a funding arrangement, any program attempting to reach community-based memoryorganizations will be hard-pressed to do so while also requiring eachcollaborator to guarantee “right, title, and interest.” Community organizations may not own all materials in their collections, or they may
steward only digital versions of physical items. Most concerningly,
some applicants perceived the agreement as a mechanism to removeor alienate collections from communities or community partners.
Various alternatives have been suggested and endorsed by professional societies and legal scholars, such as:• The encouragement of access restrictions where appropriate andin consultation with source communities, as expressed by theFirst Archivist Circle through the Protocols for Native American
Archival Materials (First Archivist Circle 2007) recently endorsedby the Society of American Archivists

61

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking•••

Use of culturally guided indicators for contextually appropriate
access to materials, such as the Traditional Knowledge Labels(Anderson and Christen 2013) for collections and CARE principles for data (Carroll et al. 2021) (See also TAVP 2022)
Instead of copyright assertions, suggestion of appropriate rights
statements, as has been promoted through initiatives such as
RightsStatements.org and Creative Commons (see Fallon 2016)
Requiring applicants to adopt informed approaches to copyrightand ethical access, such as the open copyright education materials developed by and for cultural heritage organizations recently
pioneered by the Open Copyright Education Advisory Network(OCEAN 2022)

We recommend that DHC:AUV recipients be allowed to waive
the current requirement to sign an “intellectual property” contract.
This has been perceived as a mandatory mechanism to license digitalcopies created through funded activities to CLIR or its designees inthe event an award recipient can no longer preserve or sustain access to the copies for research purposes as described in the proposal.
Instead, we would encourage the program to move toward othermodels of agreements respectful of community notions of ownership and access. For example, recipients could be required to sign amore general indemnity agreement or terms and conditions to formalize the grant arrangement.

4.6 Program Values and Voice Groups
Our conversations with applicants and reviewers indicate that the
program values and emphasis on unheard voices are highly appealing. The revised program was recognized as a critical funding
resource that joins support for collections digitization with social
justice interests, clearly expressed in the program values, which will
positively benefit the preservation of and access to more representative collections and records. In response to the call for proposals,
our analysis of the applications received suggested a good level of response from collections and organizations stewarding materials that
hold materials documenting underrepresented perspectives. To better
understand and assess the level to which the program is expanding
broad representation—and to a lesser extent public knowledge—wesuggest that it would be useful to enumerate and code applications
in ways similar to the “unheard voices” groups that we identified
earlier, based on the underrepresented groups noted in the application materials. Likewise, it would be useful to monitor invitation
rates as indicators of the program’s effectiveness in supportingcollections that document underrepresented communities. (Seesection 3.1 for additional details about these indicators.)
While all the voice groups identified would benefit from further
representation in future application pools, feedback from reviewers
and our analysis of the received applications suggested that increased
representation of the voices of incarcerated populations and disability
communities should be considered.

62

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

4.7

Program Administration

Many of the above recommendations would require additional staff
time and resources. The current program was managed by a team offour (three program officers and a grants manager) who reported toa CLIR director-level supervisor. While we would leave the specificconfiguration of the staffing to CLIR, we conclude these recommendations with some suggestions about areas of the program that might
benefit from an additional program officer or program coordinator.
Various activities identified in our findings suggest additional investments beyond current resources:• Provide enhanced, direct support to applicants to develop capacity, hone applications, and refine ideas. As noted in our analysis
of applicant support emails (section 3.6), the time required toprovide support via one channel is already significant, and anyof the additional approaches requested would require additional
staff resources.
• Expand the staff role in panel administration, including additional panelist recruitment, relationship management to facilitate
the pairing of reviewers with particular application groups, thecoordination of panel scribes, and myriad additional activities
described in the panel process recommendations.
• Reach out to community organizations to develop relationships
with potential applicants and reviewers.
• Create and maintain additional orientation materials for reviewers.
• Plan, produce, and host new applicant support webinars.
In addition to these areas of program administration, we would
suggest CLIR may consider an additional staff role that functions
as a program manager or senior program officer. This recommendation is offered as one way to provide a more coherent voice for thereview panel. Specifically, this person could chair the review panel, summarize panel decisions, and manage communications withawardees and applicants on behalf of the panel, assign applications
according to panel expertise, and be a panel moderator responsiblefor interpreting conflicting recommendations from the panel. This
would not necessarily supersede the current arrangement of panel
chairing, but it would provide a stronger voice for program policy
and assist in shaping the program.
In our discussions with panelists, we observed that each individual is an excellent evaluator of individual proposals or even small subsets of proposals, but it does not seem reasonable to leave the shapingof the program’s overall award profile to a group that only meets twice
per year. Vision and shaping of DHC:AUV’s policies, priorities, andaward profile requires a sustained and consistent engagement withthe program throughout the year.
For the reviewers, then, a program coordinator would provide a
staff voice to the panel. For applicants, this position would serve as asort of third party who can speak for the panel. Specifically, it would
be useful to have a staff member who could speak with the authorityof the panel.

63

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

A senior-level staff member would help to explain and strategize
how the program values are promoted and implemented throughout
the program. Our focus groups with reviewers suggested that whilesome values are applicable at the level of an individual proposal,
others are program wide. For example, public knowledge and broad
representation are program level: no single application should besolely responsible for advancing these values on its own; they are exemplified through the cohort of funded projects. On the other hand,sustainable infrastructures, community-centered access, and partnerships may be exemplified in each application. Therefore, the senior
officer would be highly responsible for articulating how and why thefunded projects promote public knowledge and broad representationthat can only be promoted through the aggregate.

64Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

5. Conclusion: Successes andChallenges
To close this report, we discuss some of the notable successes andchallenges of DHC:AUV’s initial implementation. These are based on
our findings and areas for attention noted previously as well as ourown perspectives on DHC:AUV after a year of observing the program
given our experience as cultural heritage practitioners, researchers,
and funders.
While we made use of brief, quantitative analyses of some program elements, we are particularly hopeful about the ways that thereport shares and illustrates the applicant experience of a cultural
heritage grant program. While many funders assess their programs,
we find it rare that applicant experiences are recorded, and we hopethat the interview-based approach that we took brings to light thevoices of stakeholders who are not always available to be consulted orcontacted when planning or revising funding programs.
Applicants and recipients alike frequently expressed high levelsof trust for CLIR and high enthusiasm for the DHC:AUV program.
Many stakeholders noted the value of the program’s continuing support for collection digitization, with the added emphasis on increasing representation. Interest among applicants was one illustration ofthis enthusiasm. Even initial-stage applicants who were not invited tosubmit full applications frequently communicated that they hoped tosubmit revised applications in future rounds of funding, if available;
these applicants usually cited reviewer feedback as a critical tool intheir work to revise proposals. And of the group of “non-applicants,”
more than half hoped to submit applications in future competitions.
The most frequently reported barrier for these “non-applicants” wasthe short amount of time between the announcement of the new program and the due date of initial applications.
In addition to enthusiasm, the program supports useful capacity-building among applicants. These are particularly notable in thewebinar series and informational resources available for applicants
via CLIR’s website. This aspect may be strengthened by providing
more complete feedback to applicants, clearly explaining application
shortcomings (if any), providing actionable remedies, and a mechanism for staff to offer clearer explanations of program-level decisions.
Additional workshops, moreover, could be made available to all applicants and potential applicants, rather than primarily to those invited
to submit full applications.
Numerous applicants also reported benefits of the program inarguing for improved recognition of the labor required to produce,
maintain, and make available memory materials and heritage collections. In multiple cases, we noted that the opportunity for increased
program funds along with feedback from the review panel werecrucial in arguing for higher salaries, greater hours, or additional resources to support collections work. This appears to advance related
movements for more responsible labor arrangements within projects
that rely on term-limited workers (Arnold et al. 2020, Rodriguez et65

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

al. 2020, Baines et al. 2014). At the same time, we heard clearly fromsome community-based applicants that such expectations were challenging since in some cases, wage expectations were higher than they
could realistically manage as an organization.
We would also highlight some of the structural challenges that
arose. At a basic level, while the intention to support community-based organizations through DHC:AUV is laudable, built-in
challenges accompany the project grant mechanism. Various community-based memory organizations have suggested that they would
benefit greatly from unrestricted operating support or types of funding that do not come attached to specific deliverables (see Jules 2019;
Ferraiolo 2019a; Caswell, Harter, and Jules 2017). In addition, manylarge organizations have built up structures that place them at anadvantage, including dedicated staff or resources that support efforts
to seek out, apply for, and manage grant-funded projects. While theapplicants from community-level organizations that we spoke withrarely had in-house grant writers and administrators, most of theindividuals from research institutions that we spoke with had robust
support for applying for and managing grants like DHC:AUV. Someof the concerns raised in the “non-applicant” survey reflect the factthat smaller organizations are often hesitant to pursue project grantsthat may entail high administrative costs for the organization.
The roots and development of DHC:AUV from the Cataloging
Hidden Collections and Digitizing Hidden Special Collections andArchives programs (see Banks 2019) suggest that project grants arean aspect of program continuity. We encourage CLIR to continue its
mindful approach of listening to applicants, responding to needs andconcerns as they are reported, and expanding upon the considerable
capacity-building resources for applicants and recipients. Along withthese efforts, the program’s website informs potential applicants, who
may not be interested in this sort of funding mechanism, of related
funding programs, some of which follow other models. Looking at
the program’s inaugural cohort of recipients (CLIR 2022), it seems
clear that a project-oriented funding mechanism can effectively support some categories of projects, such as large collaborative projects
involving community-based partners. We would not expect to see a
one-size-fits-all approach to funding in a sector like cultural heritage,
which is a relatively coherent domain of activities but carried out by aconsiderable diversity of organizations.
In concluding this program assessment, we are simultaneously optimistic about the program’s possibilities but also aware of thesignificant work required to maintain and improve cultural heritage
funding programs. We are glad to note the high enthusiasm for increased support for memory activities that will diversify the historical
record, make that record more digitally available, and ensure that cultural heritage collections are responsive to and inclusive of the communities that create the items and knowledge in many collections. At
the same time, we are aware of the complexity of funding programs incultural heritage; the significant time required for design, implementation, and management of multi-year programs; and the challengesof taking ethical action within this complex system. We identified
myriad areas for attention in DHC:AUV—attention toward the scopeof activities eligible for funding; the needs for robust applicant support; equity-centered design of the processes for applying, reviewing,66

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

and awarding proposals; clarification and communication of program
values and priorities; and the significant, day-to-day work required
to sustain and administer a funding program. Changes in some areas may lead to positive impacts in the near term, but other changes
will have indeterminate outcomes. The difficulty of comparing theunique experiences of stakeholders and measuring progress toward
the program’s goals means that tangible, long-term results may remain unknown. Overall, however, our insights from this assessment
revealed enthusiasm for the future of DHC:AUV and, more broadly,
the potential for increasing equity in representation among cultural
heritage collections.

67

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

References
A4BLiP. 2019. “Anti-Racist Description Resources.” Philadelphia:
Archives for Black Lives in Philadelphia (A4BLiP), Anti-Racist
Description Working Group. https://archivesforblacklives.files.
wordpress.com/2020/11/ardr_202010.pdf.
AAM. 2022. “Museum Assessment Program (MAP).” American
Alliance of Museums. https://www.aam-us.org/programs/
accreditation-excellence-programs/museum-assessment-programmap/.
Aggarwal, Rashi, Alan K. Louie, Mary K. Morreale, Richard Balon,
Eugene V. Beresin, John Coverdale, Anthony P. S. Guerrero, andAdam M. Brenner. 2022. “On the Art and Science of Peer Review.”
Academic Psychiatry 46 (2): 151–56. https://doi.org/10.1007/s40596022-01608-1.
Anderson, Jane, and Kimberly Christen. 2013. “‘Chuck a Copyrighton It’: Dilemmas of Digital Return and the Possibilities for Traditional
Knowledge Licenses and Labels.” Museum Anthropology Review 7(1–2): 105–26. https://scholarworks.iu.edu/journals/index.php/mar/article/view/2169.
Arnold, Hillel, Dorothy J. Berry, Elizabeth M. Caringola, Angel Diaz,
Sarah Hamerman, Erin Hurley, Anna Neatrour, Sandy Rodriguez,
Megan Senseney, Ruth Tillman, Amy Wickner, Karly Wildenhaus,and Elliot Williams. 2020. Do Better – Love(,) Us: Guidelines forDeveloping and Supporting Grant-Funded Positions in Digital
Libraries, Archives, and Museums. https://dobetterlabor.com.
Baines, Donna, John Campey, Ian Cunningham, and John Shields.
2014. “Not Profiting from Precarity: The Work of Nonprofit
Service Delivery and the Creation of Precariousness.” Just Labour:
A Canadian Journal of Work and Society 22: 74–93. http://www.
justlabour.yorku.ca/volume22/pdfs/06_baines_et_al_press.pdf.
Banks, Joy, and Christa Williford. 2018. “Collaborative Grants:
Why Do We Care?” CLIR Issues 126 (Nov–Dec). https://www.clir.
org/2018/12/clir-issues-126/.
Banks, Joy. 2019. The Foundations of Discovery: A Report on theAssessment of the Impacts of the Cataloging Hidden Collections
Program, 2008–2019. Washington, DC: Council on Library andInformation Resources. https://www.clir.org/pubs/reports/pub177/.
Bertrand, Marianne, and Sendhil Mullainathan. 2003. “Are Emily andGreg More Employable than Lakisha and Jamal? A Field Experimenton Labor Market Discrimination.” Working Paper 9873. National
Bureau of Economic Research. https://doi.org/10.3386/w9873.

68Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Carroll, Stephanie Russo, Edit Herczog, Maui Hudson, Keith Russell,and Shelley Stall. 2021. “Operationalizing the CARE and FAIR
Principles for Indigenous Data Futures.” Scientific Data 8 (1): 108.
https://doi.org/10.1038/s41597-021-00892-0.
Caswell, Michelle, Christopher Harter, and Bergis Jules. 2017.
“Diversifying the Digital Historical Record: Integrating Community
Archives in National Strategies for Access to Digital Cultural
Heritage.” D-Lib Magazine 23 (5/6). https://doi.org/10.1045/
may2017-caswell.
CLIR. 2021a. “Mellon Foundation Funds CLIR’s Digitizing Hidden
Collections and Archives: Amplifying Unheard Voices.” News
release, February 17, 2021. https://www.clir.org/2021/02/mellonfoundation-funds-amplifying-unheard-voices/.
CLIR. 2021b. Program Evaluation Team Statement of Work
document, April 20, 2021.
CLIR. 2022. “CLIR Announces Digitizing Hidden Collections:
Amplifying Unheard Voices Awards.” News release, April 28, 2022.
https://www.clir.org/2022/04/clir-announces-digitizing-hiddencollections-amplifying-unheard-voices-awards/.
Davis, Miriam L. E. Steiner, Tiffani R. Conner, Kate Miller-Bains,and Leslie Shapard. 2020. “What Makes an Effective Grants Peer
Reviewer? An Exploratory Study of the Necessary Skills.” PLOS ONE
15 (5): e0232327. https://doi.org/10.1371/journal.pone.0232327.
Dean, Jackie. 2019. “Conscious Editing of Archival Description
at UNC-Chapel Hill.” Journal of the Society for North Carolina
Archivists 16: 41–55. https://www2.archivists.org/sites/all/files/
Dean,%20Jackie.pdf.
Dedoose. 2022. “Dedoose History.” Accessed October 1, 2022. https://www.dedoose.com/about/history.
Fallon, Julia. 2016. “Rightsstatements.Org Launches at DPLAfest 2016
in Washington DC.” Europeana Pro (blog), April 14, 2016; updated
July 13, 2018. https://pro.europeana.eu/post/rightsstatements-orglaunches-at-dpla-fest-in-washington-dc.
Ferraiolo, Nicole Kang. 2019a. “More Equitable Partnerships in
Grant Funding” (CLIR blog), February 21, 2019. https://www.clir.
org/2019/02/more-equitable-partnerships-in-grant-funding/.
Ferraiolo, Nicole Kang. 2019b. “Toward a More Inclusive Grant
Program” (CLIR blog), March 15, 2019. https://www.clir.
org/2019/03/toward-a-more-inclusive-grant-program/.
First Archivist Circle. 2007. “Protocols for Native American Archival
Materials.” Available at https://www2.nau.edu/libnap-p/index.html.
Fiske, Susan T. 2002. “What We Know Now About Bias andIntergroup Conflict, the Problem of the Century.” Current
Directions in Psychological Science 11 (4): 123–28. https://doi.
org/10.1111/1467-8721.00183.

69

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Frick, Rachel L., and Merrilee Proffitt. 2022. Reimagine Descriptive
Workflows: A Community-Informed Agenda for Reparative andInclusive Descriptive Practice. Dublin, Ohio: OCLC Research.
https://doi.org/10.25333/wd4b-bs51.
Goldin, Claudia, and Cecilia Rouse. 2000. “Orchestrating
Impartiality: The Impact of ‘Blind’ Auditions on Female Musicians.”
American Economic Review 90 (4): 715–41. https://doi.org/10.1257/
aer.90.4.715.
Hughes-Watkins, Lae’l. 2018. “Moving Toward a Reparative Archive:
A Roadmap for a Holistic Approach to Disrupting Homogenous
Histories in Academic Repositories and Creating Inclusive Spaces for
Marginalized Voices.” Journal of Contemporary Archival Studies 5(1). https://elischolar.library.yale.edu/jcas/vol5/iss1/6.
Jules, Bergis. 2019. “Architecting Sustainable Futures: Exploring
Funding Models in Community-Based Archives.” Shift
Design. https://shiftdesign.org/content/uploads/2019/02/
ArchitectingSustainableFutures-2019-report.pdf.
Martín, Eloísa. 2016. “How Double-Blind Peer Review Works andWhat It Takes to Be a Good Referee.” Current Sociology 64 (5): 691–
98. https://doi.org/10.1177/0011392116656711.
NEH. 2022. “Preservation Assistance Grants for Smaller Institutions.”
National Endowment for the Humanities. https://www.neh.gov/
grants/preservation/preservation-assistance-grants-smallerinstitutions; accessed October 1, 2022.
NHPRC. 2022. “NHPRC Application Review Process.” National
Historical Publications & Records Commission, US National
Archives and Records Administration. https://www.archives.gov/
nhprc/apply/review-process.html; accessed July 27, 2022.
NSF. 2022. “DGE Panelist Information.” Division of Graduate
Education, Education and Human Resources Directorate, National
Science Foundation. Washington, DC. https://www.nsf.gov/ehr/dge/
DGEPanelistInfo.jsp.
OCEAN. 2022. “Open Copyright Education – Course Recordings.”
University of Michigan Library. https://sites.google.com/umich.edu/
oceancopyright/resources/course-recordings.
Rodriguez, Sandy, Ruth Tillman, Amy Wickner, Stacie Williams, andEmily Drabinski. 2020. “Collective Responsibility: Seeking Equity
for Contingent Labor in Libraries, Archives, and Museums.” IMLS
Project Report. https://laborforum.diglib.org/wp-content/uploads/
sites/26/2019/09/Collective_Responsibility_White_Paper.pdf.
SAA. 2022. Dictionary of Archives Terminology, s.v. “reparativedescription.” https://dictionary.archivists.org/entry/reparativedescription.html; accessed July 2022.

70

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

SSDN. 2020. “Inclusive Metadata & Conscious Editing Resources
List.” Sunshine State Digital Network Metadata Working Group
paper. https://docs.google.com/document/d/1APavAd1p1f9y1vBUu
dQIuIsYnq56ypzNYJYgDA9RNbU/edit; accessed July 2022.
SSHRC. 2022. “Manual for Merit Review Committee Members.”
Social Science and Humanities Research Council. https://www.
sshrc-crsh.gc.ca/funding-financement/merit_review-evaluation_
du_merite/adjudication_manual-guide_comite_selection-eng.aspx;
accessed July 27, 2022.
TAVP. 2022. “Centering Agency, Community, and Care in Archives
Grantmaking.” Texas After Violence Project. https://medium.com/
community-archives/centering-agency-community-and-care-inarchives-grantmaking.
Trust-Based Philanthropy Project. 2021. “The 6 Grantmaking
Practices of Trust-Based Philanthropy.” https://www.
trustbasedphilanthropy.org/resources-articles/grantmakingpractices; consulted October 2021.
Wigginton, Nick, Jesse A. Johnston, Tabbye Chavous, Eric Shaw.
2021. “Managing internal nomination and peer review processes
to reduce bias.” White paper from University of Michigan Office of
Research. Archived at https://web.archive.org/web/20210215165557/
https://research.umich.edu/sites/default/files/guidance_for_
equitably_managing_nomination_and_peer_review_process_to_
reduce_bias_on_web_2-9-21_9_pm.pdf.
Wright, Alyssa. 2021. “Jack Dorsey’s #StartSmall Initiative Grants a
Transformational $3 Million To Girls’ Education.” Forbes.com (10August). https://www.forbes.com/sites/alyssawright/2021/08/10/
jack-dorseys-startsmall-initiative-grants-a-transformational-3million-to-girls-education/.

71

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Appendix A
Implicit Bias Orientation
The assessment team developed an implicit bias orientation for the
initial meeting of the DHC:AUV review panel. We developed this inaccordance with recommendations from various studies of review
activity, which have suggested that although implicit bias cannot be
removed, discussion of the concept is a useful reminder for proposal
evaluation (see Wigginton et al. 2021). This is underscored in studies
of many other evaluation processes, such as anonymous orchestra
auditions and job applications, where we know that evaluators rely on
context clues taken from beyond the contents of materials submitted.
We recommended, therefore, that implicit bias be introduced anddiscussed among reviewers for DHC:AUV. Although this orientation
was formulated for DHC:AUV, we would suggest it can be a templatefor similar discussions among other review panels engaged in proposal evaluation.
Unconscious Bias: Implicit Bias and Tacit Bias

This short orientation provides an overview of how to understandand approach bias in the review process, which is a key consideration
to keep in mind as you discuss and enact the new program values
and apply them to your evaluation of the initial applications. We will
briefly introduce some variations of implicit bias, as well as some of
the factors in a review setting that make these issues particularly challenging. We present this in the spirit of opening the discussion and asa framework that reviewers might use to consider, discuss, and evaluate ideas and assumptions so as to most fully enact and equitably
review proposals.
These ideas offer a way to surface or amplify considerations andquestions about bias. Each evaluator brings biases into their work,
but bias awareness is an active way to address this challenge. We hopethat this conversation begins to normalize our discussion of biases
and assumptions.
Implicit Bias. An “implicit” bias may be described as an unconscious hypothesis or “schema” about a group or idea: a model thathelps us to interpret and understand the behavior of other people andgroups (see Fiske 2002). For example, a schema may shape associations about the nature, characteristics, or abilities of a group. Bias has
been quite clearly documented in multiple review situations. For example, a 2003 study by Bertrand and Mullainathan analyzed callbacks
for interviews based on resumes. After analyzing responses to nearly
5,000 resumes submitted for job advertisements, distributed between
experience and with the substantive differences being in racially“marked” names, resumes associated with “white” names received
approximately 50% more response than those for people of color.

72

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Similarly, analysis of screened (without visual cues of an applicant)and unscreened orchestra auditions by Goldin and Rouse in 2000
suggested that the presence of a screen (an anonymization method)
created some initial benefit to female applicants. This suggested thatthe removal of identity markers influenced results of auditions.
While these studies demonstrate the presence of biases, the best
way to address them is less clear, particularly in a complex process
like a grant proposal review where there are many stages of uncertainty. That said, we would encourage you to be aware of biases as
you are working and discussing proposals; be particularly sensitive
to those situations when schemas about groups or assumptions may
influence or shape your evaluations. In a review setting, this may include:• categories of gender and race (frequently discussed as sites of implicit bias),• collection type,• topic,• institution type,• perceived size and resources of the applying organization,• geographic location,• and other factors unrelated to an applicant such as perceptions
about level of application editing, socioeconomic status of an applicant, or other language cues.
Tacit Assumptions. A second area of unconscious bias that may
influence the discussion may be described as “tacit assumptions,”
which are known but unspoken schemas or models that may beframeworks for actions. For example, what is your frame for understanding a 10-point scale, and how do you apply it? Perhaps you see it
as a teacher or grader who has a certain threshold for a pass/fail performance, perhaps you aim to create a distribution of scores more-orless equally, or perhaps you want to group most scores in the middle
with only a few at the extreme highs or lows. Like these models ofgrading, your experience working as a scholar, researcher, faculty
member, or collection manager may nudge your evaluation style incertain directions. For example, if you are expert in certain fields, you
may associate mentions of well-known scholars, institutions, or collections with certain levels of quality. As you work through the initial
discussions of applications, some of these will likely surface, and there
will be a group process of norming, where everyone becomes more
comfortable with the program values, shares experiences, and comes
into alignment.
Complicating Factors during the Meeting
As you begin the review, we would remind you of some factors that
accentuate implementation of implicit bias and unconscious schemas.
These include situations in which we have limited information, which
often causes us to rely on existing schemas when evaluating aspects ofproposals. For example, your discussions may be largely constrainedby what you learn from the applications, or in some cases when apanelist may have direct knowledge of a collection or applicant.
Likewise, we may be more likely to rely on schemas in high pressure73Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

situations. These are frequent in a review panel, such as when we areunder stress to explain varied evaluations, working on potentiallycompeting tasks (such as our Zoom meeting and what’s happening
at home), or are under time pressure (to reach a decision during themeeting). Finally, despite the many efforts to create a diverse and representative panel, even groups of a dozen or more members are limited in their range of diversity, so there will likely be many situations inwhich there is not a “critical mass” of some groups or perspectives.
In all these situations, our experience has been that raising andnaming these sorts of biases is a key element in addressing them. The
complexity of the situation and the nature of the process, proposals,collections, and range of topics make it difficult to present a specific remedy to eliminate biases. However, we hope that this materialpresents a framework to help open up and begin discussions that will
help your reviews to be informed by the values of inclusion, diversity,
equity, and access as you discuss creating ethical access to collectionsof underrepresented groups, a historically marginalized groups orcommunities in “Amplifying Unheard Voices.”74Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Appendix B
Review Rubric for Full Applications
The assessment team developed the following rubric for use by reviewers while evaluating the full applications. The intent was to provide a quick, visual reference that could be printed to a single sheet or
viewed on screen. The rubric was circulated to reviewers. For the purposes of future potential applicants, such a rubric may also be usefulin thinking about how to strategically plan competitive applications
to the DHC:AUV program.

CRITERIONEXCELLENT(Recommend for award)

ADEQUATE

NOTRECOMMENDED

1. Alignment with Core Values (2 maximum for each core value)
The proposed projectwill contribute to Public
Knowledge and create
access to materials that
document underrepresented and historicallymarginalized communities/populations.

The project will createand disseminate digitized
special collections and archives as a public good. (2)The project engages withpublic knowledge, but the
project may miss opportunities to reflect publicknowledge or further
develop this aspect of the
project. (1)Little or no evidence ofengagement with publicknowledge. (0)The proposed projectwill contribute to Broad
Representation, creating
access to materials that
document underrepresented and historicallymarginalized communities/populations andoffering opportunities forthese communities andpopulations to participate
in building their historical
record.

The project will thoughtfully share the untappedstories of people, communities, and populations
who are underrepresented
in digital collections,contributing to a more
complete understanding of
human history. (2)The project unevenly
Little or no evidence of
demonstrates broad repre- broad representation. (0)sentation, but the project
misses opportunities todiversify the digital historical record or must develop
this aspect of the project.
(1)

The proposed project will
foster Authentic Partnerships in its approach tocreating access to materials
that document underrepresented and historicallymarginalized communities/populations.

The project demonstrates
meaningful engagementwith the underserved
communities whose stories
the source materials tell,and builds inclusive teams
across institutional andgeographic boundaries. (2)The project demonstrates
Little or no evidence ofuneven engagement withauthentic partnerships. (0)authentic partnerships.
The project misses opportunities to develop authentic partnerships or needs
to develop this aspect ofthe project. (1)75Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

CRITERIONEXCELLENT(Recommend for award)

ADEQUATE

NOTRECOMMENDED

The proposed project will
establish or employ Sustainable Infrastructures
while creating access tomaterials that document
underrepresented andhistorically marginalized
communities/populations.

The project will promote forward-thinking
strategies ensuring thelong-term availability,
discoverability, and interconnectedness of digitized
content. (2)The project unevenly
develops sustainable infrastructures. The project
misses opportunities tocreate sustainable infrastructures or must develop
this aspect of the project.
(1)

Little or no evidence ofsustainable infrastructures.
(0)

The proposed project
exemplifies Community-Centered Access in
its approach to creating
access to materials that
document underrepresented and historicallymarginalized communities/populations.

The project will demonstrate approaches toaccess, description, andoutreach that make digitized content as widely
available and useful as
possible within legal andethical constraints, centering digital inclusion
and respect for materials’ local contexts. (2)The project unevenly
Little or no evidence ofdevelops communicommunity-centeredty-centered access. The access. (0)project misses opportunities to create access ormust develop this aspectof the project. (1)2. Understanding of Rights and Ethics (5 maximum)

The proposed projectaddresses the rights andethical concerns that
may affect access to thecontent nominated for
digitization, centering the interests of thepeople who created or
who are represented in
the source materials in
its strategy for creating
access and sharing information about re-use.

Detailed awarenessof relevant rights andethical considerations,clearly reflected in theirstrategy for creating
community-centeredaccess to the nominatedmaterials. Demonstrates
thoughtful consideration of the needs andinterests of the creatorsand communities documented in the materials
and, when possible,
solicits input from thosecreators and communities to ensure the projectdeliverables are empowering, meaningful,and useful to the people
closely connected to thecontent. (5)Basic awareness of relevant rights and ethicalconsiderations but lacks
clear explanations for
how these issues will be
addressed or translated
into strategies for providing access to projectdeliverables and forcommunicating information about re-use.
Applicants may needmore time to develop
their approaches to creating access, to engaging
with content creators
or represented communities, or to managing
risks related to creating
access. (3-4)

Limited or inaccurate
assessments of the rights
and ethical considerations that will restrict
the ability to create
access to the content
nominated for digitization. (1-2)

76Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

CRITERIONEXCELLENT(Recommend for award)

ADEQUATE

NOTRECOMMENDED

3. Readiness to Undertake Proposed Project (5 maximum)

The project’s participants seem ready toassemble the resourcesand expertise they willneed to make the projecta success. (Rememberthat no prior experience
with digitization is required, and so a lack ofprior experience should
not detrimentally affectthis score.)

Solid understandingof the time, resources(including people),and expertise required.
Clear plan for securing
any project resources
or expertise. Realistic
timeline for completingthe project given participants’ existing capacityand experience. (5)Adequate understanding of the resources andexpertise required for
digitizing and maintaining access to rare andunique content, withsome noticeable gaps.
Missed opportunities
to fully describe one ormore essential elementsof their project design(staffing, technical standards, metadata, digitalpreservation, etc.). May
need more time to accurately estimate the time,
resources, or expertise
required. (3-4)

Vague or unattainable
expectations of theresources and expertise
required for proposed
digitization. (1-2)

4. Demonstrated Need for External Support (5 maximum)

The people and organization(s) participatingin the project havedemonstrated a needfor external support inorder to make the proposed project a success.

Clear explanation why
external investment
from this program isessential to undertakethe project. All costs inthe budget are justified
and explained. (5)Reasonable case forexternal support
through this program,but leaves some doubt
about the level of support requested. May
lack justification for
major costs cited in thebudget, or costs may besubstantially higher orlower than the reviewer
thinks sufficient given
the information in theapplication. May raisequestions about thedegree of commitmentof one or more participating organizations to
the project’s outcomes.
(3-4)

Does not explain why
internal resources are
insufficient to support
the project, does notshow sufficient evidence
of engagement withthe program’s or CLIR’s
goals and core values.
Fails to itemize and justify the costs cited in the
project budget. (1-2)

77

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Appendix C
Principles for Useful Feedback
This set of reviewer principles offers a “reviewer guide” based upon
feedback from both DHC:AUV reviewers and applicants. Our intent
is to distill some of the overarching themes that we noted regarding
proposal feedback, so we have structured this as a list of topics, briefly
elaborated. Although arising from DHC:AUV, we hope that this advice is complementary to other programs that review grant proposals
in the cultural heritage sector and that it may also be used in conjunction with similar advice from other domains (such as Aggarwal et al.
2022, Davis et al. 2020, Martín 2016).
Plan to Take More TimeMultiple reviewers mentioned that the review task always took moretime than they initially thought it would. This is related to many factors, including the time commitments in a reviewer’s own life, thecomplexity of proposals under review as well as the need to understand a complex program and evaluation approach. In addition, as
one reviewer related, even when using a rubric, a reviewer’s evaluation of proposals may change as they read through more proposals,
so there may be a need to review or re-visit evaluations and feedback
on previous proposals.
Make Substantive Comments

In some cases, reviewer comments can be brief such as “great project”
or “well planned” or “this proposal needs more work before it canbe funded.” These are good starting points, but they do not provide
actionable or specific information to an applicant. For a program like
DHC:AUV, which aims not only to support important digitization
work but also hopes to build capacity and knowledge among the applicant pool, such brief comments do not provide useful feedback.
Although longer comments are not necessarily more substantive, very
brief written comments usually do not contain actionable feedback.
The following recommendations offer additional ways to provide
more substantive comments.
Be Specific; Be Direct

It is most helpful to applicants to know which areas of a proposal
need work or require further planning:• Specific feedback may identify a section of a proposal and thenprovide advice for specific actions or plans to address that element.
• Illustrating your feedback with specific examples helps an applicant understand more clearly what sorts of things the review panel

78

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

takes into account. What makes the project so appealing? Alternatively, what is missing from the proposal?
• If your comments point to elements that an applicant has alreadyaddressed, your feedback can help to point out elements of a proposal that need to be revised or made clearer.
Some applicants reported that they appreciated directness: if a
reviewer does not support funding for a proposal or finds it ill-suitedfor the DHC:AUV program, applicants appreciate knowing that.
Build Capacity and Promote Growth

If one outcome of grant programs in cultural heritage is to support andadvance the field, it is important for reviewers to provide feedback thathelps to build capacity—a growth mindset for the cultural heritage
field. Whether or not the advice is something an applicant can implement, the planning of a project and codification of it in a proposal suggests an interest in working toward a project goal.
The proposal review process is not only for the panel to make decisions or recommendations about funding, but it is also an opportunity for applicants to receive feedback and advice, which may be used
to shape or improve future applications and project plans. Feedback
on a proposal can provide advice, encourage emphasis on particular topics or areas of activity, and provide motivation, regardless ofwhether the proposal is funded.
Adopt Empathy

When writing comments for a proposal, imagine yourself as an applicant
or potential applicant receiving the feedback:• What advice would you like to hear if your proposal is not selected for funding?
• Have you offered a clear explanation of why you reached yourrecommendation?
• Do your comments align with your evaluation of the proposal?
• What advice do you think the applicant needs to hear?

79Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Appendix D
Comparison of Initial and Final Application
Elements with Program Values
This appendix presents a table that compares the elements of thefrom the initial round application with the final, full application.
The information here is based on the description of the application
presented in various versions of the Applicant Handbook available
to applicants via the DHC:AUV website: the initial application is as
described in the initial handbook (published as version 2), while thefinal application is as described in the final handbook (published asversion 3). This table condenses a large amount of information in thehandbooks, which served as a useful way for the assessment team to
analyze the overall shape of the application, and it is presented here
as a tool for potential applicants to see how application elements fromthe initial round relate to the final round. In addition, the “Values
correlation” is presented as a possible indicator for applicants to understand where the DHC:AUV program values can most usefully be
integrated.
The first column on the left (“Initial Application Elements”)
describes the major sections and questions of the initial application, while the second column (“Notes”) describes the form of the
response (such as “Form,” which designates information typed intoa web form by the applicant, or “Upload,” which designates a file
uploaded by the applicant) and, where applicable any notable limits
such as word count or page length. The third column (“Final application”) describes major sections and questions of the full applicationand is also accompanied by a Notes column. In this application, anyelements that are listed on the same line as in the initial application
are either carried over from the initial application or based upon information in the initial application; any elements marked with NEW
are elements that are found only in the full application. Finally, in thefifth column (“Values Correlation”) we have offered a cross-reference
to the DHC:AUV program values, designated by two-letter abbreviations as explained below. Shading of the rows is used to visually
set apart application elements according to the application sections,such as application overview information, materials to be digitized,
description of the voices represented by the nominated collections,rights and ethical concerns, and project details (project personnel,
timelines, budgets, etc).
Program Values abbreviations (noted in “Values correlation”
column):
CA: Community-Centered Access
SI: Sustainable Infrastructures
AP: Authentic Partnerships
PK: Public Knowledge
BR: Broad Representation

80

81

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Summary Overview Applications Elements

Initial Application• Numerous short form responses to fill in• 3000 words (6 free-text responses)• 2-3 uploads
Final Application• Multiple short form responses pre-populated, may be updated• 4750 words (7 free-text responses)• 10-15 uploads

INITIALAPPLICATIONELEMENTS

NOTES

FINAL
APPLICATIONNOTES

Initial Eligibility

FormFinal EligibilityForm(review/update)

Initial Applicant
Information

Form

Final Application
InformationForm(review/update)

Initial Project
Overview

Forms (free-text
responses), includes“Fields of Study”
keywords

Final Project
Overview

Forms (review/
update responses),
includes “Fields ofStudy” keywords

Final Application
Adjustments

NEW: information
on any changes made(250 words)

VALUES
CORRELATION(assigned byassessment team)Project abstract may
address all values

Project Details:
Materials (“WhatMaterials will you
digitize?”)

Form (free-text
Project Details:
response, 500 words) Materials

Form (free-text
PK
response, 700 words)
NEW: geographic
scope, date range

Representative
samplesUpload (optional)

Representative
samplesUpload (up to 3
images)

NEW: Extent oforiginal material

Form (up to 10
types, specifying
extent of material
and physical format)

NEW: List ofCollections to bedigitized

Upload (spreadsheet
template provided)

Project Details:
Form (free-text
Project Details: Tell
“Whose voices will
response, 500 words) the Story
this project amplify?”
NEW: External
letters of support
Project Details:“What rights or
ethical concerns
apply to thisproject?”

Form (free-text
Project Details:
response, 500 words) Rights, Ethics, and
Re-UseForm (free-text
PK, BR, AP
response, 700 words)
Upload (at least 1, up
to 4)
Form (free-text
response, 1000
words)CA, AP

82

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

INITIALAPPLICATIONELEMENTS

NOTES

FINAL
APPLICATION
NEW: RER
Appendix

Project Details:“What are theoutcomes andaudiences for thisproject?”

Form (free-text
Project Details:
response, 500 words) Outcomes

NOTES
Upload (optional)

Form (free-text
CA, AP
response, 700 words)

NEW: Extent ofdigitization

Forms (up to 10
types, specifying
extent of files andformats)

NEW: Platform for
AccessFillable form (URL)

Project Details:
NEW: Work Plan

Upload (8 pp. max.)

Project Timeline

Upload (2 pp. max.)

Project Details:“What experienceand capacity will
you bring to andstrengthen throughthis project?”

Form (free-text
Project Details:
response, 500 words) Capacity

VALUES
CORRELATION(assigned byassessment team)CA, SI, AP

Form (free-text
CA, SI
response, 700 words)

NEW: Letters ofCommitment

Upload (2 pp. max.
each)

NEW: Resumes/Job
Descriptions

Upload (no page
limit)

NEW: Board/Trustee Upload (separate
list
instructions)
Project Details:“Why do you needsupport from
CLIR’s program
to undertake thisproject?”

Form (free-text
Project Details: Need Form (free-text
BR
response, 500 words) for Support
response, 700 words)

Budget Summary

Upload (2 pp. max.)

NEW: Budget detail

NEW: Budget
narrative
NEW: Service
estimates/quotes(if required)

Upload (template
provided)Upload (template
provided)Upload (pdf)Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

Acknowledgments
We thank the CLIR staff for their diligent and attentive work on the
Digitizing Hidden Collections (DHC) program over the years and fortheir specific support of our work throughout this project. In particular, a huge thanks to former program officer Joy Banks and seniordirector of research assessment Christa Williford, who met with us
regularly, provided all the information that we needed, and served as
wonderful sources of program information. Without their insights
and guidance, we would have had a much poorer understanding of
the program. Additional thanks to program officers Becca Quon,
Sharon Burney, and Alyson Pope, who generously shared their timeand insights while also working to implement many of our interim
recommendations. We also tip our hats to CLIR’s patient editor andproofreaders, who helped us to smooth out our writing and make thisreport look great. We also thank CLIR leadership, including Charles
Henry and members of the DHC Review Panel who selected andsupported our assessment proposal. We were thrilled by the graphic
design that Alyssa Downs provided for us, particularly the much-improved process diagram included later in the report. We also thankthe many applicants, interested parties, and program stakeholders
who shared their time, experiences, and insights, which provide thebasis of this report. We hope that these efforts accurately and respectfully reflect their observations and insights. Any errors of fact or interpretation remain squarely with the authors.

83

Evaluating Equity and Inclusion in Cultural Heritage Grantmaking

About the Authors
Jesse A. Johnston is a librarian and archivist, educator, admin-

istrator, and scholar with extensive experience in the public sector,
research, and teaching. Since mid-2022, he has served as a clinical assistant professor at the University of Michigan School of Information.
Johnston undertook the work for this project as a principal and project director for the Archives Research and Consulting (ARC) Group.
In his work as a librarian and an archivist, Johnston has served as asenior librarian for digital content at the Library of Congress and as
an archivist in the Ralph Rinzler Folklife Archives and Collections
at the Smithsonian Center for Folklife and Cultural Heritage. As an
administrator, he served as a senior program officer for preservation
and access at the National Endowment for the Humanities, where heestablished and managed “Common Heritage,” the agency’s first grant
program that directly supported community archiving.

Ricardo L. Punzalan is currently an associate professor at the

University of Michigan School of Information. Previously, Punzalan
was a teaching faculty member at the University of the Philippines
School of Library and Information Studies (2000–2006) and the
University of Maryland College of Information Studies (2013–2019).
He has worked as an archivist for a range of communities, including
the establishment of the archives of a former leprosarium, which
also included curation of a museum exhibit for the centennial of its
founding as a segregation facility; Punzalan also served as an archivist
at the Vargas Museum. His work and research have focused on the
expansion of ethical community access to collections from various
North American and Philippine indigenous communities. From 2018
to 2021, Punzalan served on the Council of the Society of American
Archivists (SAA) and in 2022, he was named co-chair of the SAA Archival Repatriation Committee.

84