ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

1 

ANDREA: Okay, hi everyone. My name is Andrea Robles 

and I work at the Office of Research and Evaluation 

at CNCS. I want to thank all of you for joining us 

today, especially given that ever-evolving 

circumstances of our home and work lives due to 

COVID-19. We hope that you, your families, friends, 

and communities remain healthy and are staying safe. 

Our office’s objectives are to support CNCS’s mission 

by building knowledge on civic engagement, 

volunteering, and national service, and to use this 

knowledge to both contribute to the broader field of 

civic engagement, as well as to improve lives and 

strengthen communities. 

 We conduct in-house research and evaluation, and fund 

research through competitive grants to scholars and 

dissertators at institutions of higher education. 

Our Research and Evidence Webinar series is one way 

we share our grantees ongoing research, studies from 

other researcher working on similar topics, and 

findings about our AmeriCorps and Senior Corps 

program evaluations. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

2 

 In today's context we believe that sustaining and 

increasing civic engagement is more important as 

ever. As a society, and at least in the near future, 

we will be discussing the differences between social 

distancing versus physical distancing, and what that 

means for our families communities and work lives. As 

researchers, evaluators, and program staff we will 

need to think creatively about how we could continue 

to engage stakeholders in meaningful ways. 

 So before we would begin we like to cover a few 

housekeeping items for Adobe Connect so I will turn 

this over to ICF next. 

JENELLE: Thank You Andrea and thank you all for your 

patience as we get started with Adobe Connect. This 

webinar will be recorded and posted online following 

the presentation. There is no dial-in phone line, all 

audio is broadcasted over the internet using your 

computer speakers. All participants will be in 

listen-only mode until the question and answer 

session following the presentation, at which time you 

can ask a question using your computer's microphone 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

3 

by selecting the Raise Hand feature from the menu 

above. You can also ask questions at any time during 

the presentation by using the chat box below. As 

mentioned earlier, this webinar is being recorded and 

if you have any questions or experienced technical 

difficulties please let us know using the chat box 

below. I believe that takes care of our housekeeping 

items. Andrea? 

ANDREA: Okay, thanks Jenelle. So our format today will be 

as follows: Dr. Mary Hyde, Director of Research and 

Evaluation at CNCS will provide some introductory 

remarks around today's topic. Her remarks will be 

followed by our first presenter Dr. Emily Zimmerman. 

Dr. Zimmerman is an Associate Professor at the 

Virginia Commonwealth University and is editor of a 

recently published book titled, Researching Health 

Together: Engaging Patients and Stakeholders from 

Topic Identification to Policy Change. And we are 

very proud to claim that she is also one of our 

research grantees. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

4 

 For the second presentation we have Michelle 

Brodesky, who is a Strategic Learning and Evaluation 

Manager at Methodist Healthcare Ministries of South 

Texas Inc. Methodist Healthcare Ministries was a 

grantee of CNCS Social Innovation Fund between 2014 

and 2019. Also, presenting with Michelle is Lisa 

Wolff, who is Vice President of Health Resources and 

Action, and worked with Michelle in Methodist 

Healthcare Ministries as an evaluator for the Social 

Innovation Fund project, which they will be 

describing during their presentation. 

 These presentations will be followed by brief 

concluding remarks by Dr. Kayla Cranston, who is in 

the Department of Environmental Studies and Director 

of Conservation Psychology, Strategy and Integration 

at Antioch University. She is also our former CNCS 

AmeriCorps VISTA sponsor. 

 So at the end of the presentations, which we estimate 

will be about 35 minutes or so, we will open up the 

discussion to Q&A. And as Jenelle said, during the 

webinar please feel free to type in questions in the 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

5 

Q&A box or provide comments in the chat box. And the 

speakers will respond to them during the discussion. 

So I will pass this on to Dr. Mary Hyde. 

DR. HYDE: Thank You Andrea and thank you everyone for 

joining us today. I hope that you will find our 

presenters remarks interesting as we do. Just a few 

framing ideas and to prime your minds a bit about 

some of the themes that you're going to be hearing 

from all of our presenters today, as the title of the 

webinar suggests, we are going to be focusing on 

participatory research. But, I wanted to really 

emphasize that these principles of community and 

partnership engagement are really principles that we 

value as an office. 

 And so the research that we either sponsor, and the 

evaluations that we sponsor, or that we just partner 

with really strive to put these principles into 

practice. And the reasons that we think that this 

type of stakeholder engagement make the research and 

the evaluation better are as follows, and you will 

hear these themes reiterated across the different 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

6 

speakers today, but just to sort of help you think 

about and ponder what will be, I think, illustrated 

nicely in both of these presentations: One, we feel 

that stakeholder engagement, whether it's community 

residents or other types of partners, helps the 

research be more valuable at the end. 

 We always want to conduct research and evaluation 

whose findings will help improve lives, that's the 

ultimate goal. So what increases the utility of these 

types of findings? It's things like being responsive 

to the purpose and needs of the folks who are 

participating in the research, and the communities in 

which these research and evaluations take place. It's 

important because it helps make the findings more 

relevant, it helps the research and revaluation be 

more feasible. Ultimately, we think that it helps 

develop stronger solutions that are much more 

representative of the diverse needs and interests of 

any community. 

 We also think it increases the likelihood of 

sustaining the work beyond any particular project, 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

7 

whether it's a research study, or an evaluation 

study, or any other type of project. We think it also 

is increasing the likelihood that this type of use of 

research will be continued beyond the life of a 

particular study. 

 Finally, I think what we value particularly about 

engaging communities and partners in our research and 

evaluation efforts is that ultimately we feel that 

this increases individual organizational and 

community capacities, whether it's in particular 

skills, whether it's in using your voice in the form 

of civic participation, whether it's gaining access 

to information that every citizen needs to make 

informed decisions, and to be aware of issues that 

are occurring in their community. 

 So, ultimately for us, our role as the Research 

Office but also the research and evaluation that we 

conduct in partnership with the person that we'll be 

presenting today, to us it’s an important form of 

civic engagement. So we feel that is consistent with 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

8 

the mission and it creates opportunities for folks to 

be civically engaged in their communities. 

 So with that, I will pass it on to Dr. Emily 

Zimmerman and to have her present on her remarks, 

thank you. 

DR. ZIMMERMAN: Thank you Mary. Good afternoon and thank 

you for joining the webinar today. My name is Emily 

Zimmerman and I'm an Associate Professor at Virginia 

Commonwealth University and Director of Community 

Engaged Research at the VCU Center on Society and 

Health, where my work focuses on social determinants 

of health and engaging stakeholders in research. At 

the bottom right of the slide you see our Twitter 

handle, in case you want to share any comments during 

this talk. 

 Many of the points I'm going to cover in this talk 

come from the book shown on the screen, Researching 

Health Together: Engaging Patients and Stakeholders 

from Topic Identification to Policy Change. It is an 

edited volume just published by Sage Publications 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

9 

with 20 chapters covering different approaches to 

engaging stakeholders in health research. I know that 

folks listening today have diverse backgrounds and 

experiences, so I want to start with the basic 

question of “why stakeholder engagement?” 

 First, let me say that what I am covering can apply 

to research generally, whether it deals with health 

or not, or even engaging stakeholders in program 

implementation and governance. Stakeholder engagement 

in health research is popular when it comes to 

addressing health disparities because community 

members, patients, and other stakeholders offer a 

wealth of expertise in the factors that cause 

disparities. 

 Also, because engaging stakeholders means answering 

questions with, not just about, those affected. 

Working with stakeholders can increase the 

accountability and transparency of research. It can 

also improve research by asking relevant questions, 

improving quality and sustainability, and 

disseminating results more effectively. It may help 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

10

speed the translation of research findings into 

practice, and for all these reasons it is 

increasingly encouraged by funders. 

 A good place to start this work is by committing to 

principles of engagement. I have listed three very 

good references for learning about those principles. 

The first one is a book available for free download, 

the second is an excellent volume on community-based 

participatory research, the third is on the website 

for the Patient-Centered Outcomes Research Institute. 

 Looking across the project's highlighted in the book, 

I now bring us to some of the challenges that were 

described. Those challenges include relationship 

building, especially when there are low levels of 

trusts, figuring out issues of representation, and 

overcoming tokenism and power differentials so that 

stakeholders are equal partners in research. 

 There are also implementation challenges such as 

cultural and geographic differences. It can be 

difficult to deal with, the time and resource demands 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

11

of engagement, to build those strong relationships I 

just mentioned, while at the same time trying to meet 

grant and program deadlines. And there are also 

infrastructure challenges, like having mechanisms in 

place that allow for the sharing of grant resources, 

generating data to show the impact of engagement on 

research processes and outcomes, and building the 

tools for sustainability. 

 Looking across projects, I sorted the project goals 

into three categories: research development deals 

with uncovering community concerns and priorities, 

grading research questions, and identifying outcomes 

and metrics of interest to stakeholders. Many of the 

projects focused on implementation such as studying 

health problems and social determinants, testing 

healthcare treatments, or implementing programs. The 

sharing column includes studies that translate health 

information, promote policy change, and evaluate 

projects. 

 There was wide diversity and project partners across 

the studies. Individuals engaged in research included 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

12

academics, community members, patients, community 

leaders, service providers, and policymakers. 

Organizations included healthcare practices, 

community-based organizations and advocacy groups, 

associations and research centers, and systems 

included health systems, insurance payers, and public 

agencies like public health departments and schools. 

 There can be many approaches to including 

stakeholders in research. I have divided some 

approaches from the book into two groups. Team-based 

approaches include community-based participatory 

research or CBPR teams, team science type approaches 

that includes stakeholder COPIs and co-investigators. 

Many projects had advisory committees with decisionmaking power such as leadership or steering 

committees. Others had advisory groups that provided 

ongoing feedback and support. Many of those 

committees had workgroups to get specific tasks done, 

and some included national or expert advisory 

committees as well. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

13

 Network approaches include research networks such as 

patient powered research networks, practice-based 

research networks or PBRNs, and coalitions such as 

collective impact initiatives. There are many ways to 

engage stakeholders. We saw that stakeholders were 

invited to share their ideas and meetings, or to 

interact at community events. They were involved in 

providing, gathering, and interpreting data in a 

variety of ways, and participating in focus groups 

and mapping, or digital storytelling. 

 And they helped develop strategies and priorities, 

for example, deliberative methods were used to lead 

participants through a consensus process, model 

building helped participants explore and map systems 

or cause and effect pathways, and community review 

boards provided input into how to conduct research 

that is appropriate and relevant. The best projects 

build in opportunities for capacity building to 

provide people with different backgrounds the 

opportunities to learn and practice new skills, 

whether it's research advocacy or even cultural 

practices. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

14

 Capacity building took many different forms from 

formal training sessions, to learning collaboratives, 

to giving people opportunities to take on new roles. 

Facilitators mentioned across projects could be 

grouped into people, processes, and infrastructure. 

Bringing together stakeholders with diverse 

backgrounds and having them act as boundary spanners 

to connect groups are very important. Putting in 

place processes to manage power differences, 

facilitate interactions, and govern is also 

important. 

 Careful facilitation can help manage power 

differences and keep meetings focused on achieving 

their objectives. Infrastructure is needed to support 

successful outcomes such as engaging long term 

coalitions that will continue the work after a 

project ends, employing a community fiscal sponsor to 

handle grant funds, holding community reviews of 

research projects. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

15

 Not surprisingly, many lessons learned relate back to 

the facilitators just mentioned. Developing 

relationships, engaging trusted leaders, obtaining 

stakeholder input as early as possible, and spending 

time to learn about the community are vitally 

important. Projects found that processes are also 

important such as addressing history in trauma, 

setting realistic expectations, supporting colearning, and having regular meetings or check-ins. 

Our sustainability is helpful to support leadership 

development, and to shift the project leadership to 

stakeholders. 

 Now I'm going to pivot to one of the approaches 

highlighted in the book. In this chapter I describe a 

method I have been working on called the SEED Method. 

It is a multi-stakeholder engagement method that 

combines participatory conceptual modeling and 

question development. It brings together diverse 

stakeholders to explore causal factors related to a 

health outcome and to develop research questions or 

action planning strategies. Having a process in place 

with clear steps, decision-making guidelines, and 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

16

engagement tools is helpful for bringing together a 

wide range of stakeholders to address difficult 

issues. 

 This table shows the three ways that the SEED Method 

engages stakeholders. The column on the right has 

examples from a project we did on diabetes and 

hypertension in Richmond. A participatory research 

team shown in the first row of the table included 

community and academic members. We also worked with 

additional stakeholders in what we call topic groups, 

shown in row two. These groups of stakeholders create 

conceptual models and research questions. 

 In the Richmond project we had three topic groups. 

Each group brought together a certain segment of 

stakeholders such as seniors, adults with low access 

to health care, and service providers. They created 

91 research questions on diabetes and hypertension, 

and prioritized 19 of them. Finally, because we can't 

involve everyone on the research team or in topic 

groups, we bring in consulting stakeholders. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

17

 The Richmond Project held five focus groups and 

eleven in-depth interviews to gain a wider 

perspective on the experiences and priorities of 

stakeholders. This graphic shows steps in the SEED 

Method starting with identifying and engaging 

stakeholders: the sharing and reviewing data, to 

creating the conceptual models, generating priorities 

in either disseminating findings or implementing 

action plans. The whole process has stakeholders 

working together for six months or more. 

 This slide shows some of the tools that can be 

downloaded and used in the SEED Method toolkit such 

as matrices for identifying stakeholders of 

facilitation guides to lead meetings focused on 

participatory conceptual modeling, developing 

questions or strategies, and prioritizing them. 

 The photo on the Left shows one of our topic groups 

conceptual models being created. We used sticky notes 

to ensure that we can work interactively changing the 

model as participant develop and contribute ideas. We 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

18

are currently using the SEED Method to create and 

implement community action plans to address the 

opioid crisis in a rural Virginia community. With my 

Co-PI Carlin Rafi [phonetic] at Virginia Tech and the 

project was funded by CNCS as a community 

conversations research [inaudible]. 

 In the opioid project our participants have included 

the research team with two faculty, one graduate 

assistant, and six community members. We had three 

topic groups with community members, service and 

health care providers. The topic group members had 

first-hand experience with opioids as users, family 

members, and community service and healthcare 

providers. We also held three focus groups - I mean, 

I'm sorry, four focus groups. 

 In terms of results, our topic groups proposed 68 

strategies for addressing the opioid crisis in their 

community and prioritized 15 of those. We then held 

two stakeholder meetings open to the wider community 

during which participants elected five high-priority 

strategies. Community work groups were then created 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

19

to implement four of those strategies. So in that 

community they're coming together to establish a drug 

court, establish a detox center, raise awareness 

about the opioid crisis, and what the community is 

doing to address it, and provide prevention education 

for students and families. The work of implementing 

the action plans with those working groups is 

ongoing. 

 Thank you for listening. I look forward to answering 

your questions at the end. I will now pass the 

presentation to Michelle Brodesky and Lisa Wolff. 

MICHELLE: Great, thank You Emily. I am Michelle 

Brodesky. And Lisa Wolff and I are excited to do a 

brief but deep dive into the partnership centered 

evaluation approach that we used in a project called 

Si Texas. A chapter on this approach is also included 

in the book that Emily discussed. 

 Si Texas was a project located in South Texas from 

2014 to 2019 that focus on improving physical and 

mental health by supporting integrated behavioral 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

20

health programs. These are programs that treat mental 

health and physical health in coordination through a 

team-based approach. Methodist Healthcare Ministries 

led this project and collaborated with health 

resources in action to evaluate it. The project was 

funded by the Corporation for National and Community 

Service through the Social Innovation Fund. And we're 

so grateful to have had the opportunity to partner 

with CNCS for this work and to share it with you 

today. 

 The Si Texas project was focused in the 12 

southernmost counties of South Texas that you see in 

pink here. This is a region that is rich in 

resources, culture, and people. Nearly 70% of the 

population in this region is Latinx, but it also 

struggles with high rates of diabetes, obesity, and 

socioeconomic challenges. Some estimates place 

diabetes prevalence as high as 26% and the behavioral 

health system strains to meet the needs of the one in 

four adults with mental health needs. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

21

 Colonias represent a unique challenge these are 

unincorporated border communities that lack basic 

infrastructure such as drinking water, sewer systems, 

safe and sanitary housing, teeth roads [phonetic], 

etc. Many patients in this project were residents of 

such communities. And our goal working in the region 

was to take on the challenge of co-occurring physical 

and mental health conditions and to catalyze the use 

of effective clinical practice in these communities. 

 So we selected eight grantees who are doing this kind 

of integrated behavioral health work and the blue 

stars indicate the main location for each of those 

organizations. They were a varied group. They 

included community mental health centers, family 

medicine residency programs, community-wide 

collaboration, small safety net clinics. They also 

had varying degrees of experience and expertise in 

research and evaluation. Also, they were implementing 

different approaches to integrated behavioral 

healthcare. So they weren't replicating identical 

models. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

22

 Our participatory evaluation efforts focused on 

conducting rigorous randomized control trial and 

quasi-experimental design studies to determine the 

effectiveness of their individual models as well as 

to conduct a portfolio-wide evaluation to better 

understand their combined impact. So we had eight 

separate evaluation studies of each of the 

organization's work, as well as one multi-site 

portfolio level study. 

 It was our goal for this process to engage deeply 

with the grantees both so they would develop capacity 

to conduct and use research and evaluation, and so 

they could provide direction as to the ways the 

evaluation would be most useful to them. At the same 

time we also had to balance that role with 

constraints of the portfolio-wide evaluation as well 

as some of the federal funding requirements. What we 

call the partnership centered evaluation approach 

emerged as we work together balancing these 

considerations. The key players in the partnership 

were the funder, Methodist Healthcare Ministries; the 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

23

evaluator, Health Resources in Action; and the 

grantees, all eight organizations. 

 The image shows the roles of the three types of 

partner shifting over time. So you see the grantee 

role growing as we move across the lifecycle of the 

evaluation from planning to dissemination. As the 

evaluation progressed, we, the funder and evaluators, 

intended to step back at as much as possible, letting 

the grantees take the reins while providing analytic 

expertise and guidance related to funding 

requirements as needed. 

 And this model is ultimately what emerged, but we 

faced many decisions on the way there. Considerations 

and questions arose, such as, how much do we 

standardize evaluations across the site given that 

we're also doing a multi-site study? Also, how do we 

support grantees capacity to do this evaluation and 

structure engagements for learning? Lisa and I are 

going to dig in on these two considerations starting 

with the issue of customization versus 

standardization. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

24

 Of course the portfolio multi-site evaluation would 

benefit from everything being standardized, but our 

commitment to grantee engagement meant that each time 

a need for customization arose we had to balance 

their portfolio studies needs with the values that we 

were aspiring to that grantee goals should take 

precedence. As much as we could we opted to let 

grantees do things differently from each other. The 

elements that were standardized and consistent across 

the eight grantees are listed on the left, at least 

some of the main ones, including study designs, 

analytic approach, and one of the most significant 

common elements was that the grantees agreed to 

measure the same patient health outcomes. 

 On the right you see a few elements that were 

customized, and I'll touch on two of them. 

Participant eligibility criteria: Grantees had 

different target populations and determine their 

participant study eligibility differently. This meant 

that some studies enrolled only diabetics, while 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

25

others enrolled people with depression, or diabetes, 

or some other condition, and so that meant across the 

whole pool of participants there were participants 

with different conditions that were participating. 

 The data collection protocols also varied in some 

instances. The five shared outcomes were patient 

health measures that in most cases were taken on all 

study participants at three standard time points. 

However, in some cases collecting a specific measure 

on all participants was countered to a clinic’s 

policy, which happened sometimes with a measure 

related to diabetes. In those cases we defer to the 

clinic's policy which led to some variation across 

the groups. 

 The benefits to providing this level of customization 

were that it allowed data collection to be more 

feasible for the grantees and appropriate to their 

programs and settings. It also put higher priority on 

the value and relevance of their individual studies 

over our interest in the multi-site portfolio level 

study. The main trade-offs however was that the 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

26

customization created challenges for the precision of 

that portfolio study. Pulling together data from 

different population groups and slightly different 

interventions, we were able to answer the question of 

the effect of integrated behavioral healthcare in 

this region at a high level, but it limited our 

ability to draw strong conclusions about specific 

components of that care and their impact. 

 A second major consideration was how to support 

grantees capacity to do evaluation at this level. And 

I'm now I'm going to hand over the presentation to 

Lisa Wolff from Health Resources in Action to share 

how we structured capacity building in this 

evaluation approach. 

LISA: Great, thank you so much Michelle. Hello everyone. 

During the four years of the project we incorporated 

robust capacity building activities around evaluation 

so that grantees could have the skills, knowledge, 

and infrastructure to be equal partners in the 

evaluation process, and that they could incorporate 

data monitoring and evaluation in the future work to 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

27

improve sustainability and leverage results for 

future funding. Capacity-building efforts included an 

evaluation learning collaborative where grantees from 

across the southern Texas region could meet in person 

quarterly during the four year period and participate 

in a day-long session of learning, peer sharing, and 

mentoring. Topic ranged in their focus area including 

interpreting impact evaluation results, communicating 

about evaluation results to internal and external 

audiences, data collection processes, and data 

management systems, quality improvement, and other 

topics. 

 Most importantly sessions aimed to be interactive and 

incorporate various techniques for engagement. While 

there is some straight presentation, most of the 

sessions involve peer coaching, role-playing, 

interactive games, and other hands-on activities. 

Here we have a photo of Sister Maria Luisa from Mercy 

Ministries with the poster she designed during one of 

the sessions. During this event grantees designed onthe-spot a poster of preliminary evaluation results 

and then practice a three-minute presentation to 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

28

others during the gallery walk activity. Other 

capacity-building components included intensive biweekly individualized technical assistance throughout 

the project, and four session mini courses on 

relevant evaluation topics. All of these sessions 

were developed with strong input from grantees. 

 This has been an exciting, engaging, and complicated 

project. And our hope is that not only have these 

interventions been successful in improving patient's 

mental and physical health, but our approach to the 

evaluation has built skills, knowledge, and 

relationships among all who have been involved. The 

main takeaways and lessons learned from this work are 

navigating dynamics is hard. Every stage in every 

situation of the evaluation was different with no 

clear-cut answers. As Michelle described in the 

partnership-centered model, each of our roles change 

throughout the process, but this was not necessarily 

pre-described at the beginning. This was something 

that was iterative as we aim to foster collaboration 

and increase ownership across the grantees over the 

multi-year project. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

29

 Being flexible was critical and building trust with 

challenging, yet so important. For example, due to 

required timelines we had to expedite the development 

of the evaluation plan, that is deciding on the study 

design measures, composition of the comparison 

groups, etc., very quickly. There were a lot of 

decisions to be made. In order to dive into the work 

immediately we had to begin building trusting 

relationships from the beginning. Even though there 

was physical distance between the funder, evaluator, 

and grantees we tried as much as possible to have 

strong engagement early on. We had numerous phone 

calls and video chats, while they were successful 

there is nothing like face-to-face contact, which I 

think we all recognize even more now. 

 To that end, in the first few months of the project 

we also scheduled numerous trips so our evaluation 

team and grantees could get to know each other and 

develop strong relationships at the beginning. 

Looking back I wish we'd spent even more time doing 

this, perhaps even through more trust building 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

30

exercises formally. We also use their full cohort 

learning collaborative sessions to include networking 

and engagement between grantees themselves. It was 

important that the grantee saw each other as 

colleagues and mentors throughout this process, and 

this definitely paid off as part way through the 

project grantees began engaging more with each other 

informally through one-off conference calls or site 

visits to learn from each other. 

 So also communication is essential to the trust 

building process. It was important for us to 

recognize the differences across grantees though. We 

tried as much as possible to tailor the frequency and 

methods of communication to grantees, but we also 

wanted to ensure a certain regularity of 

communication to keep apprised of the project and 

identify any challenges early on. This frequency of 

communication was critical to the project’s success 

and where there were the most challenges were when 

issues were not brought up early on. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

31

 As Michelle described, there was constant tension 

between customizing each of the grantees evaluations 

and still maintaining consistency for the portfolio 

level evaluation. In several cases there were changes 

discussed at the grantee level, for example, 

modifications to data collection protocols or 

expansion of follow-up time windows that needed to be 

made for a specific study, but we needed to consider 

how alterations to that one study could affect the 

portfolio level study. And it was important to 

discuss that if an exception was made to one grantee 

to allow for greater customization, did that offer 

get extended to all the grantees? And if we do decide 

that, then we needed to think about how that will be 

communicated to grantees so that the messaging was 

clear and consistent. 

 For big decisions written communication was sent to 

all grantees, in addition to follow up verbal 

discussions to ensure consistency in messaging. This 

was a successful communication strategy for this 

project. It's also important to recognize what we 

don't know. E-TRY-A [phonetic] had been hired as the 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

32

evaluation experts for this project, but it was 

important for us to recognize that we didn't 

necessarily have the clinical, administrative, or 

cultural understanding of running healthcare clinics 

on the Texas-Mexico border. There were grantees with 

no evaluation experience that were part of this 

project and were huge assets to the study. Their 

expertise is critical in providing a perspective on 

their patients staff, and ,services provided, and 

what made sense evaluation-wise within the context of 

their setting, intervention, workflow, and capacity. 

All of these issues were critical considerations to 

ensuring the evaluation process was a success. 

 Lastly, Michelle discussed the partnership-centered 

evaluation approach earlier, and one of the biggest 

reasons that approach was able to be realized was 

because of the capacity-building that was conducted 

among all parties. The grantees expertise on a 

variety of issues build our own capacity, skills, and 

knowledge so we had a better understanding of doing 

this work on the border and in these settings. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

33

 From the evaluation perspective, especially with the 

capacity building approach, we hope to work our way 

out of a job. Our goal is not only to produce a highquality product, in that there are robust findings 

for these studies, but that in the process grantees 

skills, knowledge, and capacity have been increased 

so that they can continue integrating data monitoring 

and evaluation into their work for continuous 

improvements and future sustainability. 

 So I want to thank all the participants in this 

session for being here today and for what you do, and 

to CNCS for the opportunity to tell a bit about our 

story from this incredible experience. We have 

included here our contact information and related 

websites and are happy to answer questions here or 

outside the webinar. Thank you very much. 

ANDREA: Okay, thank you so much to Emily and Michelle 

and Lisa. And before we turn this over to Q&A, I'm 

going to have Dr. Kayla Cranston give some brief 

closing remarks Kayla? 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

34

KAYLA: Fantastic, thank you all so much. Can you can you 

hear me out there? Hopefully, yeah. So, thank you all 

so much. My name is Kayla Cranston and I am the 

Director of Strategy and Integration for something 

called Conservation Psychology at Antioch University 

of New England. And our job here at Antioch is to 

take everything that you just heard about, the 

participatory methods that we just heard all about, 

and apply them towards helping environmental 

professionals navigate that complex relationship 

between wildlife conservation and human wellbeing. 

And we all know, especially right now in the mid of 

COVID how important that relationship is, and so I'm 

just so grateful to hear the experts talk about how 

they've been doing it for years and years and years. 

 A few themes that I did hear throughout what we were 

talking about here was a lot about how hard 

institutionalization is of this stuff, which is 

something that for decades I think the world of 

conservation has been talking about. About why can't 

we do this? Right? It's really difficult to do this 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

35

moving forward and it sounds like you all are finding 

really creative solutions for how to move forward 

with that, and I appreciate that very much. The other 

attention that I was hearing a lot about that I'd 

love to hear a little bit more about in the follow-up 

Q&A is how do you negotiate kind of this tension 

between customizability and generalizability? And I 

think what's interesting here is that this right 

here, this question about the difference between the 

two is exactly what got me started on my dissertation 

research on capacity building, because there was that 

question of, like, how do you know what kind of 

participation is going to work, right, to build the 

capacity of the folks that are involved? While at the 

same time keeping that scientific rigor going on and 

that generalizability of your results, and so it is 

what inspired me to create a psychometric that would 

be able to measure, kind of at a universal level, the 

types of, you know, strategies that do and do not 

predict long-term behavior, which is what I'm hearing 

a lot of here is that what we want is this longerterm behavior. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

36

 So, I just think it's so cool that you guys have put 

together such great and robust work that really 

speaks to how one might go about doing that across 

multiple sites, and as, you know, a past Vista 

project director I appreciate that on a single site, 

you know, area and I also appreciate funders like 

AmeriCorps making sure that - and CNCS - making sure 

that these types of processes are really being put 

into practice in a rigorous way. So thank you all 

very much and I think we're going to hand it over to 

Q&A now. Yes? 

ANDREA: Yes, thanks Kayla. So I will pass this on to 

Dr. Melissa Gouge, who is in Florida right now. 

What's up? 

DR. GOUGE: Hi there, thank you. Thank you so much Andrea 

and thank you Kayla for your thoughtful closing 

remarks kind of highlighting just how important this 

work is, and I will be facilitating the Q&A here and 

noticed, initially, as people were saying hello, 

we've got people from the four corners of the United 

States joining us: Arizona, California, Washington, 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

37

we've got folks from Canada online as well, so we've 

got a lot of [unintelligible – audio cuts out] And 

perhaps some like-minded people seeing that the value 

of engaging those who are impacted by research as we 

devise, conduct, and implement research, and helping 

us do better research and also at the same time 

important, particularly for this moment, making us 

better members of the communities in which we live 

and those that we work alongside. 

 So, just to get started on the question and answer 

session, for everybody you can ask your question by 

typing it in the Q&A box, you can ask a question 

using the microphone on your computer by selecting 

the Raise Hand feature on the menu, and we can grant 

you microphone rights. 

 And we had a couple of initial questions come in. I 

think the person who had the first question was Ralph 

Holcombe, especially thinking about some of the 

issues on CBPR, participatory methods, especially 

during the challenges that we are experiencing under 

conditions of COVID-19. Would any of the speaker's 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

38

like take a stab at answering some of those initial 

thoughts that you're having as we're running through 

this? 

LISA: Sure, this is Lisa Wolff, I'll start off. Obviously 

there are lots of challenges within this COVID-19 

period. I think it's we're talking about just the 

level of engagements. Clearly there are challenges 

related to people, time, and what they can focus 

their efforts on, but from even a logistics 

standpoint, we do a lot of capacity-building not just 

in evaluation, but in other areas as well. And we've 

been continuing that work even through this period 

using virtual methods. 

 So, it's amazing those different platforms out there 

on such as Zoom, as an example, that has breakout 

rooms that people can have small chats and then come 

back together. And so obviously there is a question 

of what do people have as far as technological 

infrastructure? But if they do have that capacity to 

use those kinds of platforms, we have found that with 

some innovation and creativity were able to continue 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

39

our capacity-building and engagement efforts, even 

through a virtual setting. 

FEMALE: That's exciting to hear, I just wanted – 

FEMALE: Thank you so much, Lisa. 

KAYLA: I just wanted to briefly note, this is Kayla 

Cranston. Yeah, I just wanted to note that as you're 

talking it's reminding me of something that one of 

the presenters said about how important that hightouch like in-person work is, and I realize that 

that's a challenge right now, but I also think - I 

did a webinar on this topic for conservation 

professionals a few weeks ago, and I remember saying 

then kind of what I'll say now which is that we have 

an opportunity here, especially as conservation 

professionals and those who are not at the frontlines 

right now in the health care industry, to sit back 

and really think about how do we want to move forward 

with our work once we can get back within six feet of 

each other again. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

40

 And so I really believe that that as we move forward, 

taking this time to be like, okay, how exactly would 

we get back into this in a way that would be more 

participatory and support social justice issues as 

well moving forward? So, I think it's great that 

we're all here today to talk about how we might do 

that. 

DR. GOUGE: Yeah, absolutely. Thank Kayla for that 

reminder of maybe this is the way we can learn to do 

this better when we are able to be in the same space 

with one another again. I also would like to address 

Judy Stanley’s question. I think this might be kind 

of directed for Lisa. Judy asked, how do you know 

that the relationships, that you were kind of 

discussing, isn't actually interfering with the 

evaluation results? As if you're kind of producing a 

halo effect of sorts. 

LISA: This is Lisa, again. I think that's a great 

question. I think it depends on what the situation is 

and what it is you're actually evaluating. In our 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

41

case we were evaluating interventions that were 

mainly happening in a clinical setting. We were not –

We, as the evaluators, were not in that clinical 

setting. We did look at what was the level of 

fidelity of implementation of the interventions in 

our process, or implementation evaluation, we did 

collect data on that, and constantly report it back 

to the grantees so that data could be used to inform 

their implementation along the way. 

 So in that case, that pure scientific sort of 

question around to what extent were the interventions 

implemented to fidelity? The mere fact of us giving 

them back that data likely affected the answer to 

that question, but our ideal in this was that these 

interventions would be implemented as much to 

fidelity as possible so we could see if they had an 

impact on patient mental or physical health outcomes. 

And, so, in that case, in our discussions, and we all 

discussed whether or not it made sense to feed this 

information back, and for the purpose of our study it 

was we really wanted to see the true impact of these 

interventions, and we all hoped that they would be 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

42

implemented to fidelity as much as possible. So 

giving that information back was only sort of to help 

improve that. 

 So that action may have affected that level of 

implementation, but we don't think it affected in a 

sense the impact study outcomes because, the only 

thing, it hopefully made for a stronger effect, but 

it didn't necessarily sort of alter the study methods 

or data collection methods there. So I think it's a 

it's a challenging, it's a challenging issue to 

discuss. I don't know if, Michelle, you have anything 

else to comment related to the Si Texas project on 

that question. 

MICHELLE: No, Lisa, I think you covered what I would 

have said on that one as well. 

DR. GOUGE: All right, thank you, thank you Lisa. So, it 

sounds like yes, it's a kind of a balancing act in 

some ways, if you will. We have another question from 

Steven Keller, regarding are there any resources that 

are available to community organizations that they 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

43

could leverage during these challenging times? Xo any 

of you have suggestions for resources that you'd be 

willing to share with our listeners? 

MICHELLE: This is Michelle and I don't have specific 

links or information to share, kind of, off the top 

of my head, but I know there have been a lot of 

resources coming out of Chronicle of Philanthropy and 

other kind of nonprofit supporting organizations that 

have been doing a lot of work to summarize what the 

federal funding opportunities provide for nonprofits 

and community organizations, and providing data and 

information about how those organizations can be 

supported. So I would direct you to look at some of 

those national nonprofit supporting organization. 

KAYLA: I will also say just from an academic perspective, 

so my institution, Antioch, actually offers coaching 

and how to do this type of participatory work towards 

conservation for environmental professionals, but I 

also know that Northwestern does some type of work in 

that similar vein when they're looking a 

participatory, like asset-based approaches to 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

44

participatory methods. So, if you all are interested 

in wanting to kind of learn a little more about how 

to do this in any field, not just conservation, I 

think northwestern does have that asset-based 

community development center up there. 

DR. GOUGE: All right, thank you, thank you so much, 

Kayla. It is 2:56 and we have a just a couple of 

questions in the inbox we have not had a chance to 

get to and may not have an opportunity to, but those 

questions will be passed along to our presenters so 

we can follow up with you offline to answer those 

questions. And I just wanted to ask, briefly, if 

there's any quick final comments that any of our 

presenters would like to share. And I could start 

with Mary Hyde briefly and then move along through 

the presenters before we close things out. 

MARY: Hi Melissa, this is Mary, yes. I just wanted to 

thank the presenters today and for the great 

questions, and I did see that someone raised a 

question about how funders address the power 

differential, and I do think we should follow up with 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

45

that offline when we have a little bit more time 

because I think it's a critical question and an 

important one, but other than that thank you for 

everyone's participation today. 

DR. GOUGE: All right, thank you Mary, Emily, do you have 

any closing remarks to share? 

EMILY: Yeah, I would say that, you know, this is a 

challenging time for participatory research. We'd 

like to do things face-to-face, we like to interact 

with folks ,and you know in terms of dealing with a 

crisis like COVID-19, from my perspective it would be 

hard to try to get up and running immediately for 

this kind of work, but one of the great advantages of 

participatory research is having ongoing 

relationships in place that allow you to get that 

quick feedback to understand what people are facing. 

 So, if you already have those relationships 

established you're ahead of the game. So I say it's 

never too soon to get started with forming those 

relationships and getting that set up for whatever 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

46

might come up in terms of research or opportunities 

to work with your community. 

DR. GOUGE: Yes, absolutely thanks for sharing that 

insight, Emily. Michelle and Lisa, would either of 

you have a closing remarks? 

MICHELLE: This is Michelle. I just wanted to share that 

we've talked about several challenges and several 

challenges of stakeholder engagement, and 

participatory evaluation and research came up in the 

questions, and they are very real. It is hard work to 

do and it's not always perfect, we know the funder 

dynamics, the funder power dynamics are immense and 

unavoidable, but it is so worth it. I think the 

quality of the research that came out of our work I 

feel is so much more relevant to the organization's 

we worked with. And while there are still things 

maybe we would have done differently, I think the - 

It is worth the journey. It is not a clear and linear 

path, but it is very much worth the effort to ensure 

that your research and your evaluation is much more 

connected to the community you’re working with. 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

47

LISA: And this is Lisa. I just want to thank you so much 

for this opportunity. And I agree with the other 

speakers and I think it's even more important now 

more than ever, as we think about collaboration and a 

sense of ownership among community members and 

organizations, and a sense of agency so that these 

processes and results really can be used for 

sustainability and future efforts. I also think now 

more than ever, on cross-sectoral collaboration is 

critical as all of these issues are complicated and 

intertwined. 

DR. GOUGE: Yes, absolute. Thank you so much, Lisa. And 

Kayla? 

KAYLA: Yeah, absolutely. Well, again, thank you all so 

much. And to your point Lisa, the cross-sector is 

very important. And Michelle and Emily, I'm very glad 

that we have set up our follow-up calls already 

because I know I have a lot to learn from you both. 

As we move forward, I just want to share something, 

it's kind of along the lines of what Michelle had 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

48

mentioned about this being so worth it from a 

psychological perspective, which is the way I've been 

looking at this topic for my entire career. It really 

might help to note that there has been a lot of 

research that does show that psychologically, when 

you're talking about things like ownership and 

continued behavior over time, this, this right here 

is exactly how you get that. And the quality with 

which you engage your participants over time it has a 

direct link, in fact, predicts about 34% of whether 

or not they're going to continue to engage with 

whatever it is that you're attempting to get them to 

do over time, right, whether that is take care of 

themselves or engage with other healthcare 

practitioners or resources. 

 I mean this is where it's at, right, that science has 

shown that. So, thank you all so much for being here 

and showing us, even though it's difficult, it's 

definitely, definitely worth it. 

DR. GOUGE: All right, well on that note, thank You Kayla. 

It is the top of the hour and I want to be respectful 
ICF Transcription 

20238_CNCS_ORE_AprilWebinar_v02

49

of everyone's time. I just want to thank all of our 

speakers again and our audience for this wonderful 

discussion today. I’d like to thank Jenelle Azore and 

Lauren Rosen at ICF Next International and all of 

their colleagues for their technical support and 

coordination today. 

 We will be sending you a post webinar survey. So 

please let us know your thoughts on this webinar, 

ideas for future webinars, we would love to hear from 

you, and we will be posting this recording on our 

website in a month, if you'd like to take another 

look at it, listen again, and share. And I’d just 

like to thank you all so much for being here today. 

And I hope you have a good, safe, and healthy day. 

Thank you. 

END OF FILE