# Presentation: 037384

## Recruiting Sites and Participants for Random Assignment Studies

- IES/NCER Summer Institute
- June 25, 2007
- Fred Doolittle

## Overview of the Session Topics

- What is the purpose of site and participant selection?
- The art of recruiting sites for random assignment studies
- Topics to address
- Building the foundation for longer term success of the study
- Addressing the tough issues in advance
- Assuring cooperation in study implementation

## The Goal of a “Fair Test” of the Intervention/Program

- Some elements of the “fair test”
- Sites that serve the intended target group
- Sites that can operate the intervention reasonably well
- Sites that can provide the intended service contrast
- A research design that can produce credible findings on impacts and can be implemented
- A data collection plan that preserves the good impact design and documents differences in services and outcomes at the various stages of the theory of action
- An analysis strategy and report process that produces useful findings
- In various ways, all of these involve the “sites”

## Topics for a conversation

- The process of recruiting sites
- Embedding a random assignment design and data collection into normal program operations
- Asking the tough questions
- Real understanding of the design
- Political issues
- Ethical concerns
- Working to achieve long-term cooperation in study implementation
- Lack of crossovers and withdrawals
- Data collection

## Reading the Site Recruitment Context

## Factors to Consider

- How many sites do you need?
- What restrictions are there on your choices?
***Geographic***- need to be close to you, need to be a specific places because of funder, need to test in specific kinds of places, need to be representative
***Programmatic***-
- do sites need to be doing something on which you build
- not doing something so get service contrast
***Size ***– need to be big enough to provide the desired sample of “clusters”, teachers and/or kids

## Factors (cont.)

- Are you offering something new or wanting to study existing programs?
- Does what you offer fit into their existing plans programmatically and operationally so easy sell?
- After-school example
- Reading Professional Development example
- Is the existing program long established and popular?
- Is it oversubscribed so there is “scarcity?”
- How have selection decisions been made?
- Does it have outcome-based performance standards?
- Others????

## Recruiting Framework

- What is the benefit-cost balance in participating from the perspective of sites?
***Possible Benefits-*** the potential benefits for kids because of new services, staff professional development, the visibility of being in a study, site-level findings, etc
***Possible Costs*** – the need to devote management time to the intervention and study, a divergence from local priorities, hassle of integrating the intervention into local operations, controversy of random assignment, operational hassles of random assignment, the hassle of data collection, etc.

## Responding to the Site Recruitment Context

- The context affects the sequence of topics
- The more it looks like a tough sell, the more you lead with benefits of participating
- The more it looks like an easy sell, the more you move quickly through the benefits and into the roles and responsibilities
- In either case:
- The goal is to get folks interested in participating so they can help you solve the inevitable problems, find ways to lessen the costs, and identify locally-relevant benefits
- Always include the benefits in any material because it may be passed around to new folks who need to understand why to bother with this.

## Examples of Strategies

- Building site “recruitment” into special program funding decisions
- Still need engagement process
- Selecting “representative” sites
- When might this be appropriate?
- What is the pitch?
- Using data bases to identify prospects that fit the profile and then making “cold call” contact
- Data bases may not include key criteria
- With whom to make the contact?
- What is the pitch?
- Working through service provider or funding networks to gain access
- Pros and cons
- What is the pitch?
- Substantial outreach to build demand and stimulate “applications”
- What is the balance between benefits and obligations in initial outreach?
- How orchestrate the application and selection process?

## What to Expect When Make Contact

- In districts/schools/programs:
- Everyone is overburdened and stressed
- Many things are changing simultaneously
- A new intervention is of greater interest than the research
- Getting chosen for a study and/or getting something free might be regarded as significant--but not for long
- Evaluation results –either overall or site-specific - often do not matter much because way off in the future

## Recruiting Process

- What is the point of entry?
- The pros and cons of starting “high” in an organization
- Recovering from an initial rejection
- First dates (i.e., meetings) are pivotal
- Make a “good impression” by bringing the right info and people
- Get the right people from the site in the room
- Build relationships from the outset
- Be the buyer and the seller—explain the benefits but ask the tough questions relatively early
- Try to understand their perspective
- Be willing to dig in to details

## “Embedding” Random Assignment into Normal Operations

## Why Does This Matter?

- Your goal is to mesh the research procedures with their operations
- If you do this well, it significantly reduces the cost to a site of participating
- If you get them interested enough in the possible benefits, they will start helping you figure this out

## Understanding Normal Program Operations

- How does the “sample” usually get to the place (program, school, classroom, etc) where you want to introduce the intervention and do random assignment?
- Ask managers _and_ line staff
- Focus on understanding the usual way and any exceptions, alternative routes in, etc.
- Understand the timing of the steps and the information that is available at each step
- Understand the frequency of later “corrections” and how done
- Can be because of mobility of students, late hires, turnover
- Cannot “fix” this after random assignment

## Understanding Normal Program Operations (cont.)

- For the kids:
- Do kids apply? If so how and when?
- If not, how are appropriate kids identified? Who is involved? Counselors, other teachers, referral agencies? Do they expect all they refer to get in?
- Who decides which kids are “accepted”?
- When are these decisions made?
- What data is normally available or collected on the kids at various stages?
- What percent of those selected actually participate?

## Understanding Normal Program Operations (cont.)

- For the staff:
- How and when are staff assignments made?
- What is the usual skill set of staff?
- What other responsibilities do these staff have that might affect your ability to train on intervention?
- Are there other constraints, for example union rules?
- Matching kids with staff and services
- How and when is this normally done?
- Is there a tradition of special cases – parents, staff requesting changes?
- Is there a tradition of deciding some kids “just have to have” this staff person, program services, etc. and these are handled outside the usual process?

## Normal Program Operations (cont.)

- Ask about information and time available at key stages
- What info is available on kids at different stages – to see if eligible/appropriate for intervention or to serve as baseline data?
- Is there time to introduce informed consent and baseline data collection prior to random assignment?
- When will any research delays be a problem – because scheduling complicated, because teachers/kids/parents need to know what is up, etc. ?

## Examples

- Individual random assignment
- After-school services
- Upward Bound
- School level random assignment
- Elementary school teacher professional development in reading through summer institutes and coaching
- Middle school teacher professional development in math through summer institutes and coaching

## Focusing on Service Receipt

- Two parts to this are key because the ***service contrast*** is what drives impacts
***Program group:*** Want strong implementation of the intervention to intended kids, with intended participation
- Worry about mobility of kids and turnover of staff
***Control group:*** Want clear service contrast for the control group
- Not necessarily no service
- But clearly different services

## Picking the Point of Random Assignment

- How close to the start of services do you put the lottery?
- Choice affects:
- the question you address,
- the sample,
- the baseline data you will have, and
- service participation for program and control group
- The tradeoff:
- “Late” means greater participation rate for the program group but also more hassle for staff, harder to plan, more disappointment if not selected, and -usually - more motivated control group so more services for controls
- “Early” means the opposite

## Examples

- Individual level random assignment
- KIPP middle school evaluation
- Mentoring programs
- Cluster random assignment
- The previous professional development examples

## Setting Random Assignment Ratios

- This can be part of the site recruitment discussion
- A balanced design is best for power, but not necessary
- The power drop-off is not major until move past 2:1
- An unbalanced design can make a big difference to sites and help address ethical concerns
- A very unbalanced design with many sites is an option if data collection costs are manageable
- Avoiding empty program slots is important
- “Overbook” somewhat the program group
- Use a non-research waiting list to fill empty slots
- In extreme, can set program group size to fill available program slots and the control group is the rest

## Handling the Sensitive Cases

- Much better if do this prior to random assignment
- Tell programs if absolutely necessary they have a few “chips” they can use prior to the lottery for cases where they could not live with control group designation
- They apply to use a chip for a person or cluster before random assignment
- Affects generalizability but not internal validity
- In cluster random assignment, can include some “extra” non-study staff from program clusters in program services
- Sometimes have to accept a “crossover” to save a study
- Small numbers in a large sample do not matter a lot

## Don’t let post-lottery procedures undo the “randomness”

- Post-random assignment allocation of kids to clusters
- Compare school-level random assignment to teacher-level random assignment
- In the latter, important to get commitment of kids to clusters or a clear process to do it before conduct random assignment

## Avoid Undoing Randomness 

- Data collection problems
- Low rates for sample
- Differential rates across research groups
- Build tracking into design
- Try to mesh data collection with normal participation in services
- Create incentives
- Cover the costs at a minimum
- Provide positive incentives- there are OMB limits
- Special incentives for control sample?
- Clusters
- Individuals
- Pay for a local study liaison to coordinate data collection

## Asking the Tough Questions Early

## Tough Questions to Ask

- Do you really want to do this or is someone forcing you?
- Do you really understand random assignment?
- Are you committed to using a lottery to decide who gets the intervention and who does not?
- Do you really understand your roles and responsibilities related to:
- Research procedures
- Data collection
- Implementation of the intervention

## More Tough Things to Ask

- Who could object to the study procedures and are you prepared to confront these objections and stay the course?
- unions, parents, governing board, participant referral sources
- Can you devote the staff time needed to manage the study and the intervention?

## Even More Tough Questions to Ask

- Are you about to (likely to) do something else that will affect our ability to pull off the study?
- Change local priorities so not interested?
- Introduce some other similar intervention so not a service contrast?
- Lay off staff so cannot staff it?
- Study champion is about to retire or take a new job?

## Still More Tough Questions

- Are there lurking ethical concerns?
- Uncertainty about real scarcity?
- Need to recruit more to have a control group?
- Local certainty the intervention really works so should not deny access?
- Local certainty can identify those most in need who will benefit?
- Sense of coercion despite informed consent?
- ????

## Negotiating and Closing the Deal

## Negotiating a Formal Agreement

- A formal, detailed agreement is important
- It should specify respective roles and responsibilities about both program and research activities, timelines, costs
- The agreement must be cleared directly with all relevant decision makers
- The head person must read and sign
- This takes time but worth the effort in the long run

## Negotiating and setting the stage (cont.)

- Once signed an agreement is gradually forgotten by many
- Therefore, one must have at least one properly placed internal champion to push forward the study
- The champion must be tended and kept engaged via ongoing communication
- Other relationships also matter; make the rounds periodically