Domain: Educational Services
Question:
A public health nonprofit is running multiple peer support groups each week and needs to demonstrate participant outcomes such as increased connection, hope, motivation, recovery confidence and practical recovery tools. Over a six-month period, the program supports approximately 200 participants across all group sessions. Attendance varies across groups, and some participants attend once while others attend regularly. Therefore, the evaluation process must remain brief to avoid overwhelming participants or reducing engagement due to survey fatigue within the recovery population. The program coordinator is looking for evidence of effectiveness of the support groups to strengthen future funding opportunities within peer support, and peer support staff want feedback that helps them adjust and improve the group over time using more qualitative approaches. The evaluator must select an appropriate measure, interpret the result responsibly and summarize findings in a way that supports future planning and decision-making for the organization and staff.                

What is a realistic, low-burden approach to collecting evidence without causing evaluation fatigue? Please describe in detail the methods the evaluator should use to select assessment activities (and how many to use) and what characteristics are most likely to make them successful as evaluation tools. Please also describe how an evaluator should check-in with participants and program staff throughout this process and what they should avoid doing when checking in and assessing the effectiveness of the support groups. 

Please also include the following points: 
Sampling and administration strategy (including pilot, rotation/sampling to avoid over-surveying the same people, and modes of collection).

How to justify sample size and report minimum detectable effects given the 200-participant context and likely response rates.

How to integrate brief quantitative measures with qualitative feedback so staff get actionable insights.

How and when to check in with participants and program staff during implementation, what to share back, and what to avoid doing when checking in or using results.

Why this approach best fits the nonprofit's needs.

Please also consider and respond to the following: In a peer support group setting, outcomes like connection, hope, tools learned, and motivation can look differently across participants and may differ depending on attendance frequency and individual personal circumstances. That said, how should the evaluator clearly define what "success" looks like for each outcome and select measurable indicators that are realistic and appropriate enough for peer support, and useful for both program development and reporting needs?

Expected points in a correct answer:
Explains that pairing quantitative scale questions with open-ended qualitative prompts provides staff with actionable insights that quantitative scales alone would miss
Defines what success looks like in the peer recovery setting by offering a concise definition attached to increased connection, recovery confidence, and tools learned
Describes how to calculate necessary sample size based on desired effect size by conducting an example power analysis
Recommends prioritizing formative over summative assessment methods
Recommends limiting assessment activities to 1-2 key tools rather than collecting data at every session
Recommends a rotation or sampling strategy across different peer groups to avoid overburdening any single group
Recommends a short end-of-meeting reflection tool that can be completed in approximately one minute or less to capture immediate participant reactions.
Recommends monthly check-ins as the frequency for reviewing data and engaging with program staff during implementation.
States that survey instruments should contain clear and unbiased wording
States that demographic questions should only be included when directly connected to program goals
Recommends mixed methods designs by suggesting pairs of quantitative scale questions (e.g., "On a scale of 1-10, how effective has the program been for your overall recovery?") with brief open ended prompts
States that outcomes related to attitudes and values (such as hope, connection, and motivation) are best assessed through reflective methods
Recommends backwards design logic by stating that outcomes should be defined first in observable terms before selecting measurement tools
Provides a concrete definition of program outcomes by defining goals like 'increased connection' where participants feel less isolated, share personal experiences, and exchange contact information with peers
Provides a concrete definition of recovery confidence by explaining how participants should be able to articulate increased self-efficacy in surveys
Stresses what evaluators should avoid when checking in with participants by recommending evaluators avoid over-surveying the same groups, collecting incomplete data, or asking similar questions too frequently.
Explains why the recommended approach fits the nonprofit's context by addressing how it reduces participant fatigue, increases completion rates, and ensures the organization can analyze and use results
Advises that measures should be kept specific and actionable so results can guide program improvements
Acknowledges that peer support outcomes like hope, connection, and motivation are primarily attitudinal and values-based, requiring participant self-reporting through both quantitative and qualitative methods
Explains that formative methods offer quick feedback, repeated opportunities, and lower stakes for participants
Recommends a rotation or sampling strategy across different peer groups to avoid overburdening any single group while ensuring diverse viewpoints are captured
States that open-ended prompts capture attitudes and values that quantitative scales miss
Recommends strict rules for anonymous consent with no incentives offered for positive reviews
Recommends selecting measurement indicators that priortize consistency over volume in terms of feedback
Advises that evaluators should avoid vague rubric criteria focusing on indicators of whether specific competencies related to recovery are developing
Recommends matching assessment type to the learning/outcome goal (e.g., knowledge uses objective tests, skills use rubrics, attitudes use reflective writing)
Provides a concrete definition of "tools learned" as participants leaving with at least one coping skill or resource beneficial to their recovery
States that no identifying information should be collected from participants
Explains that focusing on core questions that directly map to program outcomes reduces participant fatigue and increases completion rates
States that since peer support outcomes like hope, connection, and motivation are primarily attitudinal and values-based, the evaluator should rely heavily on participant self-reporting
Recommends that measures should communicate impact clearly to leadership in addition to guiding program improvements