# Qualitative Evaluation of the Project P.A.T.H.S.: An Integration of Findings Based on Program Implementers

Authors: Daniel T. L. Shek
Journal: The Scientific World Journal
 (2012)

---

## Abstract

An integration of the qualitative evaluation findings collected from program implementers conducting the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) in different years (n=177 participants in 36 focus groups) was carried out. General qualitative data analyses utilizing intra and interrater reliability techniques were performed. Results showed that the descriptors used to describe the program and the metaphors named by the informants that could stand for the program were generally positive in nature. Program participants also perceived the program to be beneficial to the development of the students in different psychosocial domains. The present study further supports the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong based on the perspective of the program implementers.


---

## Body

## 1. Introduction

In the process of program evaluation, understanding the client’s perspective is usually the primary focus. One example is the use of the client satisfaction approach in capturing the views of the program participants. Comparatively speaking, the viewpoint of the program implementers about the program is not adequately explored in the evaluation literature [1]. There are several justifications for including the views of the program implementers. First, as pointed out by Peterson and Esbensen [2], “personnel, consciously or unconsciously, influence the effectiveness of prevention program, it is important to assess their perceptions when evaluating a specific program to provide insight into the context in which the program operates” (page 219). Second, according to utilization-focused evaluation [3], in order “to achieve more reliable and valid evaluations, a number of data sources and perspectives should be combined” ([4]; page 1225), the program implementers are one of the stakeholders who should be involved in the evaluation process. Third, with reference to the principle of triangulation, evaluation data based on various sources can help to cross-check program effectiveness across the data collected from different sources and help to paint a full picture of program effects. Fourth, because the program implementers have professional training and experience, they may give a good assessment of program effectiveness. Fifth, inclusion of the program implementers’ views in judging the program and their own performance can give them a sense of respect and fairness and avoid biases that generate from the evaluation data based on the clients only. Finally, evaluation that includes questions that ask the program implementers about program implementation and their own performance can facilitate their reflective practice, which enhances professional growth and development [5, 6].The proposal to evaluate the view of the program implementers is also highlighted in the existing evaluation frameworks. Although different evaluation emphases exist in the evaluation literature in the international context, there are common evaluation frameworks and standards that are maintained by researchers in the mainstream scientific community. For example, the Centers for Disease Control and Prevention [7] suggested a comprehensive framework for program evaluation in public health, in which the engagement of the stakeholders is an important step. Similar focus can be seen in other evaluation frameworks, such as the What Works Clearing House [8] in the context of education. Regarding evaluation standards, the Joint Committee on Standards for Education Evaluation [9] proposed several areas of evaluation criteria in different domains. In the above evaluation frameworks, engagement of the program implementers in the evaluation process is an indispensable step.Although the experimental/quantitative approach is the dominant approach in the field and it is commonly regarded as the gold standard, it is not the only option, and there are alternate approaches. For example, according to Patton [3], quantitative evaluation (thesis), qualitative evaluation (antithesis), and utilization-focused evaluation (synthesis) are different approaches to evaluation. There is more effort to carry out qualitative evaluation where the subjective viewpoints, qualitative data, and nonartificiality in the data collection process are emphasized.How can the views of program implementers be assessed? There are different ways to capture the views of the program implementers. For example, rating scales or single-item open-ended questions are used to understand the viewpoints of the program implementers in subjective outcome evaluation. Although qualitative subjective outcome evaluation is good, its method to assess implementers’ views by some open-ended questions in paper form lead to a lack of contact between the implementers and researchers. Therefore, it would be desirable to use other means, such as in-depth interviews and/or focus groups to collect qualitative data.Reviews of the literature show that there is a remarkable surge of interest in using focus groups in program evaluation in western countries. For example, Nabors and colleagues [10, 11] used focus groups for an assessment of program needs, strengths, and weaknesses, and to gain ideas for future program development. However, little has been documented about the use of focus groups in program evaluation in the Asian context. Twinn [12] criticized that “focus groups appear to have been used quite extensively with populations of black and Hispanic ethnic origins” (page 655) because this methodology has been originally developed for Anglo-Celtic populations [12].The focus group method has been used successfully to assess client satisfaction and quality assurance in a variety of fields. It has also become a popular method in program evaluation in many research contexts, such as health settings [13, 14]. Focus groups offer many potential advantages, such as being cost and time effective in collecting information. Morgan [15] noted that a focus group of eight people may generate more ideas than eight individual interviews. Clearly, the strength of the focus group method is that it brings clients together to discuss their perceptions about the services that they have received. This allows for interaction between group members, which stimulates thoughts and recall of experiences.Focus groups can be particularly helpful for the discovery of service problems and suggestions for fixing those problems [16]. Moreover, the data drawn from focus group interviews can be used to compare data gathered from other research methods, that is, to use focus groups for triangulation [17]. Along the same line, Conners and Franklin [18] provide a strong argument for the use of a qualitative methodology. They stressed that qualitative methodologies may address some concerns about surveys that result in inflated satisfaction scores, as clients are more critical when qualitative methodologies are used, and they have more freedom to express their concerns about all aspects of care in a way that is impossible with many studies. Therefore, qualitative methods are invaluable in providing depth to the exploration of people satisfaction that is not possible with quantitative surveys. As Merriam [19] stressed, “the product of a qualitative study is richly descriptive” (page 8). As such, qualitative evaluation via focus groups is an important strategy to capture the views of the program implementers.In the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes), the Tier 1 Program is a universal positive youth development program provided for secondary 1 to 3 students in Hong Kong. There were 52 schools that joined the experimental implementation phase (2005–2008) and more than 200 schools that joined the full implementation phase (2006–2009). Several studies have already documented the positive program effects based on the students’ objective and subjective outcomes collected from survey questionnaires [20–22]. Qualitative evaluation has also been conducted in order to understand the program effects of the Project P.A.T.H.S. in Hong Kong based on the perspective of the program participants [23, 24]. The related findings were integrated and presented in another paper by Shek and Sun in this special issue. On the other hand, qualitative evaluation based on focus group methodology has been carried out in order to understand the views of the program implementers [25, 26]. Again, it is illuminating if an integration of the existing qualitative studies based on the program implementers can be carried out. Thus, the present study attempted to integrate the existing qualitative evaluation findings based on the perspective of the program implementers in the experimental and full implementation phases of the Project P.A.T.H.S. in Hong Kong.


## 2. Methods

### 2.1. Participants and Procedures

From 2005 to 2009, the total number of schools that participated in the Project P.A.T.H.S. was 244, with 669 schools across all grades. Among them, 46.27% of the respondent schools adopted the full program (i.e., 20-hour program involving 40 units), whereas 53.73% of the respondent schools adopted the core program (i.e., 10-hour program involving 20 units).Instructor focus groups were conducted for the secondary 1 level in the 2005/2006 and 2006/2007 school years, for the secondary 2 level in the 2007/08 school year, and for the secondary 3 level in the 2007/08 and 2008/09 school years. A total of 36 schools were randomly selected in the study of focus group evaluation (14 schools for secondary 1, 9 for secondary 2, and 13 for secondary 3). Among them, 28 schools joined the full program and eight schools joined the core program. Thirty six focus groups consisting of 138 teachers and 39 social workers in total were conducted. The average number of classes per school was 4.83 (range: 3–6), and the average number of respondents per school was 5.11 (range: 1–14). The characteristics of the schools joining this process evaluation study can be seen in Table1.Table 1
Description of data characteristics from 2005–2009.
2005/06 (EIP-S1)2006/07 (FIP-S1)2007/08 (FIP-S2)2007/08 (EIP-S3)2008/09 (FIP-S3)Total schools that joined P.A.T.H.S.5220719648167(i) 10-hour program239511329104(ii) 20-hour program29112831963Total schools that joined this study599310(i) 10-hour program12203(ii) 20-hour program47737(a) No. of schools incorporated into formal curriculum35837(b) No. of schools incorporated into form teacher lessons or using other mode24103Average no. of classes per school5 (5)4.9 (3–6)4.9 (3–6)4.75 (4–6)4.6 (4–6)No. of instructor focus groups599310Total instructor respondents3861231342(i) Teachers275415834(ii) Social workers117858Average no. of respondents per group7.6 (3–12)6.8 (2–14)2.6 (1–5)4.3 (2–8)4.2 (2–6)Note: EIP: experimental implementation phase; FIP: full implementation phase; S1: secondary 1 level; S2: secondary 2 level; S3: secondary 3 level.As data collection and analyses in qualitative research are very labor intensive, it is the usual practice that small samples are used. In the present context, the number of focus groups and instructor participants can be regarded as respectable. In addition, the strategy of randomly selecting informants and schools that joined the Tier 1 Program can help to enhance the generalizability of the findings. An interview guide (Table2) was used for conducting focus group interviews with instructors. The interview questions were designed with reference to the CIPP (context, input, process, product) model and previous research [25, 26].Table 2
Interview guide for the instructor focus group.
(A)Context Evaluation(i) How much do you know about “Positive Youth Development Programs” (e.g., “life skills education”)? What is your overall impression of these programs?(ii) Have you taught programs that are similar to the Project P.A.T.H.S. before?(iii) If yes, how effective do you feel they are?(iv) From your perspective, what are the differences between the Project P.A.T.H.S. and other similar programs?(v) Do you agree with the vision of the Project P.A.T.H.S.? Why?(B)Input Evaluation(i) What kind of effects do you feel that the implementation of the Project P.A.T.H.S. have on the school’s normal operation?(ii) If the school incorporates the Project P.A.T.H.S. curriculum into the normal curriculum (e.g., life education, integrated humanities, etc.), from your perspective, what are the advantages and disadvantages of this arrangement?(iii) If the school does not incorporate the Project P.A.T.H.S. curriculum into the normal curriculum (e.g., homeroom, extracurricular activities, etc.), do you feel that this arrangement is successful?(iv) To accommodate the implementation of the Project P.A.T.H.S., did the school make special arrangements?(v) Do you feel that the principal and administrative staff support the implementation of the Project P.A.T.H.S. at your school? Why or why not?(vi) Do you feel that the training you received is adequate for you to carry out the program requirements?(C)Process Evaluation(1)General Impression of the program:(i) What is your overall impression of the program? What are your feelings?(ii) All in all, did you enjoy leading the program?(iii) Regarding the program, what has given you a lasting impression?(iv) While implementing the program, did you have any unforgettable experiences?(2)Comments on the program content:(i) Regarding the program, what are the things you like? And what are the things you dislike?(ii) What are your views on the different units and content of the program?(iii) Which units do you like the most? Why?(iv) From your recollection, are there any activities that aroused students’ interest to participate in the program?(3)Comments on the program implementation:(i) While implementing the program, did you encounter any difficulties?(ii) Do you feel that the program implementation was successful?(iii) To what degree/extent did you follow the program curriculum manuals? Why?(iv) What are your thoughts on the students’ responses to the program?(D)Product Evaluation(1)Evaluation of the general effectiveness of the program:(i) Do you feel that the program is beneficial to the development of adolescents?(ii) Have you noticed any changes in students after their participation in the program? If yes, what are the changes? (free elicitation)(iii) If you noticed changes in students, what do you think are the factors that have caused such changes?(iv) If you have not noticed changes in students, what do you think are the factors that have caused students not to change?(2)Evaluation of the specific effectiveness of the program:(i) Do you think that the program can promote students’ self-confidence/ability to face the future?(ii) Do you think that the program can enhance students’ abilities in different areas?Optional Questions(iii) Do you think that the program can enhance students’ spirituality aspect?(iv) Do you think that the program can promote the students’ bonding with family, teachers, and friends?(v) Do you think that the program can establish students’ compassion and care for others?(vi) Do you think that the program can promote students’ participation and care for society?(vii) Do you think that the program can promote students’ sense of responsibility to society, family, teachers, and peers?(3)The program’s impact on the instructor:(i) Do you feel you have gained something by leading this program? And have you lost something?(ii) If you have the opportunity in the future, do you wish to lead similar programs again?(4)Other comments:(i) If you are invited to use three descriptive words to describe the program, what are the three words that you would use?(ii) If you are invited to use one incident, object/thing, or feeling (e.g., indigestion, enjoyment, child at heart, etc.) to describe the program, how would you describe the program?A total of 36 focus groups designed to elicit implementers’ perceptions of the Project P.A.T.H.S. were conducted. All focus group interviews were jointly conducted by two trained colleagues. During the interviews, the respondents were encouraged to verbalize their views and perceptions of the program. In the interviews, the interviewers adopted the role of facilitators and were conscious of being open to accommodate both positive and negative experiences expressed by the informants. As the interviewers either had training in social group work and/or substantial group work experience, they were conscious of the importance of encouraging the informants to express views of a different nature, including both positive and negative views. The interviews were audio recorded, with the respondents’ consent. The audio recordings were then fully transcribed and checked for accuracy.The data were analyzed by two trained research assistants. After initial coding, the positivity nature of the codes was determined, with four possibilities (positive code, negative code, neutral code, and undecided code). The coding and categorization were further cross-checked by another trained research assistant. To enhance the reliability of the coding on the positivity nature of the raw codes, both intra and interrater reliability were carried out. For intrarater reliability, two research assistants who had been involved in the coding individually coded 20 randomly selected responses for each question. For interrater reliability, another two research assistants who had not been involved in the data collection and analyses coded 20 randomly selected responses for each question without knowing the original codes given at the end of the scoring process with reference to the finalized codes.In qualitative research, it is important to consider ideological biases and preoccupations of the researchers. As program developers, the author might have the preoccupation that the implemented program was good and it was beneficial to the students. Additionally, the researchers might have the tendency to focus on positive evidence rather than negative evidence. Thus, several safeguards against the subtle influence of such ideological biases and preoccupations were included in the present study. To begin with, the researchers were conscious of the existence of ideological preoccupations (e.g., positive youth development programs are beneficial to adolescents) and conducted data collection and analyses in a disciplined manner. Second, both inter and intrarater reliability checks on the coding were carried out. Third, multiple researchers and research assistants were involved in the data collection and analysis processes. Fourth, the author was conscious of the importance and development of audit trails. The audio files, transcriptions, and steps involved in the development of the coding system were properly documented and systematically organized.


## 2.1. Participants and Procedures

From 2005 to 2009, the total number of schools that participated in the Project P.A.T.H.S. was 244, with 669 schools across all grades. Among them, 46.27% of the respondent schools adopted the full program (i.e., 20-hour program involving 40 units), whereas 53.73% of the respondent schools adopted the core program (i.e., 10-hour program involving 20 units).Instructor focus groups were conducted for the secondary 1 level in the 2005/2006 and 2006/2007 school years, for the secondary 2 level in the 2007/08 school year, and for the secondary 3 level in the 2007/08 and 2008/09 school years. A total of 36 schools were randomly selected in the study of focus group evaluation (14 schools for secondary 1, 9 for secondary 2, and 13 for secondary 3). Among them, 28 schools joined the full program and eight schools joined the core program. Thirty six focus groups consisting of 138 teachers and 39 social workers in total were conducted. The average number of classes per school was 4.83 (range: 3–6), and the average number of respondents per school was 5.11 (range: 1–14). The characteristics of the schools joining this process evaluation study can be seen in Table1.Table 1
Description of data characteristics from 2005–2009.
2005/06 (EIP-S1)2006/07 (FIP-S1)2007/08 (FIP-S2)2007/08 (EIP-S3)2008/09 (FIP-S3)Total schools that joined P.A.T.H.S.5220719648167(i) 10-hour program239511329104(ii) 20-hour program29112831963Total schools that joined this study599310(i) 10-hour program12203(ii) 20-hour program47737(a) No. of schools incorporated into formal curriculum35837(b) No. of schools incorporated into form teacher lessons or using other mode24103Average no. of classes per school5 (5)4.9 (3–6)4.9 (3–6)4.75 (4–6)4.6 (4–6)No. of instructor focus groups599310Total instructor respondents3861231342(i) Teachers275415834(ii) Social workers117858Average no. of respondents per group7.6 (3–12)6.8 (2–14)2.6 (1–5)4.3 (2–8)4.2 (2–6)Note: EIP: experimental implementation phase; FIP: full implementation phase; S1: secondary 1 level; S2: secondary 2 level; S3: secondary 3 level.As data collection and analyses in qualitative research are very labor intensive, it is the usual practice that small samples are used. In the present context, the number of focus groups and instructor participants can be regarded as respectable. In addition, the strategy of randomly selecting informants and schools that joined the Tier 1 Program can help to enhance the generalizability of the findings. An interview guide (Table2) was used for conducting focus group interviews with instructors. The interview questions were designed with reference to the CIPP (context, input, process, product) model and previous research [25, 26].Table 2
Interview guide for the instructor focus group.
(A)Context Evaluation(i) How much do you know about “Positive Youth Development Programs” (e.g., “life skills education”)? What is your overall impression of these programs?(ii) Have you taught programs that are similar to the Project P.A.T.H.S. before?(iii) If yes, how effective do you feel they are?(iv) From your perspective, what are the differences between the Project P.A.T.H.S. and other similar programs?(v) Do you agree with the vision of the Project P.A.T.H.S.? Why?(B)Input Evaluation(i) What kind of effects do you feel that the implementation of the Project P.A.T.H.S. have on the school’s normal operation?(ii) If the school incorporates the Project P.A.T.H.S. curriculum into the normal curriculum (e.g., life education, integrated humanities, etc.), from your perspective, what are the advantages and disadvantages of this arrangement?(iii) If the school does not incorporate the Project P.A.T.H.S. curriculum into the normal curriculum (e.g., homeroom, extracurricular activities, etc.), do you feel that this arrangement is successful?(iv) To accommodate the implementation of the Project P.A.T.H.S., did the school make special arrangements?(v) Do you feel that the principal and administrative staff support the implementation of the Project P.A.T.H.S. at your school? Why or why not?(vi) Do you feel that the training you received is adequate for you to carry out the program requirements?(C)Process Evaluation(1)General Impression of the program:(i) What is your overall impression of the program? What are your feelings?(ii) All in all, did you enjoy leading the program?(iii) Regarding the program, what has given you a lasting impression?(iv) While implementing the program, did you have any unforgettable experiences?(2)Comments on the program content:(i) Regarding the program, what are the things you like? And what are the things you dislike?(ii) What are your views on the different units and content of the program?(iii) Which units do you like the most? Why?(iv) From your recollection, are there any activities that aroused students’ interest to participate in the program?(3)Comments on the program implementation:(i) While implementing the program, did you encounter any difficulties?(ii) Do you feel that the program implementation was successful?(iii) To what degree/extent did you follow the program curriculum manuals? Why?(iv) What are your thoughts on the students’ responses to the program?(D)Product Evaluation(1)Evaluation of the general effectiveness of the program:(i) Do you feel that the program is beneficial to the development of adolescents?(ii) Have you noticed any changes in students after their participation in the program? If yes, what are the changes? (free elicitation)(iii) If you noticed changes in students, what do you think are the factors that have caused such changes?(iv) If you have not noticed changes in students, what do you think are the factors that have caused students not to change?(2)Evaluation of the specific effectiveness of the program:(i) Do you think that the program can promote students’ self-confidence/ability to face the future?(ii) Do you think that the program can enhance students’ abilities in different areas?Optional Questions(iii) Do you think that the program can enhance students’ spirituality aspect?(iv) Do you think that the program can promote the students’ bonding with family, teachers, and friends?(v) Do you think that the program can establish students’ compassion and care for others?(vi) Do you think that the program can promote students’ participation and care for society?(vii) Do you think that the program can promote students’ sense of responsibility to society, family, teachers, and peers?(3)The program’s impact on the instructor:(i) Do you feel you have gained something by leading this program? And have you lost something?(ii) If you have the opportunity in the future, do you wish to lead similar programs again?(4)Other comments:(i) If you are invited to use three descriptive words to describe the program, what are the three words that you would use?(ii) If you are invited to use one incident, object/thing, or feeling (e.g., indigestion, enjoyment, child at heart, etc.) to describe the program, how would you describe the program?A total of 36 focus groups designed to elicit implementers’ perceptions of the Project P.A.T.H.S. were conducted. All focus group interviews were jointly conducted by two trained colleagues. During the interviews, the respondents were encouraged to verbalize their views and perceptions of the program. In the interviews, the interviewers adopted the role of facilitators and were conscious of being open to accommodate both positive and negative experiences expressed by the informants. As the interviewers either had training in social group work and/or substantial group work experience, they were conscious of the importance of encouraging the informants to express views of a different nature, including both positive and negative views. The interviews were audio recorded, with the respondents’ consent. The audio recordings were then fully transcribed and checked for accuracy.The data were analyzed by two trained research assistants. After initial coding, the positivity nature of the codes was determined, with four possibilities (positive code, negative code, neutral code, and undecided code). The coding and categorization were further cross-checked by another trained research assistant. To enhance the reliability of the coding on the positivity nature of the raw codes, both intra and interrater reliability were carried out. For intrarater reliability, two research assistants who had been involved in the coding individually coded 20 randomly selected responses for each question. For interrater reliability, another two research assistants who had not been involved in the data collection and analyses coded 20 randomly selected responses for each question without knowing the original codes given at the end of the scoring process with reference to the finalized codes.In qualitative research, it is important to consider ideological biases and preoccupations of the researchers. As program developers, the author might have the preoccupation that the implemented program was good and it was beneficial to the students. Additionally, the researchers might have the tendency to focus on positive evidence rather than negative evidence. Thus, several safeguards against the subtle influence of such ideological biases and preoccupations were included in the present study. To begin with, the researchers were conscious of the existence of ideological preoccupations (e.g., positive youth development programs are beneficial to adolescents) and conducted data collection and analyses in a disciplined manner. Second, both inter and intrarater reliability checks on the coding were carried out. Third, multiple researchers and research assistants were involved in the data collection and analysis processes. Fourth, the author was conscious of the importance and development of audit trails. The audio files, transcriptions, and steps involved in the development of the coding system were properly documented and systematically organized.


## 3. Results

In this paper, qualitative findings on the following three areas are presented: (1) descriptors that were used by the informants to describe the program, (2) metaphors (i.e., incidents, objects, or feelings) that were used by the informants to depict the program, and (3) implementers’ perceptions of the benefits of the program to students.For the descriptors used by the informants to describe the program, there were 270 raw descriptors that could be further categorized into 133 categories (Table3). Among these descriptors, 169 (62.6%) were coded as positive and 7% were classified as neutral in nature. In order to examine the reliability of the coding, two research assistants who did the coding of raw data recoded 20 randomly selected raw descriptors at the end of the scoring process, and the average intrarater agreement percentage calculated on the positivity of the coding from these descriptors was 92% (range: 80–100%). Finally, these 20 randomly selected descriptors were coded by another two research staff members who did not know the original codes given, and the average interrater agreement percentage calculated on the positivity of the coding was 88.5% (range: 80–95%).Table 3
Categorization of the descriptors used by the program implementers to describe the program.
Descriptors2005/06 (EIP-S1)2006/07 (FIP-S1)2007/08 (FIP-S2)2007/08 (EIP-S3)2008/09 (FIP-S3)Total (% of total responses)Positive responsesHappy/glad/enjoy1311410Togetherness11Project with great investment11Adequate resources for students11Rich in content/comprehensive1113Challenging11Good11Clear rationale11Abundant22Self-reflection11Back to the origin of education11Role modeling11Great influence on students11New experience11Diversified/diverse1621111Wide scope, focused, and diversified11The students liked the program activities11Lively11Positive/Very positive43310Interactive415Fun and relaxed99Relaxing/very relaxing1236Systematic3115Enlightening11Meaningful4127Novel44Innovative33Practical/very practical213Clear11Focused11In-depth11All rounded66Zealous44Prospective22Cognitive enhancement11Fruitful/very fruitful448Sometimes touching11Match the topic very much11Interesting4116Effective123Step by step11Rare11Excited11Good feelings/satisfied213Worthy to implement11Closely connected with life11Have gains112Have positive expectation11Hardworking11Up-to-date information11Sharing11Good elements11Flexible11Respectful11Unlimited11Very useful11Preventive11Inspiring44Necessary11Important/very important22Reflective11Welcomed11Developmental11Impressive11Very good idea11Beneficial11Constructive11Quite good11Worthwhile11Well suited11Start11Ideal11Very magnificent11Pleasure comes through toil11Subtotal (% of total responses in each academic year)19 (61.3)58 (54.2)36 (81.8)9 (45.0)47 (69.1)169 (62.6)Negative responsesA bit rushed11Rushed/very rushed123Superficial11Could not fully apply the things learned11Heavy workload for teachers11Chaotic55To be improved11Difficult628Useless22Confused11Worried112Superficial819Helpless22Inadequate11Overlapping22Lack of connection11Overgeneralized11Not practical33Senseless11Too rich content within insufficient time11Too aggressive33Demanding and inept11Could not meet students’ needs44Headache11Lack of reflection11Too wide (scope)11Lack of time33Unrealistic11Painful213Not interested in11Impoverished11Trying to win in chaos11In war11Harsh/very harsh44Not well suited11Inadequate support11Like water off a duck’s back11Subtotal (% of total responses in each academic year)4 (12.9)47 (44.0)5 (11.4)6 (30.0)14 (20.6)76 (28.1)Neutral responsesStressful11Positive, but superficial11The program was comprehensive but needs to be enriched11Like a competition11Having a heart, but no strength11Bittersweet11Partially uncertain11Depends on individual11Task oriented11So-so33Rational11Emotional11Long awaited11Enormous11Very academic11Intensive112Subtotal (% of total responses in each academic year)4 (12.9)2 (1.9)2 (4.5)5 (25.0)6 (8.8)19 (7.0)UndecidedEffectiveness depends on teachers’ readiness11Beyond our power to do it11Struggling with program adherence11Program effectiveness was in doubt11Exclamation mark11Aggressive11Subtotal (% of total responses in each academic year)4 (12.9)01 (2.3)01 (1.5)6 (2.2)Total count31 (100)107 (100)44 (100)20 (100)68 (100)270 (100)For the metaphors that were used by the informants that could stand for the program, there were 72 raw objects involving 128 related attributes (Table4). Results showed that 40 metaphors (55.6%) and 65 related attributes (50.8%) were classified as positive in nature, while 26 metaphors (36.1%) and 47 related attributes (36.7%) were regarded as neutral responses. Reliability tests showed that the average intrarater agreement percentage calculated on the positivity of the coding from these metaphors was 89% (range: 80–100%), whereas the average interrater agreement percentage calculated on the positivity of the coding was 91% (range: 80–100%).Table 4
Categorization of the metaphors used by instructors to describe the program.
Nature of responseNo. of responses towards the nature of the metaphor2005/06(EIP-S1)2006/07 (FIP-S1)2007/08 (FIP-S2)2007/08 (EIP-S3)2008/09 (FIP-S3)Total (%)Positive items (%)  (e.g., photographs, street light, seeding, and cash box)3 (37.5)14 (43.75)9 (64.3)5 (83.3)9 (75)40 (55.6)Negative items (%)  (e.g., indigestion, tasteless water,   rowing upstream, and firework)2 (25)3 (9.4)001 (8.3)6 (8.3)Neutral items (%)  (e.g., bottle neck, perceiving the   elephant in blind, durian, and magic box)3 (37.5)15 (46.9)5 (35.7)1 (16.7)2 (16.7)26 (36.1)Total count (%)8 (100)32 (100)14 (100)6 (100)12 (100)72(100)No. of codes derived from the metaphorPositive items (%) (e.g., photographs, street light, seeding, and cash box)2 (25)26 (53.1)11 (50)5 (83.3)21 (48.9)65 (50.8)Negative items (%)  (e.g., indigestion, tasteless water, rowing upstream, and firework)5 (62.5)5 (10.2)1 (4.5)05 (11.6)16 (12.5)Neutral items (%)  (e.g., bottle neck, perceiving the  elephant in blind, durian, and magic box)1 (12.5)18 (36.7)10 (45.5)1 (16.7)17 (39.5)47 (36.7)Total count (%)8 (100)49 (100)22 (100)6 (100)43 (100)128 (100)The perceived benefits of the program to the program participants are shown in Table5. There were 518 meaningful responses decoded from the raw data that could be categorized into several levels, which are benefits at the societal level, familial level, interpersonal level, personal level, general benefits, and benefits to instructors. The findings showed that 404 responses (78%) were coded as positive responses and 64 responses (12.36%) were counted as neutral responses. In order to examine the reliability of coding, the research assistants recoded 20 randomly selected responses, with knowledge of the original codes given at the end of the scoring process. The average intrarater agreement percentage calculated from these responses was 91.5% (range: 85–97.5%). The raw benefit categories were coded again by another two research staff members who did not know the original codes given. The average interrater agreement percentage calculated from these responses was 89.5% (range: 85–92.5%).Table 5
Categorization of instructors’ responses on the perceived benefits of the Tier 1 Program.
Area of competenceSubcategoryBenefitsS105-06S106-07S207-08S307-08S308-09TotalSocietal levelSocial responsibility and affairsEnhanced understanding of mother country11Increased awareness of citizen’s responsibility11Subtotal (%)0002 (3.8)02(0.4)Familial levelFamily relationshipsImproved communication and relationship with family2327Subtotal (%)02 (1.0)03 (5.7)2 (2.4)7(1.4)Enhanced instructor-student relationship and understanding49205846Learned teamwork11General interpersonal competenceImproved peer relationships, understanding, and cooperation2136122Enhanced social skills99Learned to handle love relationship33Total in subcategory623358981Enhanced interpersonal relationship11Improved communication skills22Interpersonal levelReduced bullying behavior11Delayed gossiping11Learned how to handle conflicts/avoid conflicts213Specific interpersonal  competenceLearned how to treat people and deal with issues33Increased ability and willingness to express oneself5107224Cultivated proper views on dating11Used learned materials to help or teach others11Leadership112Learned to appreciate, accept, care, and respect others2233212Total in subcategory1318114551Subtotal (%)19 (33.3)41 (20.2)46 (38.3)12 (22.6)14 (16.5)132 (25.5)Personal levelDelayed misbehavior11Behavioral competenceTook initiative235Strengthened positive behaviors55Total in subcategory1280011Enhanced problem-solving skills3115Cognitive competenceLearned critical thinking2523517General enhancement11Total in subcategory2834623Emotional competenceEnhanced ability in handling emotions22Enhanced emotional management33Total in subcategory023005Moral competence and virtuesEnhanced sense of equality33Enhanced moral competence1247Total in subcategory1320410Beliefs in the futureFacilitated goal setting and realization of goals11Beliefs in the futureIncreased understanding of the study path in the future33Total in subcategory000314Enhanced self-understanding1517Promoted self-enrichment33Enhance personal growth/maturity31329Enhanced self-confidence101112Positive selfEnhanced self-efficacy22Became more active213Promoted sense of success11Broadened students’ horizon123Total in subcategory12419540Enhanced self-reflection4244923Personal levelSpiritualityImproved morality/spirituality33Enhanced understanding purpose of life369Total in subcategory45413935General resilience112ResilienceBe more persistent when facing difficulties11Learned how to seek help11Total in subcategory010214Significant positive influences1719Some kind of help161430Cultivated potentials11General gainsEnhanced motivation for learning11Better academic achievement11Applied what learned to daily life213Gained recognitions and encouragement from instructors235Total in subcategory732501550Subtotal (%)16 (28.1)48 (23.6)46 (38.3)31 (58.5)41 (48.2)182 (35.1)Difficult to measure11The program was useful22Misbehavior could be controlled11Misbehavior was not widespread11Effectiveness depended on individual students112Effective to those students with positive values11General benefitsPositive commentsBenefit to study11Enhanced concentration in class11Effectiveness shown in long run729Unable to assess the effectiveness in a short time22Introduced personal development education into education system11Others3333Total in subcategory53473655Could not learn anything145Unhelpful99Not much change5510Unable to help students with special needs11Negative commentsUnable to assess the effectiveness in a short time11Students’ changes were doubtful88Less effective when compared with the Adolescent Health Project22Ineffective to those students with distorted values11Total in subcategory92610137Effectiveness could be observed, but students’ interest in the program was declining11General benefitsDifficult to measure1111Not much change22Neutral commentsNeeded to refer to objective data11Effectiveness depended on the students’ learning attitude224Students’ changes were doubtful11617Unable to assess the effectiveness in a short time15823Others55Total in subcategory338202164The effectiveness was doubtful11Unable to assess the effectiveness in a short time134UndecidedUnable to perceive immediate changes in students themselves145Difficult to measure22Others11Total in subcategory30100013Subtotal (%)20 (35.1)98 (48.3)20 (16.7)3(5.7)28 (32.9)169 (32.6)Enhanced understanding towards students17210Learned a lot from the program content/teaching experiences178OthersBenefits to instructorsEnhanced knowledge and development77Promoting schools’ concern on student development11Subtotal (%)2 (3.5)14 (6.9)8 (6.7)2 (3.8)026 (5)Total count (%)57(100)203 (100)120 (100)53(100)85(100)518 (100)Grand total count in percentage11.0%39.2%23.2%10.2%16.4%100%


## 4. Discussion

As Donnermeyer and Wurschmidt [27] pointed out, implementers’ “level of enthusiasm and support for a prevention curriculum influences their effectiveness because their attitudes are communicated both explicitly and subtly to students during the time it is taught and throughout the remainder of the school day” (page 259-260). Therefore, understanding their views is very important. The purpose of this study was to evaluate the Tier 1 Program of the Project P.A.T.H.S. using findings based on focus groups involving program implementers in the experimental and full implementation phases (2005–2009) of the project. There are several characteristics of this study. First, a large sample of participants (n=177 in 36 focus groups) participated in the study. Second, different datasets collected at different points of time were included in this integrative study. Third, implementers of the program in different grades were invited to participate in the study. Fourth, this is the first known scientific study of focus group evaluation of a positive youth development program based on program implementers in China. Finally, this is also the first focus group evaluation study based on such a large sample of program implementers in the global context.Based on the integrative analyses, two salient observations can be highlighted from the findings collected from different cohorts of students. First, the program was perceived positively from the perspective of the program implementers (Tables3 and 4). The program implementers generally used positive descriptors and metaphors to describe the program. Although some implementers perceived the program in a negative light, this is not the dominant view. Second, results in Table 5 show that the program had a beneficial effect on the participants, with 78% of the responses coded as positive. Generally speaking, benefits in both the personal and interpersonal levels were observed. The above observations are generally consistent with the qualitative evaluation findings based on the program participants reported by Shek and Sun in this special issue. In short, different stakeholders had positive perceptions of the program, program implementers, and perceived benefits of the program. Based on the principle of triangulation, the present study and the previous findings suggest that based on both quantitative and qualitative evaluation findings collected from program participants and program implementers, research findings suggest that the Tier 1 Program of the Project P.A.T.H.S. is effective in promoting holistic development of the program participants.There is a growing trend for using focus group methodology in order to understand the views of stakeholders in the field of evaluation, and the number of qualitative evaluation studies is increasing in the field. For example, Chen et al. [28] employed different evaluation methods (including qualitative evaluation) and pointed out that there were several limitations in employing participatory evaluation with at-risk youth. Mahoney et al. [29] used qualitative methodology to evaluate a tobacco prevention program among 5th grade students using impressions from classroom teachers and program presenters. Pedersen et al. [30] examined relationship quality in a community mentoring program via qualitative methodology. O’Rourke and Key [31] evaluated a school-based youth development peer group with integrated medical care using focus groups. Scheer and Gavazzi [32] used focus groups to evaluate the program “Families and Systems Teams Initiative.” In line with the above examples, the present study demonstrates the value of focus group methodology in evaluation contexts.In qualitative studies, it is important to examine alternative explanations [33]. The first alternative explanation is that the positive findings are a result of demand characteristics. However, this explanation is not likely because the informants were encouraged to voice their views without restriction and negative voices were, in fact, heard. In addition, there is no reason to believe that the participants acted favorably to please the researchers. The second alternative explanation is that the findings are due to selection bias. However, this argument cannot stand as the schools and program implementers were randomly selected. The third alternative explanation is that the positive findings are due to ideological biases of the researchers. As several safeguards were used to reduce bias in the data collection and analysis process, including calculation of intra and interrater reliability, this possibility is not high. Finally, it may be argued that the perceived benefits are due to other youth enhancement programs. However, this argument can be partially dismissed because none of the schools in this study participated in the major youth enhancement programs in Hong Kong, including the Adolescent Health Project and Understanding the Adolescent Project. In addition, participants in the focus group interviews were specifically asked only about the program effects of the P.A.T.H.S. Project.There are several limitations of the study. First, although the number of schools and workers participating in the study can be regarded as on the high side according to the common practice in mainstream qualitative evaluation studies, it would be helpful if more schools and workers could be recruited. Second, besides one-shot focus group interviews, regular and ongoing qualitative evaluation data could be collected. Third, although focus group interview data were collected, inclusion of other qualitative evaluation strategies, such as in-depth individual interviews, would be helpful in order to further understand the subjective experiences of the program implementers. Despite the above limitations, the present qualitative findings based on the experiences of program implementers showed that the respondents had positive perceptions of the program and implementers, and they perceived benefits of the programs throughout the years.



---

*Source: 591816-2012-05-15.xml*