Evaluating The Children’s Aid Society -

Carrera In-School Teen Pregnancy 

Prevention Program 

2011-2014

 Tulsa and New York City

Prepared July, 2016 

Accord

16 Main Street

Accord, NY 12404

845-626-2126

Fax 845-626-3206

Sacramento

24 East Main 

Street

Winters, CA 95694

530-795-3618

Fax 530-795-3619
i

Acknowledgements 

 This report is based upon work supported by the Social Innovation Fund (SIF) which 

unites public and private resources to evaluate and grow innovative community-based solutions 

with evidence of results. The Social Innovation Fund is a program of the Corporation for 

National and Community Service, a federal agency that engages more than 5 million Americans 

in service through its AmeriCorps, Senior Corps, Social Innovation Fund, and Volunteer 

Generation Fund programs, and leads the President's national call to service initiative, United We 

Serve. 

 The authors of this report would like to acknowledge the CAS-Carrera staff and the staff 

at the schools that housed the in-school CAS-Carrera program in Tulsa and New York City for 

their continued diligence throughout the evaluation process. We would also like to thank the 

Edna McConnell Clark Foundation for its continued support. 
ii

Table of Contents 

Executive 

Summary ...........................................................................................................................................

1

The Children’s Aid Society (CAS)-Carrera 

Program ............................................................................................................................................

6

Section I: The CAS-Carrera In-School Model in 

Tulsa ..................................................................................................................................................

9

Methods .................................................................................................................................

10

Implementation ....................................................................................................................

15

Program 

Outcomes ..............................................................................................................................

17

Long-term Program 

Outcomes ..................................................................................................................

17

Pregnancy .......................................................................................................

17

Mid-term Program 

Outcomes ..................................................................................................................

17

Sexual Behavior and Contraceptive 

Use .................................................................................................................

17

Violence and 

Delinquency ...................................................................................................

18

Drugs and 

Alcohol ...........................................................................................................

19

Academic 

Indicators........................................................................................................

21

Short-term Program 

Outcomes ..................................................................................................................

22

Family Life and Sexuality Education 

(FLSE)............................................................................................................

22
iii

Financial 

Literacy ..........................................................................................................

22

Program Service 

Objectives..................................................................................................................

23

Bank 

Accounts ........................................................................................................

23

Health 

Care ................................................................................................................

23

Regression Analysis: 

Tulsa ......................................................................................................................................

24

Section II: The CAS-Carrera In-School Model in New York 

City ....................................................................................................................................................

25

Methods .................................................................................................................................

26

Implementation ....................................................................................................................

34

Program 

Outcomes ..............................................................................................................................

39

Long-term Program 

Outcomes ..................................................................................................................

39

Pregnancy .......................................................................................................

39

Mid-term Program 

Outcomes ..................................................................................................................

40

Sexual Behavior and Contraceptive 

Use .................................................................................................................

40

Violence and 

Delinquency ...................................................................................................

41

Drugs and 

Alcohol ...........................................................................................................

41
iv

Regression Analysis: 

NYC .......................................................................................................................................

43

Conclusion ........................................................................................................................................

44

Appendix A: Youth 

Survey................................................................................................................................................

47

Appendix B: Teacher 

Survey................................................................................................................................................

55

Appendix C: Statistical Tests in 

Tulsa ..................................................................................................................................................

58

Appendix D: Statistical Tests in 

NYC ...................................................................................................................................................

64
1

Executive Summary 

The Children’s Aid Society (CAS)-Carrera Adolescent Pregnancy Prevention Program 

(CAS-Carrera) is a school-based adaptation of an evidence-based program designed to develop 

young people’s capacity and desire to avoid parenthood and other risky behaviors during 

adolescence, and help them break the cycle of poverty and despair affecting their full 

development. CAS-Carrera uses a holistic, “above the waist” approach to ensure young people 

develop ambitious personal goals, improve their sexual literacy, and cultivate aspirations for a 

productive future. Guided by a philosophy that sees youth as "at promise" rather than "at risk," 

CAS-Carrera begins working with boys and girls in middle school at ages 10 or 11, and follows 

them through high school graduation and college admission. The program provides participants 

with seven integrated, scientifically accurate1, and age-appropriate components including: 

 daily academic enrichment, 

 weekly exposure to the “world of work”/Job Club,

 weekly Power Group/mental health services, 

 weekly family life and sexuality education (FLSE), 

 comprehensive medical and dental services as needed, 

 regularly scheduled lifetime individual sports, and

 regularly scheduled self-expression activities. 

 The program components are designed to work in concert to build developmental 

competency, identity formation, and other social and emotional youth development outcomes, 

ultimately leading to the avoidance of early parenthood. Initially offered as an after-school 

program, this evaluation focuses on CAS-Carrera’s Integrated School Model approach.

 Starting in the 2011-12 academic school year, two quasi-experimental evaluations were 

conducted: one in New York City and the other in Tulsa. 

Philliber Research & Evaluation (Philliber) collected survey data from 150 Tulsa CASCarrera program youth who were surveyed over three years from February, 2012, through 

February, 2015. Philliber also collected survey data from 153 comparison youth from February,

2012, through February, 2014. Data were also collected from 219 NYC program youth in Fall 

2011 and Fall 2013. This report shows key risk behaviors and progress indicators among 

program students in Tulsa and then compares them to non-program students who were surveyed 

in the same grades one year earlier – a lagged cohort design. Additionally, NYC program youth 

are compared to samples of similar students selected from the 2011 and 2013 Youth Risk 

Behavior Surveillance (YRBS) databases from New York City maintained by the Centers for 

Disease Control and Prevention (CDC). Matching procedures were used to select the YRBS 

samples. 

 All outcome data were gathered from self-report surveys of youth, providing potential for 

self-report bias. Additionally, there is a possibility of differential bias, meaning the CAS-Carrera 

program experience may teach youth the socially desirable response. To minimize such bias, 

1

 Reviewed by U.S Department of Health and Human Services and Office of Adolescent Health. 

 
2

students were assured of confidentiality and asked to be honest in their answers. Additionally, 

given this was an intent-to-treat evaluation design, an attempt was made to survey all students 

who were surveyed in Fall 2011 (February, 2012, for Tulsa) in subsequent data collections, 

regardless of whether they were actively attending the CAS-Carrera program. 

 The confirmatory hypotheses in this evaluation are stated as follows. Students exposed to 

the CAS-Carrera program relative to comparison youth will have: 

 Lower rates of teen pregnancy (long-term outcome), 

 Higher rates of sexual abstinence (mid-term outcome), 

 Higher rates of contraceptive use (mid-term outcome), and 

 Fewer risk behaviors such as fighting, drug or alcohol use, or arrests (mid-term 

outcomes). 

 Short-term outcomes include knowledge about family life and sexuality education topics, 

as well as financial literacy. Additionally, there are a number of program service objectives. 

Relative to comparison youth, program youth should be more likely to have bank accounts and 

show improved receipt of medical care including annual comprehensive medical, dental, 

reproductive, and specialty care, psychosocial assessments, Hepatitis B and HPV vaccinations, 

and care from sites other than the emergency room. 

What happened in Tulsa? 

 In Tulsa, the program began by serving 6th graders. The evaluation followed these young 

people into the 9th grade. Specifically:

 48% were female; 

 50% were Latino or African-American, 22% were Caucasian, and 23% were multi-ethnic; 

 100% were middle school students at baseline; 

 39% did not live in two-parent homes, 28% received public assistance or welfare, and 9% 

lived with an unemployed adult; 

 50% had mothers who graduated from high school and 36% had fathers with this much 

education; 

 39% had mothers with at least some college experience and 25% had fathers with such 

experience. 

 Relative to comparison youth who were in the 9th grade one year before the program 

students, what are the statistically significant outcomes for Tulsa 9th grade program youth? The 

analysis of outcomes focused on the 9th graders, after three years of exposure to the program. 

Program participation had a positive, independent, and significant effect on two program service 

objectives (eye exams and bank accounts), and on two short-term program outcomes (family life 

and sexuality knowledge and financial literacy). Specifically, compared to 9th grade comparison 

youth, 9th grade program youth: 

 were 70% more likely to report having an eye exam within the past year, 

 were 68% more likely to report having a bank account, 

 scored 27% higher on a scale measuring family life and sexuality knowledge, and 

 scored 18% higher on a scale measuring financial literacy. 
3

 While not significant, program youth were 9% less likely to have reported ever having 

sex (p<.067) and 3% less likely to have ever been pregnant/caused a pregnancy (p<.052), 

controlling for the effects of demographic variables. 

 We do not yet find statistically significant differences in the program’s desired long-term 

and mid-term outcomes in Tulsa – sexual activity, pregnancy (although no pregnancies were 

reported among program participants), or use of contraception. These differences may be more 

substantial in subsequent years when participants are older and more at risk of these behaviors. 

What happened in NYC? 

 The New York City evaluation began in 2011 with a design similar to that in Tulsa – a 

lagged cohort design using the class one year ahead of the class receiving program as 

comparison. However, it became clear that the chosen comparison group was also receiving 

program services and thus, could not serve as a no-treatment comparison. Instead, two samples 

of youth were pulled from the 2011 and 2013 NYC YRBS survey data sets, creating an exact 

match of the program and comparison youth on gender, age, grade, ethnicity, and borough. This 

is a limited set of variables on which to match so these comparison data might be best viewed as 

providing context for the outcomes of the program students. Overall, we compared 219 

treatment youth in NYC to 219 YRBS comparison youth in both 2011 and 2013. 

The NYC program youth included in this study were in grades 9 through 12. Specifically 

in 2011: 

 68% were female; 

 62% were African-American, 19% were Latino, and 11% were multi-ethnic; 

 41% were in the 9th grade, 19% were in the 10th grade, and 40% were in the 11th grade. 

 40% were from the Bronx, 42% were from Brooklyn, and 18% were from Manhattan. 

in 2013: 

 73% were female; 

 59% were African-American, 27% were Latino, and 11% were multi-ethnic. 

 1% were in the 9th grade, 41% were in the 10th grade, 39% were in the 11th grade, and 19% 

were in the 12th grade. 

 40% were from the Bronx, 41% were from Brooklyn, and 19% were from Manhattan. 

 Relative to comparison youth, what are the statistically significant outcomes for NYC 

program youth? Results for the 2011 sample indicate that those in the CAS-Carrera program had 

significantly lower pregnancy rates than the YRBS sample (long-term outcome). Additionally, 

students in the CAS-Carrera program were more likely to use condoms at last intercourse, 

engaged in less physical fighting during the past 12 months, were less likely to have ever smoked 

cigarettes, less likely to have smoked cigarettes in the past 30 days, and less likely to have used 

alcohol and/or marijuana in the past 30 days (all mid-term outcomes). Specifically, compared to 

2011 YRBS comparison youth, 2011 program youth were: 

 7% less likely to report having been pregnant/caused a pregnancy, 

 33% more likely to report using a condom at last intercourse, 
4

 13% less likely to report having been in a physical fight during the past 12 months, 

 20% less likely to report ever smoking cigarettes, 

 6% less likely to report having smoked cigarettes during the past 30 days, 

 14% less likely to report having used alcohol during the past 30 days, and 

 16% less likely to report having used marijuana during the past 30 days. 

 Results also indicate that, among the 2013 sample, for students in the program group 

there is a positive, independent, and significant effect on four mid-term outcomes (condom use at 

last intercourse, physical fighting during the past 12 months, ever smoked cigarettes, and 

marijuana use in the past 30 days). Specifically, compared to 2013 YRBS comparison youth, 

2013 program youth were: 

 18% more likely to report using a condom at last intercourse, 

 16% less likely to report having been in a physical fight during the past 12 months, 

 15% less likely to report ever smoking cigarettes, and 

 9% less likely to report having used marijuana during the past 30 days. 

Implementation 

 Implementation data suggest the program was generally delivered with regularity in 

Tulsa but not as much in the New York City sites. There was moderate staff turnover in both 

NYC and Tulsa. Fidelity data and daily attendance data were not collected during this study. 

Discussion 

 

 The Tulsa and NYC sites are, of course, very different especially in age, grade and 

ethnicity. They serve different populations of students in different settings. In Tulsa, 

comparison data came from students in the same school system, whereas in NYC the CDC does 

not disclose the school sites from which the student surveys are collected. Still, some of the 

trends found in these two locations are similar. 

 The CAS-Carrera model is strong enough to saturate a school. While this challenged our 

original design, it is a positive indicator of the potential of this integrated in-school model. The 

in-school version of the program can be delivered at lower cost than the after-school version 

given the larger number of youth served in the in-school model. The in-school model reaches a 

larger number of youth than a typical after-school program and serves students who might not 

normally join an after-school program. 

 However, there are challenges to this delivery strategy. The lower student to staff ratio in 

a typical after-school program is not present in the in-school model. A typical after-school CASCarrera program serves approximately 50 to 60 youth, whereas the in-school model can serve 

over 200 youth per grade. Additionally, the program needs to adapt to the scheduling, logistical, 

and administrative demands of the school, constraints that are less problematic in an after-school 

program. Still, the in-school model allows for many full-time staff that an after-school program 

cannot accommodate. Additionally, program staff believe that while the in-school version of the 

program has some challenges, the benefits of being in the school outweigh 
5

these challenges since it allows for greater contact with the students and better linkages with 

students’ teachers.

 Because of the limitations of this study in NYC with regard to the YRBS comparison 

data and the underpowered sample sizes in Tulsa, this research is at best suggestive of the 

model’s potential and additional study is recommended to better measure the effectiveness of the 

in-school version of the CAS-Carrera program model. 
6

The Children’s Aid Society (CAS)-Carrera Program 

 In 1984, Dr. Michael A. Carrera and The Children's Aid Society (CAS) developed an 

Adolescent Pregnancy Prevention Program that uses a holistic approach to empower youth, to 

help them develop personal goals and the desire for a productive future, in addition to developing 

their sexual literacy and educating them about the consequences of sexual activity. In its after-school 

model, the program worked with boys and girls 13-15 years old and followed them through high 

school and beyond. Guided by a philosophy that sees youth as “at promise” instead of “at risk,” 

CAS-Carrera works to develop a participant’s capacity and desire to avoid pregnancy. The 

program model also provides opportunities for young people to discover interests and develop 

talents, as well as emphasizing education and employment. CAS-Carrera sees the sum of these 

activities as having a contraceptive effect. 

The model includes seven fundamental components: 

1. Education: Daily (and Saturday) homework help, remediation, and enrichment with trained 

teachers and tutors driven by Individual Academic Plans (IAP) for each participant; 

2. Employment/Job Club: Weekly exposure to the “world of work,” including earning 

stipends, opening bank accounts, exploring career choices, and participating in 

entrepreneurial projects; 

3. Family Life and Sexuality Education (FLSE): Weekly comprehensive, medically accurate 

sexuality education sessions taught in an age-appropriate fashion by a trained professional; 

4. Self-Expression: Weekly music, dance, writing and drama workshops led by theater and art 

professionals, where children can discover talents and build self-esteem; 

5. Lifetime Individual Sports: A fitness program emphasizing sports that build self-discipline 

and can be played throughout life, including golf, squash, swimming, and others; 

6. Full Medical and Dental Care: Comprehensive medical and dental services provided in 

partnership with local providers; 

7. Mental Health Services: Weekly discussion groups led by licensed social workers; 

individual counseling, case management, and crisis intervention as needed. 

 As an after-school program, this approach has been evaluated (1997-2000) using a 

random assignment design across six sites in New York City and elsewhere—a strong level of 

evidence. Results of the evaluation in New York City found that, compared to controls, girls in 
7

the program were 18% less likely to have had sex, 55% less likely to become pregnant and 80% 

more likely to use dual methods of contraception at last intercourse.2

 In 2006, CAS-Carrera staff members created a school-based model of the program, 

delivering all of its original services over several years but integrating the program into the 

normal school day. Beginning in Fall 2008, the program has been offered in four schools in New 

York City and one school in Tulsa (beginning in February 2012). These sites are: 

1. Academy for Scholarship and Entrepreneurship (ASE), Bronx , NY 

2. Bronx Preparatory Charter (BPC), Bronx, NY 

3. Opportunity Charter School (OCS), Manhattan (Harlem), NY 

4. Urban Assembly Institute of Math and Science for Young Women (UAI), Brooklyn, NY 

5. Union Public School (UPS), Tulsa, OK3

CAS-Carrera provides its programming to young people in underserved communities, 

characterized by higher than average rates of poverty, unemployment, teen pregnancy, and high 

school dropout rates. In NYC, CAS-Carrera is serving high-needs communities in Brooklyn, 

Harlem, and the Bronx. Additionally, at the time of initial engagement with stakeholders, Tulsa had 

the second highest birth rate in the country among teens aged 15 to 19. 

 The staffing structure in a CAS-Carrera program includes: one Program Director, one 

Community Organizer, and individual component leaders and social workers (licensed mental 

health professionals). The Program Director is responsible for day-to-day operations of the 

program, including guidance, training, and supervision of program staff. This person works 

directly with school leadership staff and insures the program maintains fidelity to the model. 

The Community Organizer builds relationships between youth, their families, and program staff. 

They may work with local agencies, make home visits, coordinate medical/dental appointments, 

and even chaperon medical/dental visits. This person serves as a back-up to the Program Director 

when he/she is unavailable. All staff are monitored closely by CAS-Carrera implementation/ 

fidelity managers. 

 Programming is delivered to students by trained CAS-Carrera program facilitators who 

receive ongoing staff development. All staff are experienced professionals with an 

undergraduate degree in a relevant discipline; many have advanced degrees and training. Staff 

also have component/content knowledge and youth development experience. CAS-Carrera

organizes orientations and ongoing trainings to ensure that (new) staff are appropriately 

acclimated to the model’s core principles and philosophy and are sufficiently trained to be able 

to execute the model as prescribed by their role. In addition, CAS-Carrera staff provide TA for 

effective program delivery and overall problem solving. 

 

2

 Philliber, S., Williams Kaye, J., Herrling, S., & West, E. (2002). Preventing pregnancy and improving health care 

access among teenagers: An evaluation of the Children’s Aid Society–Carrera Program. Perspectives on Sexual and 

Reproductive Health, 34(5), 244–251. 

3

 There are actually three separate school buildings about a half mile apart in Tulsa: Union 6th and 7th Grade Center, 

Union 8th Grade Center, and Union 9th Grade Center. 
8

 This report focuses on the in-school integrated model in these five schools. In this model, 

students attend weekly CAS-Carrera components during the school day as class periods. Class time 

for components is provided during the school day through advisory and guidance periods, resource 

time, and other time slots where CAS-Carrera program staff can provide programming. The 

medical/dental component is offered as needed. In this model, new cohorts of youth are added each 

program year as the previous cohort moves up in grade which increases the size of the population of 

program participants (doubling in the second year, tripling in the third year, etc). An early study of 

the integrated school model conducted by Brigham Nahas Research Associates4 in 2008 stated that 

this approach “allows for broader reach, serving many more students that can be accommodated in 

after school programs, allowing for greater economies of scale and, as such, a lower cost per 

participant.” This approach ensures content dosage over a long period of time and provides for swift 

recognition and follow-up of mental health and medical/dental problems – the issues which can 

negatively affect learning and growth. The logic model for the program shows its main activities, 

and its expected shorter-term and longer-term objectives. 

 

CAS-Carrera Program Logic Model

Activities Program Service 

Objectives

Shorter-term Objectives Longer-term 

Objectives

School leadership 

and Carrera staff for 

each school are 

trained in program 

theory, philosophy, 

logic model, and full 

curricula for each 

component and 

philosophy

Students will receive 

all core components 

of the program 

including:

 Job club

 Academic 

assistance

 Medical and dental 

care

 Family life and 

sexuality education

 Lifetime individual 

sports

 Self-expression

 Mental health 

(Power group)

Relative to 

comparison students, 

program students 

will:

 Have bank 

accounts.

 Have improved 

receipt of 

medical care 

including annual 

comprehensive 

medical, dental, 

reproductive, 

and specialty 

care 

psychosocial 

assessments, 

Hep B and HPV 

vaccinations, 

and care from 

sites other than 

the ER.

Short-term Outcomes

Relative to comparison students, 

program students will:

 Gain greater knowledge about 

sexuality and reproduction.

 Gain greater financial literacy.

Mid-term Outcomes

 Be more likely to be sexually 

abstinent.

 If sexually active, be more likely 

to use protection from pregnancy 

and STIs.

 Improve their academic 

performance.

 Engage in fewer risk behaviors 

such as fighting, drug or alcohol 

use, or arrests.

Teachers with students in the program 

will be more likely than teachers 

whose students have not been exposed 

to:

Relative to comparison 

students, program 

students will have:

 Lower rates of teen 

pregnancy.

 Lower rates of teen 

births.

In future years beyond 

the current research, 

program students will 

have:

 Higher rates of 

high school 

graduation.

 Higher rates of 

college admission.

 

4 Children’s Aid Society/Carrera Integrated School Model: Documentation of Early Implementation in Four 

Schools. Brigham Nahas Research Associates; Roblyn Anderson Brigham and Jennifer Nahas. October 2008. 
9

 Report more school engagement 

among their students.

 Report that student problems 

receive swifter intervention. 

 Report they are able to increase 

their instructional time. 

Section I: The CAS-Carrera In-School Model 

in Tulsa 
10

Methods 

 In Fall 2011, staff and administrators at the Union 6th and 7th Grade Center (Union Public 

Schools) in Tulsa selected over 200 6th grade youth to participate in the CAS-Carrera program. 

These youth were perceived to be those with the greatest social, emotional, and academic needs. 

Specifically they chose youth who lacked positive adult mentors, who needed regular 

medical/dental/vision care, had low scores on state assessments, were below grade level in 

reading, had a parent or a sibling who was a teen parent, had disengaged parents, and/or who had 

a known history of trauma (domestic violence, abuse, separation from family, instability at 

home, multiple losses). Parents were given the choice to opt out of the program. These selection 

criteria were also used to select comparison youth from the student cohort just above the 6th

graders in the program (7th graders). 

 Data collection began in February, 2012 with a survey of 201 6th graders receiving the 

program and 192 comparison students in the 7th grade. The numbers surveyed in each 

subsequent year are shown in Table 1 below. Thus, program youth can be compared to the 

students one grade ahead of them after a delay of one year—a lagged cohort design. Those 

surveyed in February 2012 comprised an intent-to-treat design and were rigorously followed for 

three years, whether or not they received three years of programming or were retained in the 

school. Overall, there were 150 program youth and 153 comparison youth who were surveyed at 

all three data collection points as 7th graders, 8th graders, and 9th graders. Eighth graders attend 

the Union 8th Grade Center and 9th graders attend the Union 9

th Grade Center. 

Table 1: Number surveyed in Tulsa over time 

Data Collection Point 6

th grade 7th grade 8th grade 9th grade

February 2012 201 program 192 comparison

February 2013 181 program 173 comparison 

February 2014 182 program 155 comparison

February 2015 178 program

 In Tulsa, generally the 7th grade program youth had one year of program exposure 

(dosage), 8

th graders had two years, and 9th graders had been in the program for three years. 

Inactive program youth may have been removed from the program by their parents, and both 

program and comparison youth might have moved or transferred to other schools. In Tulsa, the 

program and comparison youth are housed in separate buildings a half mile apart so there is little 

or no contamination. This analysis however, is somewhat under-powered, given the small 

sample sizes.

 The attrition rate among program youth was 17% and among comparison youth it was 

20%. These youth were lost to follow-up and were not surveyed. The program youth surveyed 

in February 2012 who were not surveyed in subsequent data collections were compared to those 

who were surveyed. These program youth were significantly more likely to live with neither 

parent, to live in families who receive public assistance, to not report parental employment, 

and/or to not report parental education. This suggests that they are more “at-risk” than students 

surveyed every year. 
11

 At final follow-up, 13% of the surveyed Tulsa program youth included in this report were 

inactive youth (they had left the study schools at some point during the three-year period). These 

youth did not differ significantly from those who remained active in the study for all three years. 

 Still, as shown in Table 2, the resulting analytic samples were well matched. There were 

no significant differences in grade seven (the first comparison point) between the program and 

comparison students, except age. On average, comparison youth were two months older. Both 

samples are ethnically and socioeconomically diverse. 

Table 2: Demographics and family characteristics among program 

and comparison youth in Tulsa in 7th grade 

Demographics Program Tulsa

(n=150)

Comparison 

Tulsa (n=153)

Gender:

Male

Female

52%

48%

58%

42%

Ethnicity:

African-American

Caribbean

Latino

Caucasian

Asian

Multi-ethnic

Other

23%

0%

27%

22%

3%

23%

2%

20%

1%

21%

29%

1%

23%

5%

Average age: 13.1 13.3**

Living situation:

Two-parents

Single parent

Other

61%

33%

6%

66%

29%

5%

% in foster care: 1% 2%

% who live in family that receives public assistance/welfare: 28% 20%

% who live with unemployed adult: 9% 7%

Mother’s education:

Less than high school

High school graduate

Some college

College graduate or more

Don’t know/not reported

11%

11%

17%

22%

39%

11%

21%

13%

21%

34%

Father’s education:

Less than high school

High school graduate

Some college

College graduate or more

Don’t know/not reported

10%

11%

12%

13%

54%

8%

19%

7%

18%

48%

Difference is statistically significant at **01.

 
12

Student survey data 

The evaluation team collected and computer-entered all data, using a self-report survey 

(see Appendix A) including questions about demographics, family composition, socio-economic 

status, health care, educational aspirations and other school-related items, employment, financial 

literacy, recreational activities, drug use, violence and delinquency, involvement in the juvenile 

justice system, communication, sexuality knowledge, sexual activity, contraceptive use, and 

pregnancy. The 30-item family life and sexuality education (FLSE) knowledge items were 

developed by CAS-Carrera staff to track knowledge gained about communication, parental 

involvement, puberty/human development, sexual and reproductive anatomy, STI/HIV 

prevention and treatment, social and emotional development, conception/pregnancy/ 

contraception, sex and gender, and sexual abstinence. 

All data were computer-entered at Philliber and subsequently analyzed. Demographic 

data were presented as frequencies with computed means as appropriate. Outcome data were 

analyzed using ANOVAs. Regression analyses were performed to determine if program 

participation was related to outcomes. To predict outcome variables, the independent variables 

were: program assignment, gender, age, and ethnicity. 

Teacher survey data 

 As part of this evaluation, Philliber surveyed teachers online in 2012 and 2013 to assess 

their perceptions of the program in their schools (see Appendix B). In Spring 2012, 10 out of 12 

teachers responded, and in Spring 2013, 39 out of 44 teachers responded. Questions included 

items measuring their involvement and knowledge of the program, interaction with program 

staff, perceptions of student improvement, perceptions about the effectiveness of the 

components, timeliness of the intervention, and perceptions of program impacts. 

Student attendance data (dosage) 

 Daily attendance data were unavailable for this study. However, dosage can generally be 

inferred based on length of stay in the program given the in-school program structure. In Tulsa, 

7

th graders had generally been in the program for one year, 8th graders for two years, and 9th

graders for three years. 

Validity 

 The level of evidence in this evaluation is preliminary at best. In Tulsa, participants went 

through a selection process for inclusion in this study. 

Logistics 

 Philliber staff collected all of the teacher surveys via Survey Monkey. Additionally, over 

98% of the student survey data were collected by Philliber staff during yearly scheduled oneweek data collections. Philliber staff read the survey items and response choices to all students. 

Youth were reassured of confidentiality and asked to be honest in their 
13

answers. If youth were not present during the one-week scheduled data collections, CASCarrera program staff collected survey data at a later time. These program staff had youth 

complete the surveys in a quiet area with no distractions. These youth were instructed to seal 

their completed surveys in an envelope and deliver them to the Program Coordinator who then 

mailed the surveys to Philliber. CAS-Carrera program staff never viewed student data. 

Inactivity and attrition 

 Since this was an intent-to-treat design, Philliber attempted to resurvey every youth who 

was surveyed at baseline in subsequent data collections. Philliber mailed surveys to inactive 

youth (those who had left the study schools) and also made follow-up phone calls to remind 

inactive youth to complete their surveys. All middle school youth received a $15 cash stipend 

and high school youth received a $25 cash stipend for a completed survey. At final follow-up, 

13% of the Tulsa program youth included in this report were inactive (they had left the study 

schools at some point during the three-year period). These surveyed inactive youth did not differ 

from those who remained active in the study for all three years. 

 The attrition rate among program youth was 17%. Youth leave the schools due to 

transferring to a different school or moving to a new neighborhood and either cannot be found or 

decline to be surveyed. Students also transfer to new schools to receive specialized academic 

concentrations or sports. 

Data storage 

 All data were computer-entered into SPSS data files with assigned Philliber IDs. Student 

names were kept separately and were not linked to survey data. All paper data from this study 

were kept in a locked storage facility. There were no problems with agreements for data access, 

storage, use, or reporting. There were no changes to Memoranda of Understanding (MOU’s).

 Data from each annual data collection were stored in separate SPSS files and merged to 

create one “master file”. Each year’s data had different variable names to help distinguish one 

year from the next (i.e. ever had sex1, ever had sex2, ever had sex3, etc). 

Data cleaning and missing data

 All data went through an extensive cleaning process. All surveys were checked for 

accuracy to ensure data entry was correct. During data analysis, inconsistent data were removed 

depending on the variables being measured. For example, if a participant checked “yes” for 

having sex at baseline but then checked “no” on a subsequent survey, this participant’s data were 

removed from that calculation. If a participant checked “yes” for having sex in the past three 

months but left the ever had sex question missing, the missing data were filled in. This logic 

applied throughout the analysis. Philliber imputed all variable values possible. For example, if a 

student was Latina on the first two surveys, and left ethnicity blank on the last survey, she was 

coded as Latina. If the data on ever had sex and use of contraception were inconsistent, (i.e., 

denies ever had sex but uses contraception), older surveys were consulted and a weight-of-
14

evidence approach was used. Throughout this study, missing data were less than 5% on any 

given indicator. 

IRB 

 This study was fully approved by an IRB. There were no changes made to the IRB 

protocol or to personnel on the evaluation team during the study. Currently the study is closed. 

A budget change in 2014 added additional funds to pay for the final Tulsa data collection. 

Evaluator role/involvement and budget 

 There were no changes regarding the role of the evaluator. The senior staff member 

overseeing this evaluation at Philliber remained consistent for the duration of this study.

Levels of significance 

Throughout this report, asterisks (*) indicate whether or not a difference or change is 

statistically significant. The “levels of significance” indicate the likelihood that an observed 

difference or change may be due to chance. Thus, when differences are marked with one asterisk 

(*), it is an indication that the difference would occur by chance alone less than 5% of the time. 

Two asterisks (**) denote differences that could have occurred by chance alone less than 1% of 

the time, and three asterisks (***) mean that only 1 time in 1,000 would a difference this large 

occur by chance alone. In other words, these differences are likely to be “real” rather than being 

statistical artifacts produced by sampling errors. This report includes a large number of 

analytical comparisons and thus some likelihood that differences may emerge by chance. For 

example, there were 37 comparisons made between 9th grade program and comparison youth in 

Tulsa: 29 (or 78%) favored the program youth and 5 (13%) favored the comparison youth. 
15

Implementation 

 It is important to note that the Tulsa site currently serves a much larger number of 

program youth than are included in this report. This section of the report reflects only those 

youth who were surveyed in February 2012 and subsequently followed for three years as per the 

design of this study. 

 In a typical after-school CAS-Carrera program, the academic component is offered daily 

(Monday through Friday), and Job Club, FLSE, and Mental Health are offered weekly. The 

dosage requirement for self-expression and individual lifetime sports is to have at least two 

exposures per year which are offered as part of weekly scheduled programming, and/or during 

school breaks and summer. Medical services/referrals are offered as needed. In the in-school 

model, the academic component is offered daily, and Job Club, Mental Health, and FLSE are 

offered weekly (Table 3). Tulsa program staff report that during the first program year (February 

through June 2012), program services/curricula were not delivered with as much regularity as in 

subsequent years. Some components were being delivered through classroom pull-outs. 

Table 3: Component delivery 

Site and components 2011-20125 2012-2013 2013-2014 2014-2015

Tulsa

Academic

Job Club

FLSE

Mental Health

Self-expression

Lifetime individual sports

Medical services/referrals

Daily

Weekly

Weekly

Weekly

Weekly

Weekly

As needed

Daily

Weekly

Weekly

Weekly

Weekly

Weekly

As needed

Daily

Weekly

Weekly

Weekly

Weekly

Weekly

As needed

Daily

Weekly

Weekly

Weekly

Weekly

Weekly

As needed

 The CAS-Carrera program provides fidelity management and oversight to the Tulsa inschool site. This site has a Fidelity Manager in addition to content specialists for Mental Health, 

FLSE, and Job Club, as well as academic support staff. CAS-Carrera also provides ongoing 

performance management through the Carrera Management Information System (CMIS). 

However, the formalized collection of observation or fidelity data was not included as a part of 

this study design. 

 The Tulsa site experienced moderate staff turnover during this three-year study. The 

Program Director position turned over three times at this site. Additionally, there have been 

some component leader and social worker/mental health specialist turnovers. 

Teacher Perceptions of the CAS-Carrera program 

 Ten Tulsa teachers completed a survey in Spring 2012 and 39 completed a survey in 

Spring 2013. Over 80% of the teachers were female and Caucasian. Over 90% of them claimed 

to have at least some knowledge of the CAS-Carrera program. Their teaching experience varied 

but 41% had six years or less as teachers. 

5

 This first year reflects February 2012 through June 2012. 

 
16

 Overall, teachers believed that the program had made a positive impact on their students. 

Specifically, they reported: 

 Interacting with CAS-Carrera staff on issues “somewhat often.”

 Student gains in sexual awareness, knowledge, general well-being, self-confidence, 

academics and/or behavior. 

 Favorable ratings on the effectiveness of the CAS-Carrera components.

 Improved timeliness of interventions and/or processes at their schools since the CAS-Carrera 

program was created. 

 The CAS-Carrera program had some (42%) or a great deal (13%) of positive impact on their 

students. 

 Improved student knowledge about puberty/development as a result of the CAS-Carrera

program. 

Overall, school staff perceive the program has had a positive impact on their students. 
17

Program Outcomes 

 The analytic sample used to examine outcomes includes 150 9th grade Tulsa program 

youth and the 153 9

th grade comparison youth who all completed surveys at all three data 

collection points (as 7th graders, 8th graders, and 9th graders). We emphasize the 9th grade results 

because these students have had a full three-year exposure to the CAS-Carrera program and 

because these older students are more likely to be engaging in sexual risk behaviors than the 

younger students. The outcome exploration begins with differences in frequencies of outcomes 

among program and comparison students and when positive and significant differences are 

found, moves to a regression analysis to better isolate the role of the program in producing these 

outcomes. Appendix C provides detailed information on the specific statistical tests used in 

Tulsa by grade.

Long Term Program Outcomes 

Pregnancy 

 The CAS-Carrera program seeks 

lower rates of pregnancy among the students 

they enroll than among comparison youth. 

There were no pregnancies reported among 

program youth (Figure 1), significantly fewer 

than among comparison youth. This 

difference occurred among 8th grade youth as 

well. Among comparison youth reporting a 

pregnancy, three were male and two were 

female.

Mid-term Program Outcomes 

Sexual Behavior and Contraceptive Use

 Other confirmatory hypotheses are that 

CAS-Carrera program youth will have higher 

rates of sexual abstinence and will be more 

likely than comparison youth to use 

contraception when having sex. As seen here, 

9

th grade program youth were significantly 

less likely than comparison youth to report 

ever having sexual intercourse (Figure 2). 

This significant difference did not occur until 

youth reached the 9th grade.
18

 Program students who had had sexual intercourse were not significantly more likely than 

comparison students to report using a condom at last intercourse (Figure 3). Very few sexually 

experienced youth in Tulsa reported using a condom and a non-coital method of contraception 

such as Depo-Provera or birth control pills at last intercourse (Figure 4) and these behaviors did 

not differ significantly between program and comparison students. 

Students were asked to indicate how 

frequently they used birth control or 

contraception in the past six months. An 

overall score was created where 0=‘never,’ 

1=’some of the time,’ 2=’about half the time,’ 

3=’most of the time,’ and 4=’every time.’ As 

seen in Figure 5, program students reported 

more frequent use of contraception/birth 

control than comparison students, but these 

differences are again from very small samples 

and are not statistically significant. This 

pattern appeared among younger youth as well. 

 Other confirmatory hypotheses tested were that CAS-Carrera program youth will engage 

in fewer risk behaviors such as fighting, drug or alcohol use, and that they would be less likely to 

be arrested. 

Violence and Delinquency 

 As seen in Figure 6, there were no 

significant differences among program and 

comparison students in physical fighting. 
19

Similarly there were no significant differences in carrying a weapon (Figure 7) nor in arrests 

(Figure 8).

Drugs and Alcohol 

 These data show no significant differences between program and comparison students in 

ever having smoked cigarettes or in having smoked in the past 30 days (Figure 9).

23% 27%

5% 8%

0%

20%

40%

60%

80%

100%

Prog.

(n=150)

Comp.

(n=150)

Prog.

(n=150)

Comp.

(n=146)

Figure 9: Cigarette smoking in Tulsa

% ever used 3 0-day use

use

9th 

Grade

9th 

Grade
20

 Similarly, ever using alcohol or recent alcohol use did not differ between these two 

student groups (Figure 10). 

 Marijuana use did not differ between program and comparison youth. In data not shown 

here, 8

th grade program youth were significantly less likely to report recent use of marijuana than 

comparison students but by the 9th grade, the rates were about equal. 

35% 31%

17% 18%

0%

20%

40%

60%

80%

100%

Prog.

(n=150)

Comp.

(n=153)

Prog.

(n=150)

Comp.

(n=151)

Figure 11: Marijuana use in Tulsa

% ever used 3 0-day use

9th 

Grade

9th 

Grade

53%

59%

13% 16%

0%

20%

40%

60%

80%

100%

Prog.

(n=150)

Comp.

(n=150)

Prog.

(n=150)

Comp.

(n=148)

Figure 10: Alcohol use in Tulsa

% ever used 3 0-day use

9th 

Grade

9th 

Grade
21

Academic Indicators 

Also examined were differences between CAS-Carrera 9th grade program and 

comparison students in college plans, frequency of turning in homework on time, and having to 

attend summer school because of poor academic performance.6 There were no significant 

differences in any of these outcomes between these two groups. 

 

6

 To compare college plans between groups, a 4-point scale was created where 1=’definitely not,’ 2=’probably not 

but maybe,’ 3=’yes, but later – not right after high school,’ and 4=’yes, right after high school.’ Higher scores 

indicate greater intent on going to college. To compare homework completion over time, a 4-point scale was 

created where 1=’hardly ever,’ 2=’sometimes,’ 3=’usually,’ and 4=’always.’ Higher scores indicate more frequent 

on-time homework completion.

2% 2%

0%

20%

40%

60%

80%

100%

Program

(n=150)

 Comparison

(n=153)

Figure 14: Summer school attendance the previous 

Summer in Tulsa

9th Grade

2.00 1.96

0

1

2

3

Program

(n=150)

 Comparison

(n=153)

Figure 13: On-time homework completion in Tulsa

9th Grade

2.47 2.59

0

1

2

3

Program

(n=149)

 Comparison

(n=152)

Figure 12: College intent in Tulsa

9th Grade
22

Short-term Program Outcomes 

 Expected short-term program outcomes were improved knowledge about sexuality and 

reproduction and financial literacy. 

Family Life and Sexuality Education (FLSE) 

 Program youth had significantly higher 

family life and sexuality education knowledge 

scores than comparison youth (Figure 15). 

This difference appeared among 8th grade 

youth as well. Table 4 shows the sub-topic 

FLSE scores. Tulsa program youth scored 

significantly higher than comparison youth in 

all subtopic areas. 

Table 4: FLSE sub-topic scores in Tulsa 

% Correct

Prog. (n=150) Comp. (n=153)

Puberty/Human development 97% 87%***

Parental involvement 91% 81%*

Social and emotional development 84% 74%**

STI/HIV prevention and treatment 80% 61%***

Sexual abstinence 82% 54%***

Sexual and reproductive anatomy 83% 53%***

Conception/Pregnancy/ Contraception 75% 43%***

Sex and gender 82% 40%***

Communication 69% 16%***

Difference is statistically significant at *p<.05 / **p.01 / ***p.001. 

Financial Literacy 

 Young people were asked four questions 

designed to measure financial literacy. When 

combined into an overall measure, financial 

literacy was significantly greater among program 

students than among comparison students (Figure 

16). This advantage for program youth appeared 

in both the 7th and 8th grades as well. The Job 

Club component specifically addresses financial 

literacy. 

74%

57%

0%

20%

40%

60%

80%

100%

Program

(n=150)

 Comparison

(n=153)

Difference is statistically significant at ***<.001

Figure 16: Financial literacy in Tulsa

9th Grade

***

82%

55%

0%

20%

40%

60%

80%

100%

Program

(n=150)

 Comparison

(n=153)

Difference is statistically significant at ***<.001

Figure 15: FLSE knowledge scores in Tulsa

9th Grade

***
23

Program Service Objectives 

 The service objectives for the CAS-Carrera program include students having bank 

accounts and improved receipt of medical care. 

Bank Accounts 

 Program students were significantly 

more likely than comparison students to 

report having a bank account (Figure 17) and 

this difference emerged among 7th graders 

shortly after enrollment. 

Health Care 

 Among 9th graders there was no significant difference between program and comparison 

youth in using the emergency room as their only source of health care (Figure 18), nor was there 

a difference between these groups in having regular checkups or a dental exam within the past 

year. Program students were significantly more likely than comparison students to report having 

an eye exam within the past year. The CAS-Carrera program in Tulsa instituted an eye glasses 

program that likely accounts for this large difference. 

 

 

9% 7%

78% 77%

87% 83% 83%

12%

0%

20%

40%

60%

80%

100%

Prog.

(n=150)

Comp.

(n=153)

Prog.

(n=149)

Comp.

(n=150)

Prog.

(n=150)

Comp.

(n=149)

Prog.

(n=149)

Comp.

(n=148)

Difference is statistically significant at ***<.001

Figure 18: Percent who ...

9th Grade

used hospital E.R. had regular checkup had teeth checked had eyes checked

***

92%

24%

0%

20%

40%

60%

80%

100%

Program

(n=149)

 Comparison

(n=151)

Difference is statistically significant at ***<.001

Figure 17: Percent who have a bank account in Tulsa

9th Grade

***
24

Regression Analysis: Tulsa 

A linear regression analysis was conducted to control for gender, age, and ethnicity and 

determine whether and to what degree being in the CAS-Carrera program remained significantly 

related to the outcomes reported above. The entire list of outcomes was examined but no new 

significant relationships with program participation appeared (Table 5). As seen below, program 

participation had a positive, independent, and significant effect on four interim outcomes: FLSE 

knowledge, receipt of eye exams, having a bank account, and financial literacy. Additionally 

(although not significant), program youth were 9% less likely to have reported ever having sex 

(p<.067) and 3% less likely to have reported ever been pregnant/caused a pregnancy (p<.052).7

Table 5: Relationships of program participation and outcomes among Tulsa 9th grade students 

using a linear regression model

Dependent variables Percentage difference in outcome between program and 

comparison students, net of gender, age, and ethnicity

Long-term Program Outcomes:

Ever been pregnant/caused a pregnancy (1=yes, 0=no): -.029

Mid-term Program Outcomes:

Ever had sexual intercourse (1=yes, 0=no): -.090

Condom use at last intercourse (1=yes, 0=no): -.037

Use of condom and non-coital method at last intercourse (1=yes, 0=no): -.034

Frequency of birth control use in the past 6 months: .280

In physical fight during past 12 months (1=yes, 0=no): .011

Carried a weapon during past 30 days (1=yes, 0=no): -.059

Arrested during past 6 months (1=yes, 0=no): .007

Ever smoked cigarettes (1=yes, 0=no): -.023

Smoked cigarettes in the past 30 days (1=yes, 0=no): -.032

Ever used alcohol (1=yes, 0=no): -.072

Used alcohol in the past 30 days (1=yes, 0=no): -.026

Ever used marijuana (1=yes, 0=no): .056

Used marijuana in the past 30 days (1=yes, 0=no): -.015

College intent: -.159

Homework completion: .024

Short-term Program Outcomes:

FLSE scores: .266***

Financial literacy: .175***

Program Service Objectives:

Having a bank account (1=yes, 0=no): .676***

Using the ER as the only source of health care (1=yes, 0=no): .029

Having had a regular checkup within the past year (1=yes, 0=no): .021

Having a dental exam within the past year (1=yes, 0=no): .042

Having had an eye exam within the past year (1=yes, 0=no): .700***

Difference is statistically significant at ***p<.001. 

Note: Gender, age and ethnicity were included as independent variables in these regressions. 

7

 The difference between program and comparison youth on these two measures was significant when looking at 

simple ANOVAs. However, when controlling for demographic variables in regression analyses, these differences 

were no longer significant. 

 
25

Section II: The CAS-Carrera In-School Model 

in New York City 
26

Methods 

 In NYC sites, all students who enter a CAS-Carrera program grade are deemed CASCarrera students. There are no selection criteria for enrollment. There were 346 program 

participants surveyed by Philliber at these sites in Fall 2011. Also, in Fall 2011, 290 comparison 

students (those with no program intervention) in the grades immediately in front of the program 

students were surveyed. For example, at BPC 93 8th grade program students and 99 comparison 

9

th grade students were surveyed in Fall 2011. In Fall 2012, 87 of those program youth were 

surveyed again. These 87 youth were 9th graders in Fall 2012. These 87 youth were also 

surveyed in Fall 2013. Table 6 shows the number of youth surveyed per group in Fall 2011, 

2012, and 2013. 

Table 6: Number surveyed in Fall 2011, Fall 2012, and Fall 2013 in NYC 

School Program Comparison

Fall 2011 Fall 2012 Fall 2013 Fall 2011 Fall 2012

BPC (NYC) 93 (8th grade) 87 (9th grade; 94%) 87 (10th grade; 94%) 99 (9th grade) 85 (10th grade; 86%)

UAI (NYC) 109 (9th grade) 101 (10th grade; 93%) 91 (11th grade; 83%) 67 (10th grade) 60 (11th grade; 90%)

OCS (NYC) 49 (10th grade) 44 (11th grade; 90%) 41 (12th grade; 84%) 52 (11th grade) 46 (12th grade; 88%)

ASE (NYC) 95 (11th grade) 87 (12th grade; 92%) A 72 (12th grade) na

Total 346 319 (92%) 219 (87%)C 290 191 (88%)B

A There were 12 seniors from Fall 2012 who were retained in grade in Fall 2013. These students have been removed from the analysis.

B This percentage is based on 218 possible comparison students. At ASE there were no comparison students left in Fall 2012. 

C This percentage is based on the BPC, UAI, and OCS samples. 

 The NYC sites serve a much larger number of youth than are included in this report. In 

Fall 2011, there were over 1,400 program youth surveyed across all four sites. However, only 

346 met the requirements of the lagged cohort design (they could be compared to non-program 

youth after one year). These 346 program youth and 290 comparison youth were part of an 

intent-to-treat design, meaning they were rigorously followed for three years. 

 The initial intention was to use a lagged cohort design in NYC, where program youth 

could be compared to non-program youth after one year. However, the first year analysis of the 

outcome data obtained from program and non-program students showed achievement of some of 

the expected interim outcomes such as greater knowledge about sexuality or greater financial 

literacy, but no data suggesting that the program was having the expected effects on rates of 

sexual activity, use of contraception, or pregnancy. It also became apparent that each of these 

schools had other programs, some with similar services to the CAS-Carrera program. Both 

program staff and non-program students were also making comments to the evaluation team that 

led us to suspect that the intended comparison group was receiving program services. 

Interview data from NYC teachers and focus groups with non-program youth in NYC 

 In December 2013 and January 2014, interview data with NYC program staff and school 

staff and focus group data with youth were conducted by Philliber at the four NYC CAS-Carrera

sites. The purposes for this data collection were to determine: how aware the non-program youth 

were of the CAS-Carrera program; the extent to which non-program youth were receiving 

services from program staff; what other school services similar to CAS-
27

Carrera services were being delivered to both the program and non-program youth in the schools; 

and the successes and challenges in implementing the integrated school version of the CASCarrera program. 

 Specifically, in December 2013 and January 2014, Philliber conducted interviews with 36 

CAS-Carrera program staff members (program directors, community organizers, coordinators, 

and social workers) and 22 administrative staff 

(heads of school, principals, teachers, deans, 

social workers, supervisors, and counselors) at 

the four program schools (Table 7). In addition, 

four focus groups were conducted with 73 nonprogram students at BPC and UAI. Forty-six 

youth participated from BPC and 27 from UAI. 

All youth were in the 11th or 12th grades.

 These NYC interview data suggested high satisfaction with the CAS-Carrera program but 

clear contamination of the comparison group of students since the program’s services had spread 

rapidly throughout the schools. School administrative staff commented that they had frequently 

sent non-program youth to CAS-Carrera staff members for services. Additionally, CAS-Carrera 

staff commented that they had frequently served non-program youth when there was a need 

identified by school staff. 

 It thus became clear that using the comparison group data in the NYC schools would 

provide little benefit to the evaluation. In Spring 2014 the study began using NYC Youth Risk 

Behavior Surveillance (YRBS) high school data for comparison. Finding comparison schools in 

the NYC neighborhoods was not logistically possible, nor was funding available for that option. 

Therefore, for each of the four NYC CAS-Carrera program sites, a comparison group of students 

was selected using propensity score matching on borough, grade, gender, race, and ethnicity. A 

total comparison sample of 219 high school youth was created from the 2011 YRBS data and 

from the 2013 YRBS data. These are two separate samples, not longitudinal data, since 

identifiers do not exist. Additionally, YRBS data are gathered every two years (2012 data do not 

exist).

 YRBS does not allow data users to have access to the names of the school included in 

their data sets. However, the CAS-Carrera program schools are generally lower performing than 

New York City schools as a whole—creating a challenge for the program schools to match the 

outcomes in this potentially higher achieving citywide comparison sample. However, it is 

possible to compare the schools in the program study in a specific borough to YRBS data in 

those same boroughs. 

 To illustrate, we have drawn data from the NYC 2011-2012 Department of Education 

(DOE) School Survey. Each year parents, teachers, and students complete a NYC DOE survey 

which covers learning conditions at their schools. Table 8 shows the average satisfaction scores 

for each CAS-Carrera program school and the city average in 2011-2012. As seen below, scores 

for each program school in almost every domain were lower than the NYC average. 

Table 7: Number of NYC interviews

School School staff Program 

staff

ASE 4 5

BPC 5 8

OCS 10 10

UAI 3 13
28

Table 8: Average satisfaction scores of CAS-Carrera schools 

2011-2012 OCS UAI BPC ASE Citywide average

Academic Expectations 6.4 7.1 7.2 6.6 7.6

Communication 6.3 6.4 6.5 6.0 6.9

Engagement 6.3 6.7 7.1 6.1 7.3

Safety & Respect 6.2 7.2 7.4 6.5 7.4

 In addition, we have examined data from the NYC DOE 2011-2012 Progress Report for 

each school. Progress Reports help parents, teachers, principals, and school communities 

understand schools' strengths and weaknesses. Progress reports grade each school with an A, B, 

C, D, or F based on student progress (60%), student performance (25%), and school environment 

(15%). Scores are based on comparing results from one school to a peer group of up to 40 

schools with the most similar student population and to all schools citywide. As seen in Table 9, 

three of the four CAS-Carrera program schools were ranked below the 20th percentile and had an 

overall score well below the NYC average. 

Table 9: Progress report for each CAS-Carrera school 

2011-2012 2011-2012 Overall 

Grade

2011-2012 Overall 

Score

2011-2012 percentile 

ranking

OCS C 41.6 17.4

UAI D 31.0 6.9

BPC D 31.1 7.2

ASE n/a n/a n/a

Citywide average 57.4 50.3

Note: The grade for ASE is for 2010-11 and the overall scores and percentile ranking for this school are for the high school only since the middle 

school was being phased out and the NYC DOE did not compute these data for this year. 

 Given the small number of variables with which to match CAS-Carrera program youth to 

YRBS youth, and the potential differences between program schools and YRBS schools, using 

YRBS data as comparison data provides only a context in which to understand program 

effectiveness.

 However, YRBS data have been used in many contexts. According to the National YRBS 

Q and A website8, “state and local agencies and nongovernmental organizations use the YRBS 

data to set school health and health promotion goals, support modification of school health 

curricula or other programs, support new legislation and policies that promote health, and seek 

funding for new initiatives. For example: 

 Milwaukee Public Schools (MPS) used YRBS data to support adoption of evidencebased curricula in MPS schools, community schools, after-school programs, and 

alternative settings for school-aged youth. 

 The Montana Office of Public Instruction and its partners used YRBS data for program 

planning and improvement.

 In Vermont, YRBS data were used to examine the success of statewide tobacco control 

programs and promote tobacco prevention programs for youth. 

 

8

 Youth Risk Behavior Surveillance – United States, 2005. Q’s and A’s for Participating Sites. 

http://www.emc.cmich.edu/YRBS/Qs%20and%20As.pdf
29

 In Wisconsin, classroom activities designed to teach social norms were developed based 

on YRBS data.”

Procedure used for obtaining the NYC YRBS comparison group 

 Nine demographic variables (common to both program and YRBS youth) were used in 

the matching procedure. The treatment variable was whether students received CAS-Carrera

programing (coded 1) or completed a YRBS (coded 0). The demographic variables used for 

selecting the sample were: 

 group (1=CAS-Carrera program, 0=YRBS comparison) 

 gender (1=male, 2=female) 

 grade (9th, 10th, 11th, 12th) 

 Black (1=yes, 0=not black) 

 Latino (1=yes, 0=not Latino)

 White (1=yes, 0=not White) 

 Asian (1=yes, 0=not Asian) 

 multi-racial (1=yes, 0=not multi-racial)

 other race (1=yes, 0=not other race)

 borough (1=Bronx, 2=Brooklyn, 3=Manhattan) 

 Propensity scoring without replacement was selected 

instead of with replacement (both of which use nearest neighbor 

matching) for two reasons: 

1) There was a large comparison sample of NYC students 

completing the YRBS both in 2011 and 2013 (n=7,653 

and n=5,150, respectively), and 

2) We wanted to eliminate possible outliers generated by 

propensity scoring with replacement. The sample 

generated with replacement created a weighted 

comparison sample, one individual in each sample being 

used 60 or more times thus comprising more than a 

quarter of the comparison sample (Table 10).

 Except for the “borough” variable, complete balance was 

achieved in the 2011 sample. Note that, while the matching did 

not reduce the bias as much for the 2013 sample as for the 2011 

sample, the overall reduction in bias was 25%, and no 

statistically significant differences were reported for the control 

(demographic) variables (Table 11, following page). 

Table 10: Matching with replacement

Weight of 

comparisons 

w/ 

replacement

Frequency 

2011

Frequency 

2013

1 3 7

2 3 3

3 1 1

4 3 3

5 1 0

7 0 2

8 0 1

9 2 0

10 0 1

11 3 0

12 0 1

14 1 1

15 1 0

16 0 1

18 0 2

21 1 1

27 1 0

60 0 1

62 1

219 total 21 total 21 total
30

Table 11: Reduction in bias 

Demographic variables used 

for selecting sample

% reduction in bias 

2011 propensity 

matched (n=219)

% reduction in bias 

2013 propensity 

matched (n=219)

gender 100% (p=1.000) 100% (p=1.000)

grade 100% (p=1.000) 95.3% (p=.851)

black 100% (p=1.000) 94.1% (p=.772)

latino 100% (p=1.000) 100% (p=1.000)

white No whites in sample No whites in sample

asian 100% (p=1.000) 100% (p=1.000)

multiracial 100% (p=1.000) 100% (p=1.000)

other race 100% (p=1.000) 33.5% (p=.432)

borough 91.7% (p=.948) 100% (p=1.000)

Mean Bias .1% (reduced 31%) 1.5% (reduced 25%)

 There is no common support problem, meaning that 100% of the cases were matched 

with a propensity score with a difference <.001 and for nearly all (99%), the difference was zero. 

While matching without replacement is not always recommended, in this instance it is fully 

supported since it eliminated almost all of the bias. 

 Table 12 shows the demographics of the NYC program youth and the YRBS comparison 

youth samples from 2011 and 2013. As seen here, the CAS-Carrera program youth and the 

YRBS comparison youth samples were extremely well matched on the available demographic 

variables.

Table 12: Comparing demographics of NYC CAS-Carrera program youth vs. 

YRBS comparison youth in 2011 and 2013 

Demographics 2011 2013

Program 

Youth in 

NYC (n=219)

YRBS 

Youth 

(n=219)

Program 

Youth in 

NYC (n=219)

YRBS 

Youth 

(n=219)

Gender:

Male

Female

32%

68%

32%

68%

27%

73%

27%

73%

Ethnicity:

African-American

Latino

Caucasian

Asian

Multi-ethnic

Other

62%

19%

0%

<1%

11%

8%

62%

19%

0%

<1%

11%

8%

59%

27%

0%

<1%

11%

3%

58%

27%

0%

<1%

11%

4%

Grade:

9

th

10th

11th

12th

41%

19%

40%

0%

41%

19%

40%

0%

1%

41%

39%

19%

1%

41%

40%

18%

Borough:

Bronx

Brooklyn

Manhattan

40%

42%

18%

40%

41%

19%

40%

41%

19%

40%

41%

19%
31

 However, the limited number of variables available for matching prevents knowing that 

the program youth are well-matched to the YRBS comparison youth, other than on these few 

variables. Thus, the YRBS data included in this section of the report are intended to serve as 

contextual data in gauging outcomes among program youth. 

Student survey data 

Philliber collected baseline student survey data from NYC youth in Fall 2011. The first 

follow-up data collection occurred in Fall 2012, the second in Fall 2013, and the final data 

collection occurred in Fall 2014. All data were collected using a single self-report survey (see 

Appendix A). The survey included questions about demographics, family composition, socioeconomic status, health care, educational aspirations and other school-related items, 

employment, financial literacy, recreational activities, drug use, violence and delinquency, 

involvement in the juvenile justice system, communication, sexuality knowledge, sexual activity, 

contraceptive use, and pregnancy. The family life and sexuality education (FLSE) items were 

developed by Dr. Carrera and senior FLSE staff to track the dimensions of the FLSE curriculum 

offered in the program. In total there are 30 items (10 true/false items, 10 multiple choice items, 

and 10 fill-in-the-blank items) measuring knowledge about communication, parental 

involvement, puberty/human development, sexual and reproductive anatomy, STI/HIV 

prevention and treatment, social and emotional development, conception/pregnancy/ 

contraception, sex and gender, and sexual abstinence. 

All data were computer-entered at Philliber and subsequently analyzed. Demographic 

data were presented as frequencies with computed means as appropriate. Outcome data were 

analyzed using ANOVAs. Regression analyses were performed to determine if program 

participation was related to outcomes. To predict outcome variables, the independent variables 

were: program assignment, gender, age, ethnicity, grade, and borough (school site). 

Teacher survey data 

 Philliber surveyed teachers each Spring (2011 through 2013) to assess their perceptions 

of the program in their schools (see Appendix B). In Spring 2011, an online survey was created 

using Survey Monkey and emailed to teachers to solicit feedback at the four NYC sites. In 

Spring 2012 and Spring 2013, this survey was repeated in these same schools. In Spring 2011, 

155 out of 243 teachers responded, in Spring 2012, 117 of 170 teachers responded, and in Spring 

2013, 127 of 191 teachers responded. Questions included measures of their involvement and 

knowledge of the program, interaction with program staff, perceptions of student improvement, 

perceptions about the effectiveness of the components, timeliness of the intervention, and 

perceptions of program impacts. 

Student attendance data (dosage) 

 Daily attendance data were unavailable for this study. However, dosage can generally be 

inferred based on length of stay in the program given the in-school program structure. In the 

NYC sites, 85% of the program youth included in this study had been in the program prior to the 

Fall 2011 ‘baseline’ survey. 
32

Validity 

 The level of evidence in this evaluation is preliminary at best. In the NYC sites, all 

students entering the schools are deemed CAS-Carrera program youth. Thus, there is no 

selection bias. Additionally, exact matching was used with NYC YRBS youth to create 

comparison data because the rapid spread of this program in the NYC schools precluded using a 

lagged cohort design as originally intended. Such a comparison does provide some context 

within which to understand outcomes among the program youth in NYC. 

Logistics 

 Philliber staff collected all of the NYC interview data from CAS-Carrera program staff 

and school administrative staff. Philliber also collected all of the teacher surveys via Survey 

Monkey. 

 Over 98% of the student survey data were collected by Philliber staff during yearly 

scheduled one-week data collections at each site. Philliber staff read the survey items and 

response choices to all students. Youth were reassured of confidentiality and asked to be honest 

in their answers. If youth were not present during the one-week scheduled data collections, 

CAS-Carrera program staff collected survey data at a later time. These program staff had youth 

complete the surveys in a quiet area with no distractions. These youth were instructed to seal 

their completed surveys in an envelope and deliver them to the Program Coordinator who then 

mailed the surveys to Philliber. CAS-Carrera program staff never viewed student data. 

Inactivity & attrition 

 Since this was an intent-to-treat design, Philliber attempted to resurvey every youth who 

was surveyed at baseline in subsequent data collections. Philliber mailed surveys to inactive 

youth (those who had left the study schools) and also made follow-up phone calls to remind 

inactive youth to complete their surveys. All middle school youth received a $15 cash stipend 

and high school youth received a $25 cash stipend for a completed survey. At final follow-up, 

21% of the NYC program youth included in this report were inactive (they had left the study 

schools at some point during the three-year period). These surveyed inactive NYC youth were 

significantly older and were significantly more likely to have reported living in foster care.

 The attrition rate among NYC program youth was 13%. Youth leave the schools due to 

transferring to a different school or moving to a new neighborhood and either cannot be found or 

decline to be surveyed. Students also transfer to new schools to receive specialized academic 

concentrations or sports. These non-surveyed youth who were lost to follow-up, did not differ 

significantly from those who were surveyed. 

Data storage 

 All data were computer-entered into SPSS data files with assigned Philliber IDs. Student 

names were kept separately and were not linked to survey data. All paper 
33

data from this study were kept in a locked storage facility. There were no problems with 

agreements for data access, storage, use, or reporting. There were no changes to Memoranda of 

Understanding (MOU’s).

 Data from each annual data collection were stored in separate SPSS files and merged to 

create one “master file”. Each year’s data had different variable names to help distinguish one 

year from the next (i.e. ever had sex1, ever had sex2, ever had sex3, etc). 

 Upon request, a CDC representative sent Philliber the NYC YRBS data for 2011 and 

2013, along with a code book describing all variables contained in each data set. Philliber then 

created a matched file to use as comparison data.

Data cleaning and missing data

 All data went through an extensive cleaning process. All surveys were checked for 

accuracy to ensure data entry was correct. During data analysis, inconsistent data were removed 

depending on the variables being measured. For example, if a participant checked “yes” for 

having sex at baseline but then checked “no” on a subsequent survey, this participant’s data were 

removed from that calculation. If a participant checked “yes” for having sex in the past three 

months but left the ever had sex question missing, the missing data were filled in. This logic 

applied throughout the analysis. Philliber imputed all variable values possible. For example, if a 

student was Latino on the first two surveys, and left ethnicity blank on the last survey, she is 

probably Latino. If the data on ever had sex and use of contraception were inconsistent, (i.e., 

denies ever had sex but uses contraception), older surveys were consulted and the use of weightof-evidence approach was used. Throughout this study, missing data were negligible (less than 

5% on any given indicator). 

IRB 

 This study was fully approved by an IRB. There were no changes made to the IRB 

during the course of this study. Currently the study is closed. 

Evaluator role/involvement and budget 

 There were no changes regarding the role of the evaluator. The senior staff member 

overseeing this evaluation at Philliber remained consistent for the duration of this study. 

Levels of significance 

 Throughout this report, asterisks (*) indicate whether or not a difference or change is 

statistically significant. The “levels of significance” indicate the likelihood that an observed 

difference or change may be due to chance. Thus, when differences are marked with one asterisk 

(*), it is an indication that the difference would occur by chance alone less than 5% of the time. 

Two asterisks (**) denote differences that could have occurred by chance alone less than 1% of 

the time, and three asterisks (***) mean that only 1 time in 1,000 would a difference this large 
34

occur by chance alone. In other words, these differences are likely to be “real” rather than being 

statistical artifacts produced by sampling errors. 
35

Implementation 

 The four NYC sites began program implementation in Fall 2008, but the first survey data 

were collected in Spring 2009. While the program began in four of the schools six years ago, the 

grades covered in each year have varied. In 2008, BPC began programming with only 5th

graders and UAI began serving 6

th graders. At OCS, on the other hand, the program began with 

6

th and 7th graders in the first year and at ASE, 6th, 7th, and 8th graders received the program in its 

first year. In each school a new grade level has been added each year as those served move up 

and new students enter the school. Additionally, as new students enter these schools in any 

grade receiving the program they are enrolled in the program. This means that grade level may 

not necessarily indicate program exposure. 

 Thus, 85% of the program participants in this NYC sample received the CAS-Carrera 

program prior to Fall 2011. This means that 85% of these youth had been exposed to at least one 

year of program intervention at baseline (dosage), so that the Fall 2011 survey is not a “clean” 

baseline. 

 Another evaluation option would have been to choose a number of schools and to 

randomly assign the program to half of those schools. This more robust option would have 

required much larger amounts of funding than were available in Spring, 2011. 

 It is important to note that the NYC sites serve a much larger number of program youth 

than are included in this report. For example, in 2012, the NYC sites served more than 1,700 

program youth in grades 5 through 12. This report includes only those who were surveyed in Fall 

2011 and subsequently followed for three years. Any program participants who were not 

surveyed in Fall 2011 (those joining after this time period) are not included. 

Service Delivery: Was the program delivered as intended? Was each program component 

offered each program year? 

 In a typical after-school CAS-Carrera program, the academic component is offered daily 

(Monday through Friday), and Job Club, FLSE, and Mental Health are offered weekly. The 

dosage requirement for self-expression and individual lifetime sports is to have at least two 

exposures per year which are offered as part of weekly scheduled programming, and/or during 

school breaks and summer. Medical services/referrals are offered as needed. In the in-school 

model, the academic component is offered daily, and Job Club, Mental Health, and FLSE are 

offered weekly (Table 13). Among the NYC sites the sports component was delivered during the 

summer months and self-expression was delivered in either the summer or in the spring. At 

OCS, the self-expression component did not exist as a separate component but instead was 

combined with other component activities. At ASE, the program was winding down and largely 

did not offer programming in 2012-13 or 2013-14 (this school was phasing out the middle school 

grades). 

 The CAS-Carrera program provides fidelity management and oversight to the NYC inschool sites. All sites have a Fidelity Manager in addition to content specialists for Mental 

Health, FLSE, and Job Club, as well as academic support staff. CAS-Carrera 
36

also provides ongoing performance management through the Carrera Management Information 

System (CMIS). However, the formalized collection of observation or fidelity data was not 

included as a part of this study design. 

Table 13: Component delivery 

Site and components 2011-2012 2012-2013 2013-2014 2014-2015

BPC

Academic

Job Club

FLSE

Mental Health

Self-expression

Lifetime individual sports

Medical services/referrals

Daily

Weekly

Weekly

Weekly

Summer only

Summer only

As needed

Daily

Weekly

Weekly

Weekly

Summer only

Summer only

As needed

Daily

Weekly

Weekly

Weekly

Summer only

Summer only

As needed

Daily

Weekly

Weekly

Weekly

Summer only

Summer only

As needed

UAI

Academic

Job Club

FLSE

Mental Health

Self-expression

Lifetime individual sports

Medical services/referrals

Daily

Weekly

Weekly

Weekly

Spring only

Summer only

As needed

Daily

Weekly

Weekly

Weekly

Spring only

Summer only

As needed

Daily

Weekly

Weekly

Weekly

Spring only

Summer only

As needed

Daily

Weekly

Weekly

Weekly

Spring only

Summer only

As needed

OCS

Academic

Job Club

FLSE

Mental Health

Self-expression

Lifetime individual sports

Medical services/referrals

Daily

Weekly

Weekly

Weekly

Combined w/ other comp.

Summer only

As needed

Daily

Weekly

Weekly

Weekly

Combined w/ other comp.

Summer only

As needed

Daily

Weekly

Weekly

Weekly

Combined w/ other comp.

Summer only

As needed

Daily

Weekly

Weekly

Weekly

Combined w/ other comp.

Summer only

As needed

ASE

Academic

Job Club

FLSE

Mental Health

Self-expression

Lifetime individual sports

Medical services/referrals

Daily

Weekly

Weekly

Weekly

Summer only

Summer only

As needed

Not delivered

Not delivered

Weekly

As needed

Not delivered

Not delivered

As needed

Not delivered

Not delivered

Weekly

As needed

Not delivered

Not delivered

As needed

Program

closed

 In order to integrate CAS-Carrera components into a school, school leadership needs to 

be willing to set aside classroom time for programming. School leadership may consider various 

scheduling options to integrate programming including the use of advisory periods, electives, 

physical education/art class, health class, and study halls. Academic supports can also be 

scheduled as push-ins/pull-outs and other small group formats. 

 Interview data suggest that the NYC sites experienced moderate staff turnover during this 

three-year study. The Program Director position turned over once at UAI. At the other three 

NYC sites, the Program Director had been in that position for at least five years. Additionally, at 

UAI the community organizer position turned over twice. At the other three sites, the 

Community Organizer had been in that position for at least three years. There have been 

numerous component leader and social worker/mental health specialist turnovers. Staffing 

changes can affect the continuity of program delivery and may have a negative effect on 

outcomes. 

 Program staff interacted with school administrative staff often. In fact, they are often 

included in school administration meetings. Staff/trainer data were not collected. 
37

Context: What were the barriers to and facilitators of program delivery? What positive and 

negative roles did school administrators play? How did the school setting affect program 

delivery? How did school staff see the program and its contribution to their work? 

Highlights from NYC interviews include: 

 Very high effectiveness ratings of the CAS-Carrera program from both program and 

school staff. 

 Consistent reports of school wide changes as a result of the program. 

 Particular praise for the eye glass program for students. 

 Consistent school staff reports of sending non-program students to CAS-Carrera staff for 

assistance/services. 

 Widespread reporting by program staff of having provided program services to nonprogram students.

 The presence of other programs in these schools, offering at least some of the CASCarrera services to both program and non-program students. 

 In one school, adoption of the CAS-Carrera family life and sex education curriculum for 

all students (including the intended comparison youth) in Fall 2013.

 High awareness among non-program students of the program and its services, knowledge 

about how to obtain those services, and frequent use of them. 

 Some irregularity in offering the program’s seven components compared to the afterschool version of the program.

 Uneven perceptions by CAS-Carrera program students that they are part of a “special” 

group receiving program services. 

 Program staff beliefs that while the in-school version of the program has some 

challenges, including bending the program to the school’s goals, need for more staff, and 

transfers of students out of the school, the benefits of being in the school outweigh these 

challenges since it allows greater contact with the students and better linkages with 

students’ teachers.

 Nearly all NYC administrators interviewed reported having a lot of contact with CASCarrera staff. Challenges reported by school staff in working with the CAS-Carrera program 

include: program staff lacking classroom management training, student pull-outs, trying to meet 

all of the needs of the students quickly (volume), scheduling/logistics, staff turnover, and 

building new relationships when teachers or CAS-Carrera program staff leave. In fact, about 

80% of the BPC middle school teaching staff was new in the 2013 school year. This was 

problematic for the CAS-Carrera program staff because they needed to build all new 

relationships with these new BPC staff members. Program staff are largely seen as the 

“veterans” of the schools.

 NYC program staff mentioned challenges around administrative polices; needing to 

adjust their “way of thinking” to accommodate the school and the Department of Education 

(DOE). One CAS-Carrera program staff member stated: 

We are visitors here. So we have to sway with what the administration says. That 

bothers me tremendously because we are here to help and they should take 
38

advantage of that. We’re here to give them a hand and sometimes they slap it away. The 

administration is so focused on ELA, math, and state test scores that they neglect the 

other parts that are so essential for the students. 

 Still, administrators and CAS-Carrera program staff feel the program makes big positive 

changes for students. Specifically, they mentioned the impact of the social workers, increased 

parental involvement, eye glasses, and dental/medical care. Administrators also say they have 

noticed school-wide changes as a result of the program such as increased teacher support from 

CAS-Carrera staff members, an increase in college interest among youth, greater communication 

between students and program staff, and a decrease in disciplinary actions and suspensions. One 

administrator commented: 

The students that we educate here are some of the city’s neediest students. These are 

students, who in other schools may not have been successful at staying in school. And the 

Carrera program provides a whole range of services that we as a public charter school 

can’t provide. This year they’re integrally part of the college preparation process. 

They’ve taken the lead with our guidance department on making sure that all of our 

students are getting the information that they need and have the applications filled out 

that they need to fill out in order to go to the colleges that they’d like to go to. A big part 

of what Carrera does is they take away impediments to learning. Every year they provide 

glasses to the students. And it may seem like a small thing, but for family’s struggling to 

survive, eyeglasses and constantly taking their kids to the doctor to get their prescriptions 

updated becomes a costly thing and may not happen every year. A very big part of what 

they do is removing these types of roadblocks. 

 As noted earlier, these data suggest high satisfaction with the NYC program but clear 

contamination of the comparison group of students since the program’s services had spread 

rapidly throughout the schools. When asked if comparison youth were sent to CAS-Carrera staff 

for services, one administrator said: 

Oh, I did it all the time. We had kids who were non-program kids (comparison kids) who 

were dealing with sexuality issues or they were sexuality active or whatever and we had a 

group of experts on staff (CAS-Carrera staff) where you could always bring the students 

to. We availed ourselves of that often.

Teacher Perceptions of the CAS-Carrera program 

 In Spring 2011, 155 surveys were received from NYC teachers, in 2012, 117 surveys 

were received, and in 2013, 127 surveys were received. 

Who are the 2013 participating teachers in NYC? 

 72% were female, 48% were Caucasian, 20% were African-American, and 15% were Latino. 

These teachers taught in the 5th through 12th grades. 

 54% had been teaching for six years or less and 61% had been at their current schools for 

three years or less; one-quarter were in their first year. 
39

 92% had at least some knowledge of the CAS-Carrera program; 86% were involved with the 

program. 

What did the 2013 teachers have to say about the CAS-Carrera program? 

 Teachers reported interacting with CAS-Carrera staff on issues somewhat often. 

 Teachers felt students had made some improvement regarding sexual awareness, sexual 

knowledge, general well-being, self-confidence, academics and/or behavior. 

 Teachers rated the effectiveness of the CAS-Carrera components favorably. 

 Teachers rated the timeliness of interventions and/or processes at their schools to be a bit 

better than before the CAS-Carrera program existed in their schools. 

 79% felt the CAS-Carrera program had some (38%) or a great deal (41%) of positive impact 

on their students. 

 55% had observed improved student knowledge about puberty/development as a result of the 

CAS-Carrera program. 

What about changes over time among the teachers who had completed more than one survey? 

 Perceived positive impact of the CAS-Carrera program remained high. 

 Over half maintained they had observed improved student knowledge about puberty/ 

development as a result of the CAS-Carrera program. 

 Interaction with CAS-Carrera staff had increased slightly. 

 Perceptions of student improvement had increased slightly. 

 Effectiveness ratings of the CAS-Carrera components remained favorable. 

 Teachers rated the timeliness of interventions and/or processes at their schools less favorably 

in 2013 compared to prior years. 

 These interview and survey data also suggest that the in-school model wasn’t delivered 

with as much regularity as a typical after-school CAS-Carrera program. The in-school programs 

must adapt to the requirements and schedules of the schools and in particular to the demands of 

school administrations. However, program staff believe that while the in-school version of the 

program has some challenges, the benefits of being in the school outweigh these challenges since 

it allows for greater contact with the students and better linkages with students’ teachers. School 

staff perceive the program has had a positive impact on their students. 
40

Program Outcomes 

 The analysis of NYC student survey data began with a descriptive, longitudinal 

examination of outcomes among those students who completed a survey at all three time periods 

(removing ASE from the analysis completely) to determine if interval in the program affected 

outcomes. Results showed three outcomes significantly improved over time: physical fighting 

during the past 12 months decreased, family life and sexuality education knowledge increased, 

and financial literacy increased (Table 14).

Table 14: Significant improvements among the 219 NYC youth who were 

surveyed in 2011, 2012, and 2013 

2011 2012 2013

Mid-term Program Outcomes:

In physical fight during past 12 months (% yes)***: 25% 18% 12%

Short-term Program Outcomes:

FLSE scores (% correct)***: 71% 73% 81%

Financial literacy (% correct)***: 63% 67% 74%

Difference is statistically significant when ***p<.001. 

The analysis also compared selected outcomes among program students and the matched 

YRBS samples from 2011 and 2013. Totals for NYC in Fall 2013 do not include the ASE site 

since they had aged out and totals for 2011 in NYC do not include BPC on items measured by 

the YRBS since BPC youth were 8th graders at the time. Results are shown for only those 

measures which were common to both program and comparison surveys. Additionally, 21% of 

the program youth were inactive at the time of the Fall 2013 survey (not including ASE where 

nearly all were inactive in 2013 due to high school graduation). These surveyed students who 

had transferred schools or moved were significantly older (mean = 14.4 yrs. vs. 14.8 yrs.) and 

were significantly more likely to report being in foster care (0% vs. 4%). The attrition rate 

among program youth was 13%, meaning they were not surveyed at follow-up. Appendix C 

provides detailed information regarding the specific statistical tests used in NYC. 

Long Term Program Outcomes 

Pregnancy 

 Pregnancy rates were significantly lower 

among program youth than YRBS comparison youth 

in 2011 but this was not the case in 2013 (Figure 19

and Table 15). 

3%

10% 5% 5%

0%

20%

40%

60%

80%

100%

Prog.

(n=219)

YRBS

(n=202)

Prog.

(n=219)

YRBS

(n=187)

Difference is statistically significant at **<.01

Figure 19: Pregnancy in NYC

Fall 

2011

Fall

2013

**
41

Table 15: Been pregnant/caused a pregnancy in NYC by gender 

Fall 2011 Fall 2013 

(n) Prog. Youth % (n) Prog. Youth %

Total NYC males

Total NYC females

Total NYC (all program youth)

71

148

219

1%

5%

3%

60

159

219

3%

6%

5%

Mid-term Program Outcomes 

Sexual Behavior and Contraceptive Use

 In 2013, program youth in NYC were slightly less likely than YRBS comparison youth to 

report ever having sexual intercourse (Figure 20). Additionally, program youth who were 

sexually active were significantly more likely than YRBS comparison youth to report condom 

use at last intercourse in both years (Figure 21).

 

82%

46%

80%

62%

0%

20%

40%

60%

80%

100%

Prog.

(n=83)

YRBS

(n=39)

Prog.

(n=83)

YRBS

(n=42)

Difference is statistically significant at *<.05, ***<.001

Figure 21: Condom use at last intercourse 

in NYC

Fall 

2011

Fall

2013

***

*

38% 36% 38%

45%

0%

20%

40%

60%

80%

100%

Prog.

(n=219)

YRBS

(n=175)

Prog.

(n=219)

YRBS

(n=182)

Figure 20: Percent who have had sexual 

intercourse in NYC

Fall 

2011

Fall

2013
42

Violence and Delinquency 

 In 2011 and 2013, NYC program youth were significantly less likely than YRBS 

comparison youth to report being in a physical fight during the past year (Figure 22). 

Additionally, in 2011 and 2013, NYC program youth were less likely than YRBS comparison 

youth to report carrying a weapon during the past 30 days, although not significantly so (Figure 

23).

Drugs and Alcohol 

 In 2011, program students had significantly lower rates of cigarette use (ever and 30-day 

use) than YRBS comparison youth (Figure 24). In 2013, this difference exists only on the “ever 

used” measure.

6%

26%

6%

21%

1%

7% 2% 6%

0%

20%

40%

60%

80%

100%

Prog.

(n=216)

YRBS

(n=214)

Prog.

(n=218)

YRBS

(n=213)

Prog.

(n=216)

YRBS

(n=214)

Prog.

(n=216)

YRBS

(n=208)

Difference is statistically significant at **<.01, ***<.001

Figure 24: Cigarette smoking in NYC

Fall 

2011

% ever used 3 0-day use

Fall

2013

Fall 

2011

Fall

2013

*** ***

**

6% 9% 5% 9%

0%

20%

40%

60%

80%

100%

Prog.

(n=219)

YRBS

(n=213)

Prog.

(n=218)

YRBS

(n=218)

Figure 23: Percent who carried a weapon 

during the past 30 days in NYC

Fall 

2011

Fall

2013

23%

35%

12%

28%

0%

20%

40%

60%

80%

100%

Prog.

(n=218)

YRBS

(n=218)

Prog.

(n=217)

YRBS

(n=219)

Difference is statistically significant at **<.01,***<.001

Figure 22: Percent in a physical fight 

during the past year in NYC

Fall 

2011

Fall

2013

**

***
43

 In 2011, program students had significantly lower rates of alcohol use during the past 30 

days than YRBS comparison youth (Figure 25). In 2013, the 30-day alcohol use rates were 

lower among program youth but not significantly so. 

 Program students had significantly lower rates of marijuana use than YRBS youth during 

the past 30 days prior to survey in 2011 and in 2013 (Figure 26).

8%

24%

12%

20%

0%

20%

40%

60%

80%

100%

Prog.

(n=217)

YRBS

(n=208)

Prog.

(n=218)

YRBS

(n=208)

Difference is statistically significant at *<.05, ***<.001

Figure 26: 30-day Marijuana use in NYC

Fall 

2011

Fall

2013

*** *

18%

32%

23%

30%

0%

20%

40%

60%

80%

100%

Prog.

(n=217)

YRBS

(n=197)

Prog.

(n=217)

YRBS

(n=204)

Difference is statistically significant at **<.01

Figure 25: 30-day Alcohol use in NYC

Fall 

2011

Fall

2013

**
44

Regression Analysis: NYC 

A linear regression analysis was conducted to determine whether and to what degree 

being in the CAS-Carrera program in NYC affected each of the outcomes among the 2011 and 

2013 samples. Whether the student was in the CAS-Carrera or YRBS comparison group, 

gender, grade, borough, and ethnicity were included in each equation (demographics common to 

both program and YRBS samples). Specifically, dependent variables consisted of the outcome 

measures and the independent variables were constructed as follows: 

 Participation (1=program group, 0=YRBS comparison group); 

 Gender: female (1=yes, 0=else);

 Grade (9th, 10th, 11th, 12th);

 Borough: (Bronx=1, 0=else), (Brooklyn=1, 0=else); 

 Ethnicity: African-American (1=yes, 0=else), Latino (1=yes, 0=else).

 Table 16 summarizes the effects that being in the program had on the outcome variables 

(positive or negative, the strength, and whether the effect was significant). Results showed 

among the 2011 sample that program participation had a positive, independent, and significant 

impact on seven of the nine outcomes common to both program and YRBS youth. Of particular 

interest are condom use at last intercourse and pregnancy. A CAS-Carrera program participant is 

33% more likely to report using a condom at last intercourse than a YRBS comparison student 

and a CAS-Carrera participant is nearly 7% less likely to report having been pregnant/caused a 

pregnancy. Other significant differences include cigarette smoking (ever and 30-day use), 

alcohol use in the past 30 days, marijuana use in the past 30 days, and physical fighting during 

the past 12 months. Each of these outcomes favored those in the program. 

 Results showed among the 2013 sample that program participation had a positive, 

independent, and significant impact on four of the nine outcomes. As seen below, a CASCarrera participant is 18% more likely than a YRBS comparison student to report using a 

condom at last intercourse. Other notable outcomes include ever smoking cigarettes, marijuana 

use in the past 30 days, and physical fighting during the past 12 months. 

Table 16: Impact of program participation on outcomes among 2011 & 2013 NYC sample

Dependent variables

Percentage difference in outcome between program 

and comparison students, net of gender, grade, 

borough, and ethnicity

2011 Sample 2013 Sample

Long-term Program Outcomes:

Ever been pregnant/caused a pregnancy (1=yes, 0=no): -.067** .002

Mid-term Program Outcomes:

Ever had sexual intercourse (1=yes, 0=no): .010 -.069

Condom use at last intercourse (1=yes, 0=no): .334*** .182*

In physical fight during past 12 months (1=yes, 0=no): -.125** -.159***

Carried a weapon during past 30 days (1=yes, 0=no): -.031 -.046

Ever smoked cigarettes (1=yes, 0=no): -.203*** -.146***

Smoked cigarettes in the past 30 days (1=yes, 0=no): -.060** -.036

Used alcohol in the past 30 days (1=yes, 0=no): -.135** -.075
45

Used marijuana in the past 30 days (1=yes, 0=no): -.157*** -.088*

Difference is statistically significant at *p<.05 / **p<.01 / ***p<.001. 

Conclusion 

 In Tulsa, 150 program youth and 153 comparison youth were surveyed over three 

program years. Except on age where there was a two-month average difference, these groups 

were well matched. Relative to comparison youth, Tulsa program youth were more likely to 

report having bank accounts, eye exams, and knowledge regarding financial literacy and family 

life and sexuality. These positive findings are centered on services directly delivered to program 

youth. Perhaps the younger youth in Tulsa who have had far less program exposure than the 

NYC youth, may experience additional outcomes as they progress in the program. A promising 

finding is that 9th grade Tulsa program youth were 9% less likely to report having had sex 

(p<.067) and 3% less likely to report having ever been pregnant/caused a pregnancy (p<.052), 

controlling for the effects of demographic variables. 

 In NYC, a descriptive longitudinal analysis found that program students significantly 

decreased fighting over three years of program exposure and significantly increased their 

sexuality knowledge and their financial literacy. The analysis also compared 219 program high 

school youth to 219 YRBS comparison youth. These groups were well-matched on the available 

demographic variables. These program youth performed better than YRBS comparison youth on 

both long-term and mid-term outcomes related to pregnancy, contraception, drug and alcohol 

use, violence and delinquency. 

It is important to note that in this ‘intent-to-treat’ model, evaluators made an effort each 

year to survey every student in the CAS-Carrera program group who completed a ‘baseline’

survey. At last follow-up, 21% of the NYC program participants and 13% of the Tulsa program 

youth no longer attended the study schools (they had moved and/or transferred to other 

neighboring schools). These youth were surveyed and included in this report. Thus, this may 

have had an impact on results. 

 There are many factors in an in-school setting that can affect these findings. While the 

original after-school versions of this program were located within community agencies, those 

agencies did not govern program offerings, hours, or strategies in the ways that schools do. 

After-school programs do not have bells signaling strict time limits or testing days when program 

cannot be offered at all. Children in the integrated school model come to be in the program by 

virtue of their grade level or administrator choice and do not choose membership, as they did in 

the after-school settings. Additionally, school choice can enable students to move between 

schools, thereby affecting program dosage. 

 Additionally, there is a different student-to-facilitator ratio in the in-school model than in 

the typical after-school CAS-Carrera program. A typical after-school program works with no 

more than 60 youth at a time (based on current CAS-Carrera after-school sites). The in-school 

programs are delivering the program to hundreds of youth per school and the number of CASCarrera staff has not increased proportionately. Thus, the typical one-on-one relationships that 

exist in an after-school program are more challenging in the in-school model. However, CASCarrera staff report that an overall staffing plan is developed in conjunction 
46

with school leadership and is guided by multiple factors, including the number of incoming 

students and related classroom sizes. While the kind of staff (component/administrative) is

designated by the program model, the number of staff and specific school based considerations 

are unique to each school. Economies of scale are achieved with larger grades, and staffing is 

scaled accordingly. As the number of program participants increases each year in a typical 

scenario, it becomes mandatory to add new staff since the service needs are expanded. Efforts 

are made to maximize schedules for component leaders/teachers and counselors. 

 In terms of program staff, CAS-Carrera has job descriptions which outline the roles of 

staff within the context of school-based delivery. The core content expertise that is required of 

program staff, whether in-school or in the after-school environment, does not change. However 

there is an expectation that staff has experience with schools and depending on roles, has 

demonstrated capacity on key school considerations like classroom management. Some schools 

may require specific licenses or credentials to lead classroom work. If there are specific 

requirements as determined by the school, these designations would become a part of CASCarrera staffing requirements. Additionally, if there are specific needs articulated by the school 

relating to the student population served, these considerations are included in staffing decisions. 

For example, if a school serves a large number of students with IEPs, the CAS-Carrera education 

component will be staffed with professionals who have specific expertise in this arena. 

 The advantages of the in-school model include the ability to reach more young people 

and thus lower program costs, and the ability to reach young people who might not participate in 

an after-school program but who nevertheless need these services. Qualitative research suggests 

that the in-school model is perceived to have a positive effect on the climate of these schools and 

both teachers and administrators praise its success in connecting young people quickly and 

efficiently to needed services. 

 Comparing these findings to those of the original RCT is not recommended. The original 

RCT study was based on the original after-school design that included 484 youth after three 

years. This current study is an adaptation of the original program design and its evaluation is 

preliminary. The current evaluation involves fewer youth in a non-RCT design. In order to 

compare findings, a thorough RCT design would need to be employed for the in-school model 

where entire schools are randomly assigned to either the treatment or control condition, thus 

barring any possibility of contamination. 

 Thus, the methodological challenges of these evaluations lead us to label these findings 

as encouraging. Further study using a school level randomized control trial (RCT) will be needed 

to produce more compelling data on the outcomes of the in-school CAS-Carrera program. 

 While all students can benefit from CAS-Carrera’s holistic approach, the program seeks 

to serve those young people living in poverty and experiencing disparities. Below are 

characteristics of youth who are typically served by CAS-Carrera: 

 live in poor/near poor family circumstances (standard definition),

 come from single parent households, 

 receive modest, little, or no after-school home supervision, 
47

 have had very little cultural experiences or advantages, 

 have not received comprehensive regular medical, dental, and vision services, and 

 frequently reside in environments with high teen pregnancy rates, and where legal 

justice and anti-social behaviors are common (drug use, drug sales, violence, gangs, 

use of weapons). 

CAS-Carrera has a codified set of considerations that are used in exploring the feasibility 

of a school partnership. Some of these assessment criteria include: 

 philosophical alignment/commitment to programming; 

 school location; 

 leadership/management considerations; 

 school operations/infrastructure; 

 dedicated space for the CAS-Carrera staff; 

 performance management; 

 financial viability; 

 school performance (local/state standards); 

 reputation of the school in the community; 

 an understanding that CAS-Carrera will be a part of school leadership teams; 

 capacity to deliver all aspects/components of CAS-Carrera; and 

 an understanding that long term sustainability requires school resources. 

 If schools/communities would like to implement a teen pregnancy prevention program 

such as the CAS-Carrera program but are unsure about which program to choose, we recommend 

consulting the Office of Adolescent Health evidence-based list of 37 programs deemed effective 

in reducing teen pregnancy.9 This list will provide details about each program with respect to 

length, intensity, and previous research. 

 

9 http://tppevidencereview.aspe.hhs.gov/EvidencePrograms.aspx 
48

Appendix A: Youth Survey 

Today’s date: _____/_____/_____ ID: _______________ 

 

month day year 

1. Birth date: _____/_____/_____ 

month day year 

2. Gender:

male

female

3. Ethnicity: (please check all that apply)

African American/Black

Caribbean

Latino/Latina

Caucasian

Asian

other (specify: ___________________)

4. What grade are you in?

5

th

6

th

7

th

8

th

9

th

10th

11th

12th

5. Please check all the people you live with now. 

 mother your child(ren) (how many? _____) 

 father brother(s) (how many? _____) 

 grandparent(s) sister(s) (how many? _____) 

 step parent foster parent(s) 

 guardian other (who? ____________________________) 

5a. Are you in foster care? yes no

6. Do you or your family receive public assistance (food stamps, welfare, other)? 

 yes 

no

I don’t know

7. Do you receive a free or reduced lunch at school? 

 yes 

no

8. What is the highest grade in school completed by your parents/guardians? 

Mother Father

 less than high school less than high school 

 high school graduate high school graduate 

 some college 

 college graduate or more 

I don’t know 

 some college 

 college graduate or more 

I don’t know

9. Do you live with an adult who is working? 

 yes 

no

10. Where do you go to the doctor? (please check all that apply) 

I don’t have anyplace to go

private doctor’s office

 clinic at a hospital 

 clinic not at a hospital 

 school-based health center 

 emergency department 
49
50

11. Yes No

In the past year have you had a regular checkup and/or physical? 

In the past year have you had a dental exam? 

In the past year did you have a vision screening at your school? 

12. Do you have a bank account? 

yes  About how much money is in your account today? $__________

no

13. Which of the following is a “need” rather than a 

“want”?

A designer sweater

Money to pay for a meal

A new video iPod

A new CD by your favorite singer

I don’t know

14. What is a budget?

A plan for how to spend the money you receive

A cash register

The interest you pay on a loan

A document required to get a mortgage or a loan

I don’t know

15. Buying stock means that you are: 

Buying part of a company

Lending money to a company

Making a real-estate investment with a company

Paying interest to a company

I don’t know

16. Which of the following is NOT true about credit 

cards?

Interest rates vary widely

If you pay the minimum required each month, you 

won't pay any interest

Many require an annual fee

Some credit cards give you rewards

I don’t know

17. How often do you turn in your homework on time? 

I always turn it in on time

Usually

Sometimes

I hardly ever turn it in on time

18. Do you think you will go to college? 

yes—right after high school

yes—but later—not right after high school

probably not—but maybe

definitely not

19. Did you have to attend summer school last summer due to poor grades or missing too much school during 

the regular school year? 

yes

no

20. Have you ever had a paid job or paid internship? 

yes

no

21. Have you ever... Yes No

Smoked cigarettes? 

Had at least one drink of alcohol? 

Used marijuana? 

22. During the past 30 days, have you... Yes No

Smoked cigarettes? 

Had at least one drink of alcohol? 
51

Used marijuana? 

23. Have you carried a weapon during the past 30 days? 

yes

no

24. Have you been in a physical fight during the past 12 months? 

yes

no

25. Have you been arrested during the past 6 months? 

yes

no

26. Have you ever had sexual intercourse?

yes

no

27. How old were you when you had sexual intercourse for the first time?

 I have never had sexual intercourse 

11 years old or younger

12 years old

13 years old

14 years old

15 years old

16 years old

17 years old or older

28. The last time you had sexual intercourse, did you and your partner use a condom? 

 I have never had sexual intercourse 

yes

no 

29. The last time you had sexual intercourse, what ONE method did you and your partner use to prevent 

pregnancy? (Please check only one answer). 

 I have never had sexual intercourse 

 no method was used to prevent pregnancy

 birth control pills

 condoms

 Depo-Provera (injectable birth control)

 withdrawal

 some other method

 not sure

 

30. Thinking back over all the times you have had sexual intercourse in the past 6 months, how often have 

you and your partner(s) used or done something to keep from getting a sexually transmitted disease or to 

keep a pregnancy from happening? 

 I have never had sexual intercourse 

never

some of the time

about half of the time

most of the time

every time

30a. Have you ever been to a doctor or to a clinic like Planned Parenthood to get condoms/birth control, to 

have a pelvic exam or to get a checkup for sexually transmitted infections? 

yes
52

no
53

31. For females only:

Have you ever been pregnant?

I have never had sexual intercourse

no

yes  How many times? _____

How many babies have you given birth to? _____

Have you been pregnant in the past year?  yes no

32. For males only:

Have you ever gotten someone pregnant?

I have never had sexual intercourse

no—I know for sure I haven’t gotten someone pregnant

I don’t know

yes—I know for sure I have gotten someone pregnant

How many times have you gotten someone pregnant? _____

How many biological children do you have? _____

Have you gotten someone pregnant in the past year?  yes no

33. Which sports do you currently play? (please check all that apply) 

 swimming martial arts 

 golf karate 

 tennis other (specify: ___________________) 

 no participation this past year 

34. Do you play any of these sports on your own time? 

 yes 

no

35. Which activities do you currently participate in? (please check all that apply)

 art drama 

 music rap 

 dance drumming 

 writing/poetry t-shirt design 

 mask making other (specify: ___________________) 

 photography 

36. Do you do any of these activities on your own time? 

 yes 

no

37. Are you involved in any clubs or groups at school, church, or in the community? 

no

yes...if yes, please tell us which ones:__________________________________________________ 

38. Have you ever attended any sexuality education (either in a school class or in some other program you 

were in) including education about HIV/AIDS and sexually transmitted diseases (STDs) or how to prevent 

pregnancies? 

no

yes...if yes, please tell us where, the name of the class, and when: 

Place where you got the education Did the class have a name? What grade were you in?
54

38a. Have you ever attended any education on cigarette use, drug use, or alcohol use (either in a school class 

or in some other program you were in)? 

no

yes...if yes, please tell us where, the name of the class, and when:

Place where you got the education Did the class have a name? What grade were you in?

Here are some questions about family life and sexuality education to which you may or may not know the 

answers. Just mark (X) “don’t know” if you are unsure -- don’t guess. Be sure to mark (X) only one answer.

39. Ovulation usually happens two weeks before a woman’s menstrual period.

 True 

 False 

Don’t know

40. Pregnancy can happen anytime a woman has unprotected vaginal intercourse with a man. 

 True 

 False 

Don’t know

41. Puberty always starts at the same age for boys and girls. 

 True 

 False 

Don’t know

42. Sexuality is more than just sexual intercourse; it includes many different parts of what makes you a 

human being. 

 True 

 False 

Don’t know

43. “Sex” and “Gender” are two words for the same thing.

 True 

 False 

Don’t know

44. Sexual orientation refers to a person’s romantic attraction to someone of the same and/or different sex.

 True 

 False 

Don’t know

45. Assertiveness means communicating your needs or feelings in a way that hurts other people. 

 True 

 False 

Don’t know 

46. Teenagers in romantic relationships can only express love by having sexual intercourse. 

 True 

 False 

Don’t know
55

47. One reason to choose sexual abstinence is to live by your personal values. 

 True 

 False 

Don’t know

48. A person can choose sexual abstinence even if he or she has already had intercourse. 

 True 

 False 

Don’t know

Read the following questions and mark (X) the answer that you think is the best response. Answer them to the 

best of your ability and do not guess. If you do not know the answer you should choose “Don’t know.” Be sure 

to mark (X) only one answer.

49. Where is a man’s sperm produced?

In his urethra

In his testicles

In his foreskin

Don’t know

50. What do contraceptives prevent? 

HIV/AIDS

Peer pressure 

Pregnancy

Don’t know

51. What can a young man do when he starts producing sperm? 

Reproduce by getting a woman pregnant

Communicate more clearly

Urinate

Don’t know

52. What causes emotional changes during puberty for both boys and girls? 

Hormones

Sperm

Ova

Don’t know

53. What does “Homophobia” mean?

Being open-minded

Discriminating against a person because he or she is homosexual

Deciding not to have sexual intercourse

Don’t know

54. If a couple decides to have sexual intercourse, what can they do to prevent most sexually transmitted 

infections? 

Use the birth control pill regularly

Use a condom every time they have sexual intercourse

Use spermicidal gel most of the time

Don’t know

55. What are two ways that HIV is spread? 

Mosquito bites and drinking from water fountains

Donating blood and kissing

Sexual intercourse and sharing drug needles
56

Don’t know

56. Which sexually transmitted infection has a cure? 

Chlamydia

HIV

Herpes

Don’t know

57. Who are the best people to help you make difficult, personal decisions? 

Friends you don’t know very well

A parent, another family member, religious leader, or teacher 

Strangers who you will never have to see again

Don’t know

58. How should people choose the best method of contraception for themselves? 

Based on the color of the packaging

Based on what their friends recommend 

Based on the advantages and disadvantages of the method

Don’t know

Each of the following sentences is missing a word or phrase. Mark (X) the best answer you think should fill in 

the blank to complete the sentence. Answer them to the best of your ability and do not guess. If you do not 

know the answer you should choose “Don’t know.”

Be sure to mark (X) only one answer.

59. __________________ can prevent pregnancy. 

Antibiotics

Contraception

Menstruation

Don’t know

60. A young woman’s ability to reproduce starts when she begins to __________________.

Abstain

Communicate

Menstruate

Don’t know

61. __________________ are attitudes, beliefs, and ideas that help us make important decisions. 

Social changes

Emotional changes

Personal values

Don’t know

62. __________________ is the only 100% guaranteed way to prevent pregnancy and sexually transmitted 

infections. 

Sexual abstinence

Drinking alcohol

Dating

Don’t know

63. Sexually transmitted infections can be __________________ even if the person does not have signs of 

infection. 

Posted on Facebook

Transmitted

Seen
57

Don’t know

64. Teasing a person for being a virgin is an example of __________________. 

Helping

Peer pressure

Being a good friend

Don’t know

65. People usually make healthier decisions when they feel __________________ about themselves. 

Nervous

Embarrassed

Good

Don’t know

66. __________________ are the ways society expects boys and girls to act. 

Sexual orientation

The Five Ways to Be

Gender roles

Don’t know

67. To make a decision you should think about all of the long-term and short-term _____________. 

Opinions

Consequences

Feelings

Don’t know

68. __________________ means making the choice not to do something that could have bad consequences. 

Abstinence

Dating

Negotiation

Don’t know

Thank you! 
58

Appendix B: Teacher Survey 

Today’s date: _____/_____/_____ 

 

 month day year 

1. Gender:

male

female

2. Ethnicity: (please check all that apply)

African American/Black

Caribbean

Latino/Latina

Caucasian

Asian

other (specify: ___________________)

3. What grade(s) do you 

currently teach? (please 

check all that apply)

5

th 9th

6

th 10th

7

th 11th

8

th 12th

4. How long have you been a teacher? 

 This is my first year 

 One to three years 

 Four to six years 

 Seven to ten years 

 More than ten years 

5. In which school are you currently working?

Bronx Preparatory Charter

Urban Assembly for Math and Science for 

Young Women

Opportunity Charter

Academy for Scholarship and 

Entrepreneurship

Union Public School

6. How long have you been a teacher at this school?

This is my first year

One to three years

Four to six years

Seven to ten years

More than ten years

7. How much do you know about the CAS-Carrera 

program?

A great deal

Some

Not very much

I know nothing about the program

8. How involved are you with the CAS-Carrera 

program?

Very involved

Somewhat involved

Not very involved

I have no involvement with the program

8a. If you have no involvement with the CAS-Carrera program, what are your impressions about this 

program? 

 I have a very favorable impression of the program 

 I have a somewhat favorable impression of the program 

I have a somewhat negative impression of the program

I have a very negative impression of the program

9. What percentage of your time do you spend on...

 ...classroom instruction? ______% 

 ...non-disciplinary student needs?______% 

...student behavioral/disciplinary issues? ______% 

10. Overall, has the CAS-Carrera program had any affect on the time you can spend on instruction with your 

students? 

No

Yes...How? 

 I can now spend a lot more time on instruction 

 I can now spend a little more time on instruction 

 I can’t spend as much time on instruction
59

11. How often have you done the following over the past year? 

Very 

often Often Not very often Never

Worked in collaboration/conferred with Carrera social 

workers regarding approaches to managing student behavior.

Made a referral to Carrera staff regarding a specific student 

medical or dental issue.

Made a referral to a Carrera social worker regarding a 

specific classroom issue and/or crisis situation.

Referred a student/family to Carrera social workers.

Collaborated with Carrera education staff for additional 

assistance/advocacy.

Made a referral to the Carrera family life and sexuality 

instructor to address student needs.

12. Over the past year how much improvement have you seen in the students you teach regarding their...

A great 

deal Some Not very much None I don’t know

Behavior

General health/well-being

Academics

Sexual awareness

Sexual knowledge

Self-confidence

13. For the CAS-Carrera program components for which you are familiar, please indicate how helpful you 

believe each is to the students you teach. 

Very 

helpful Helpful

Not 

very 

helpful

Not 

helpful 

at all

I don’t 

have 

enough 

information 

to know

Weekly, in-class Power Group sessions (mental 

health classes) led by Carrera social workers.

Individual social work services with young people.

Weekly exposure to Job Club where young people 

earn stipends and open bank accounts.

Weekly family life and sexuality education.

Self expression.

Lifetime individual sports.

Comprehensive, no cost medical and dental services.

Additional education supports including 

remediation, homework help, and enrichment.

14. Compared to before the CAS-Carrera program existed in your school, how would you rate the timeliness 

of the following at your school? Would you say it’s better, worse, or the same as before the program existed 

in your school? 

Timeliness of... Better

About 

the 

same

Worse

I don’t have 

enough 

information 

to know

Interventions with young people who are experiencing mental 

health issues.
60

Interventions with young people who are experiencing 

medical/dental issues.

Interventions with young people who are experiencing behavioral 

issues.

Academic interventions and support.

A systematic referral process for students.

Student understanding of puberty and other developmental 

changes.

15. Overall, how much positive impact would you say the CAS-Carrera program is having on the students 

you teach? 

 A great deal of positive impact 

 Some positive impact 

 A little positive impact 

 No positive impact at all 

I don’t know

16. Has your way of teaching been influenced by CAS-Carrera youth development approaches/ holistic 

philosophy? 

I don’t know what that philosophy is

No

Yes...in what way? _________________________________________________________

17. Has the CAS-Carrera program changed the dynamics of your classroom in any way? 

No

Yes...in what way? _________________________________________________________

18. As a result of the CAS-Carrera program, have you observed improved student knowledge about 

puberty/development? 

I don’t know

No

 Yes 

19. Has the CAS-Carrera program had any negative effects on your job? 

No

Yes...Please explain: ____________________________________________________ 

 ____________________________________________________ 

 ____________________________________________________ 

20. Has the CAS-Carrera program had any negative affects on the students you teach? 

No

Yes...Please explain: ____________________________________________________ 

 ____________________________________________________ 

 ____________________________________________________ 

21. Can you suggest any improvements in the way the CAS-Carrera program is being implemented in your 

school, the staff running the program, or the way the program is managed? 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

22. Is there anything else you would like to tell us about the CAS-Carrera program? 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 
61

 ______________________________________________________________________________ 

Thank You!! 

Appendix C: Statistical Tests in Tulsa 

 The following tables show details regarding the statistical tests used in Tulsa for all longterm, mid-term, short-term outcomes, and program service objectives. The tables provide the 

reader with the percentages on these variables by grade for program and comparison youth). 

Tables showing regression results show effect sizes and significance levels for each independent 

variable by grade. 

Long-term Program Outcomes 

Pregnancy 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Ever been pregnant/caused a pregnancy (7th grade) 150 0% 153 0% -- -- --

Ever been pregnant/caused a pregnancy (8th grade) 150 0% 153 2% 302 4.00 .046

Ever been pregnant/caused a pregnancy (9th grade) 150 0% 153 3% 302 5.034 .026

Mid-term Program Outcomes 

Sexual Behavior and Contraception 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Ever had sex (7th grade) 150 4% 152 5% 301 .067 .796

Ever had sex (8th grade) 150 14% 152 18% 301 1.082 .299

Ever had sex (9th grade) 150 18% 153 29% 302 5.508 .020

Condom use at last intercourse (7th grade) 6 67% 7 57% 12 .106 .751

Condom use at last intercourse (8th grade) 21 76% 28 71% 48 .134 .716

Condom use at last intercourse (9th grade) 27 63% 45 62% 71 .004 .951

Use of condom & non-coital method at last intercourse (7th grade) 6 17% 7 0% 12 1.185 .300

Use of condom & non-coital method at last intercourse (8th grade) 21 0% 28 4% 48 .746 .392

Use of condom & non-coital method at last intercourse (9th grade) 27 4% 45 7% 72 .276 .601

Outcome (ANOVA) Program Comparison df F value Sig. N Score N Score

Birth control use over past 6 months (7th graders) 6 2.67 7 1.57 12 .946 .352

Birth control use over past 6 months (8th graders) 21 2.67 27 2.19 47 1.026 .316

Birth control use over past 6 months (9th graders) 27 2.78 45 2.38 71 1.031 .313
62

Violence and Delinquency 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Physical fighting in past year (7th grade) 150 29% 152 26% 301 .208 .649

Physical fighting in past year (8th grade) 150 25% 153 30% 302 .843 .359

Physical fighting in past year (9th grade) 150 22% 153 23% 302 .033 .856

Carried a weapon during the past 30 days (7th grade) 150 11% 152 8% 301 1.025 .312

Carried a weapon during the past 30 days (8th grade) 150 8% 152 16% 301 5.062 .025

Carried a weapon during the past 30 days (9th grade) 150 9% 153 16% 302 3.499 .062

Arrested in past 6 months (7th grade) 150 1% 152 1% 301 .348 .556

Arrested in past 6 months (8th grade) 149 2% 153 4% 301 .947 .331

Arrested in past 6 months (9th grade) 149 2% 153 1% 301 .230 .632

Drugs and Alcohol 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Ever smoked cigarettes (7th grade) 149 15% 152 9% 300 2.834 .093

Ever smoked cigarettes (8th grade) 150 21% 152 19% 301 .119 .731

Ever smoked cigarettes (9th grade) 150 23% 150 27% 299 .632 .427

Smoked in past 30 days (7th grade) 149 2% 152 1% 300 1.051 .306

Smoked in past 30 days (8th grade) 150 3% 152 5% 301 .680 .410

Smoked in past 30 days (9th grade) 150 5% 146 8% 295 1.552 .214

Ever used alcohol (7th grade) 149 28% 152 26% 300 .132 .716

Ever used alcohol (8th grade) 150 43% 152 52% 301 2.629 .106

Ever used alcohol (9th grade) 150 53% 150 59% 299 1.350 .246

Used alcohol in past 30 days (7th grade) 149 8% 152 4% 300 2.258 .134

Used alcohol in past 30 days (8th grade) 150 8% 152 15% 301 3.769 .053

Used alcohol in past 30 days (9th grade) 150 13% 149 16% 297 .489 .485

Ever used marijuana (7th grade) 150 9% 152 6% 301 1.246 .265

Ever used marijuana (8th grade) 150 25% 152 20% 301 1.041 .308

Ever used marijuana (9th grade) 150 35% 153 31% 302 .726 .395

Used marijuana in past 30 days (7th grade) 150 3% 152 4% 301 .385 .536

Used marijuana in past 30 days (8th grade) 149 5% 153 14% 301 6.156 .014

Used marijuana in past 30 days (9th grade) 150 17% 151 18% 300 .077 .781

Academic Indicators 

Outcome (ANOVA) Program Comparison df F value Sig. N Score N Score

College intent (7th graders) 149 2.61 151 2.50 299 1.641 .201

College intent (8th graders) 150 2.53 153 2.62 302 1.273 .260

College intent (9th graders) 149 2.47 152 2.59 300 2.308 .130

On-time homework completion (7th grade) 150 1.93 152 1.82 301 1.396 .238

On-time homework completion (8th grade) 150 1.99 152 2.07 301 .732 .393

On-time homework completion (9th grade) 150 2.00 153 1.96 302 .179 .672

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Summer school the previous summer (7th grade) 149 5% 151 13% 299 2.968 .015

Summer school the previous summer (8th grade) 150 5% 153 3% 302 .909 .341

Summer school the previous summer (9th grade) 150 2% 153 2% 302 .001 .981
63

Short-term Program Outcomes 

Family Life and Sexuality Education (FLSE) 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

7

th grade 150 42% 153 42% 302 .032 .859

8

th grade 150 79% 153 49% 302 190.449 .000

9

th grade 150 82% 153 55% 302 135.529 .000

Financial Literacy 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

7

th grade 150 46% 153 29% 302 32.137 .000

8

th grade 150 71% 153 51% 302 44.259 .000

9

th grade 150 74% 153 57% 302 31.429 .000

Program Service Objectives 

Bank Accounts 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Has bank account (7th grade) 148 95% 151 19% 298 422.685 .000

Has bank account (8th grade) 150 93% 151 30% 300 215.226 .000

Has bank account (9th grade) 149 92% 151 24% 299 269.597 .000

Health care 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Use of ER (7th grade) 150 9% 153 16% 302 4.094 .044

Use of ER (8th grade) 150 12% 153 8% 302 2.025 .156

Use of ER (9th grade) 150 9% 153 7% 302 .809 .369

Health checkup (7th grade) 149 75% 152 68% 300 1.688 .195

Health checkup (8th grade) 150 75% 151 80% 300 1.282 .258

Health checkup (9th grade) 149 78% 150 77% 298 .012 .915

Dental exam (7th grade) 150 85% 151 63% 300 19.461 .000

Dental exam (8th grade) 150 80% 152 89% 301 4.502 .035

Dental exam (9th grade) 150 87% 149 83% 298 1.335 .249

Eye exam (7th grade) 148 70% 150 17% 297 122.430 .000

Eye exam (8

th grade) 150 91% 150 15% 299 428.655 .000

Eye exam (9

th grade) 149 83% 148 12% 296 291.144 .000
64

 The following table shows the effect sizes and significance for each independent variable 

used in the linear regression equations for Tulsa 7

th graders. 

7

th Grade

Dependent variables

Effect sizes of independent variables

Male Age Caucasian AfricanAmerican Latino

Net effect 

of 

program 

partic.

Long-term Program Outcomes:

Ever been pregnant/caused a pregnancy (1=yes, 0=no): -- -- -- -- -- --

Mid-term Program Outcomes:

Ever had sexual intercourse (1=yes, 0=no): .014 .004 .004 .056 -.021 -.005

Condom use at last intercourse (1=yes, 0=no): .299 -.271 .193 .126 .034 .074

Use of condom and non-coital method at last intercourse (1=yes, 

0=no): .032 .138 -.370 -.366 -.748 .328

Frequency of birth control use in the past 6 months: 1.44 .238 .399 2.299 -1.350 1.595

In physical fight during past 12 months (1=yes, 0=no): .200*** .064 -.077 .041 -.117 .045

Carried a weapon during past 30 days (1=yes, 0=no): .120*** -.059 .060 -.005 -.013 .035

Arrested during past 6 months (1=yes, 0=no): -.019 -.005 .012 .028 -8.63 .005

Ever smoked cigarettes (1=yes, 0=no): .000 -8.00 -.009 -.040 -.061 .066

Smoked cigarettes in the past 30 days (1=yes, 0=no): -.001 -.006 -.009 -.009 -.024 .013

Ever used alcohol (1=yes, 0=no): .002 -.044 -.038 -.062 -.078 .014

Used alcohol in the past 30 days (1=yes, 0=no): -.024 -.007 -.055 .003 -.042 .036

Ever used marijuana (1=yes, 0=no): .022 .060 -.074 -.047 -.067 .046

Used marijuana in the past 30 days (1=yes, 0=no): .017 .000 -.033 -.015 -.005 -.014

College intent: -.088 -.069 .019 .077 -.307** .106

Homework completion: -.190 .039 -.006 -.344* -.259 .141

Short-term Program Outcomes:

FLSE scores: -.024 -.022 .009 -.038 -.103** .007

Financial literacy: -.022 -.074* .003 -.073 -.097* .166***

Program Service Objectives:

Having a bank account (1=yes, 0=no): .059 -.041 -.067 -.048 -.111* .759***

Using the ER as the only source of health care (1=yes, 0=no): -.004 -.034 -.062 .043 .033 -.091*

Having had a regular checkup within the past year (1=yes, 0=no): -.038 .027 -.144* -.096 -.071 .066

Having a dental exam within the past year (1=yes, 0=no): .046 -.103 -.022 -.010 -.055 .203***

Having had an eye exam within the past year (1=yes, 0=no): .077 -.069 -.028 -.064 .009 .528***

Difference is statistically significant at *p<.05 / **p<.01 / ***p<.001. 
65

 The following table shows the effect sizes and significance for each independent variable 

used in the linear regression equations for Tulsa 8th graders. 

8

th grade

Dependent variables

Effect sizes of independent variables

Male Age Caucasian AfricanAmerican Latino

Net effect 

of 

program 

partic.

Long-term Program Outcomes:

Ever been pregnant/caused a pregnancy (1=yes, 0=no): .007 .026 -.005 .019 -.011 -.021

Mid-term Program Outcomes:

Ever had sexual intercourse (1=yes, 0=no): .127** .086 -.017 .127* -.057 -.024

Condom use at last intercourse (1=yes, 0=no): .302 -.191 .071 .136 -.033 .012

Use of condom and non-coital method at last intercourse (1=yes, 

0=no): -.103 .015 -.053 .037 -.045 -.040

Frequency of birth control use in the past 6 months: .775 -.039 -.730 .279 -.703 .379

In physical fight during past 12 months (1=yes, 0=no): .216*** -.056 .146* .141* -.035 -.038

Carried a weapon during past 30 days (1=yes, 0=no): .095* -.009 -.012 -.072 -.060 -.075*

Arrested during past 6 months (1=yes, 0=no): .019 -.016 .004 .109*** .030 -.027

Ever smoked cigarettes (1=yes, 0=no): .020 .013 -.023 -.147* -.132* .031

Smoked cigarettes in the past 30 days (1=yes, 0=no): .018 -.040 .038 .017 -.004 -.023

Ever used alcohol (1=yes, 0=no): -.056 -.032 -.021 -.183* -.089 -.092

Used alcohol in the past 30 days (1=yes, 0=no): .004 .025 .092 -.014 .008 -.059

Ever used marijuana (1=yes, 0=no): .117* .002 -.086 -.015 -.036 .052

Used marijuana in the past 30 days (1=yes, 0=no): .042 -.021 .011 .106* .021 -.090**

College intent: -.073 -.082 -.079 .003 -.356** -.092

Homework completion: -.083 -.031 .191 -.199 -.081 -.063

Short-term Program Outcomes:

FLSE scores: -.042 -.026 .021 -.072* -.117*** .303***

Financial literacy: .029 -.036 .083* -.052 -.122** .209***

Program Service Objectives:

Having a bank account (1=yes, 0=no): .035 -.008 .047 -.024 -.069 .637***

Using the ER as the only source of health care (1=yes, 0=no): .034 .039 -.006 .006 .052 .053

Having had a regular checkup within the past year (1=yes, 0=no): .021 .030 -.009 -.004 -.087 -.044

Having a dental exam within the past year (1=yes, 0=no): -.048 -.029 .055 .015 -.037 -.091*

Having had an eye exam within the past year (1=yes, 0=no): -.005 .047 -.005 .015 .019 .773***

Difference is statistically significant at *p<.05 / **p<.01 / ***p<.001. 

 
66

 The following table shows the effect sizes and significance for each independent variable 

used in the linear regression equations for Tulsa 9th graders. 

9

th grade

Dependent variables

Effect sizes of independent variables

Male Age Caucasian AfricanAmerican Latino

Net effect 

of 

program 

partic.

Long-term Program Outcomes:

Ever been pregnant/caused a pregnancy (1=yes, 0=no): .001 .017 .009 .019 -.010 -.029

Mid-term Program Outcomes:

Ever had sexual intercourse (1=yes, 0=no): .126* .107* -.087 -.012 -.068 -.090

Condom use at last intercourse (1=yes, 0=no): .321* -.244* .125 -.023 .027 -.037

Use of condom and non-coital method at last intercourse (1=yes, 

0=no): -.063 .006 -.043 -.096 -.026 -.034

Frequency of birth control use in the past 6 months: .710 -.951* .277 .078 -.188 .280

In physical fight during past 12 months (1=yes, 0=no): .145** .044 .005 .109 -.125 .011

Carried a weapon during past 30 days (1=yes, 0=no): .122** -.040 .059 -.042 -.104* -.059

Arrested during past 6 months (1=yes, 0=no): .004 -.004 -.009 -.008 -.010 .007

Ever smoked cigarettes (1=yes, 0=no): .005 -.033 .100 -.177* -.093 -.023

Smoked cigarettes in the past 30 days (1=yes, 0=no): .022 -.041 .142*** .032 .009 -.032

Ever used alcohol (1=yes, 0=no): -.066 -.045 -.029 -.136 -.028 -.072

Used alcohol in the past 30 days (1=yes, 0=no): -.031 .017 -.005 .002 -.021 -.026

Ever used marijuana (1=yes, 0=no): .048 .053 -.129 -.083 -.068 .056

Used marijuana in the past 30 days (1=yes, 0=no): -.008 .017 .044 .105 .080 -.015

College intent: -.083 -.189* -.051 .093 -.168 -.159

Homework completion: -.172 -.188 .177 -.155 -.159 .024

Short-term Program Outcomes:

FLSE scores: -.030 -.048* .005 -.085** -.141*** .266***

Financial literacy: .053 -.080* .088* -.087* -.122** .175***

Program Service Objectives:

Having a bank account (1=yes, 0=no): .003 -.042 -.053 -.002 -.101 .676***

Using the ER as the only source of health care (1=yes, 0=no): .031 .030 -.033 .035 .038 .029

Having had a regular checkup within the past year (1=yes, 0=no): .057 .026 -.011 -.076 -.125 .021

Having a dental exam within the past year (1=yes, 0=no): -.016 -.029 -.049 -.078 -.009 .042

Having had an eye exam within the past year (1=yes, 0=no): -.024 -.029 -.107 -.120* -.105 .700***

Difference is statistically significant at *p<.05 / **p<.01 / ***p<.001. 
67

Appendix D: Statistical Tests in NYC 

 The following tables show details regarding the statistical tests used in NYC for each 

outcome that exists in both the CAS-Carrera program sample and the YRBS comparison samples 

in 2011 and 2013. 

Long-term Program Outcomes 

Pregnancy 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Ever been pregnant/caused a pregnancy (2011) 219 3% 202 10% 420 7.55 .006

Ever been pregnant/caused a pregnancy (2013) 219 5% 187 5% 405 .01 .922

Mid-term Program Outcomes 

Sexual Behavior and Contraception 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Ever had sex (2011) 219 38% 175 36% 393 .07 .787

Ever had sex (2013) 219 38% 182 45% 400 2.10 .147

Condom use at last intercourse (2011) 83 82% 39 46% 121 18.54 .000

Condom use at last intercourse (2013) 83 80% 42 62% 124 4.54 .035

Violence and Delinquency 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Physical fighting in past year (2011) 218 23% 218 35% 435 8.21 .004

Physical fighting in past year (2013) 217 12% 219 28% 435 17.49 .000

Carried a weapon during the past 30 days (2011) 219 6% 213 9% 431 1.40 .237

Carried a weapon during the past 30 days (2013) 218 5% 218 9% 435 3.59 .058

Drugs and Alcohol 

Outcome (ANOVA) Program Comparison df F value Sig. N % N %

Ever smoked cigarettes (2011) 216 6% 214 26% 429 34.03 .000

Ever smoked cigarettes (2013) 218 6% 213 21% 430 20.57 .000

Smoked in past 30 days (2011) 216 1% 214 7% 429 9.60 .002

Smoked in past 30 days (2013) 216 2% 208 6% 423 2.99 .084

Used alcohol in past 30 days (2011) 217 18% 197 32% 413 10.28 .001

Used alcohol in past 30 days (2013) 217 23% 204 30% 420 2.93 .087

Used marijuana in past 30 days (2011) 217 8% 208 24% 424 20.44 .000

Used marijuana in past 30 days (2013) 218 12% 208 20% 425 6.06 .014
68

 The following table shows the effect sizes and significance for each independent variable 

used in the linear regression equations for the NYC 2011 sample.

Dependent variables (2011)

Effect sizes of independent variables

Female Grade AfricanAmerican Latino Bronx Brooklyn

Net effect 

of 

program 

partic.

Long-term Program Outcomes:

Ever been pregnant/caused a pregnancy (1=yes, 0=no): .022 .060 -.017 -.018 -.100 -- -.067**

Mid-term Program Outcomes:

Ever had sexual intercourse (1=yes, 0=no): -.103 .040 .003 .053 -- -.078 .010

Condom use at last intercourse (1=yes, 0=no): -.280** -.030 -.024 -.170 .005 -- .334***

In physical fight during past 12 months (1=yes, 0=no): -.013 -.460 -.030 -.062 .380 -.402 -.125**

Carried a weapon during past 30 days (1=yes, 0=no): -.048 -.127 -.012 .003 .085 -.160 -.031

Ever smoked cigarettes (1=yes, 0=no): .007 -.297 -.046 .072 .331 -.241 -.203***

Smoked cigarettes in the past 30 days (1=yes, 0=no): -.025 -.111 -.036 .036 .105 -.110 -.060**

Used alcohol in the past 30 days (1=yes, 0=no): -.047 -.385 -.078 .042 .465 -.352 -.135**

Used marijuana in the past 30 days (1=yes, 0=no): .031 -.232 -.070 .003 .165 -.330 -.157***

Difference is statistically significant at **p<.01 / ***p<.001. 

 The following table shows the effect sizes and significance for each independent variable 

used in the linear regression equations for the NYC 2013 sample.

Dependent variables (2013)

Effect sizes of independent variables

Female Grade AfricanAmerican Latino Bronx Brooklyn

Net effect 

of 

program 

partic.

Long-term Program Outcomes:

Ever been pregnant/caused a pregnancy (1=yes, 0=no): .038 -.026 -.045 .029 -.155 -.128* .002

Mid-term Program Outcomes:

Ever had sexual intercourse (1=yes, 0=no): -.070 -.005 -.069 -.140 -.251 -.149 -.069

Condom use at last intercourse (1=yes, 0=no): -.073 -.132 .009 -.171 -.335 -.220 .182*

In physical fight during past 12 months (1=yes, 0=no): -.040 -.098 .006 -.066 -.089 .012 -.159***

Carried a weapon during past 30 days (1=yes, 0=no): .007 .017 -.053 -.036 .034 -.005 -.046

Ever smoked cigarettes (1=yes, 0=no): .045 .123 -.112 -.053 .144 .132 -.146***

Smoked cigarettes in the past 30 days (1=yes, 0=no): .019 -.033 -.065* .008 -.088 .002 -.036

Used alcohol in the past 30 days (1=yes, 0=no): .082 -.004 -.111 -.081 -.041 -.029 -.075

Used marijuana in the past 30 days (1=yes, 0=no): .063 -.039 -.055 -.053 -.259 -.175* -.088*

Difference is statistically significant at *p<.05 / ***p<.001.