CHAPTER 7

PROJECT OBJECTIVES, OUTCOMES, AND DATA

The main focus of this chapter is the ways in which Talent Search may
affect program participants. Before describing specific outcomes,
however, we consider projects' formal goals and informal expectations
concerning the ways that participants are intended to benefit from the
program. After presenting both quantitative and qualitative information
on participant outcomes, we discuss projects' data-collection and
evaluation practices.

# Talent Search Projects' Outcome Objectives 

Talent Search projects' outcome objectives are goals for the way the
program strives to affect participants. As discussed in chapter 2, the
national TRIO office requires all Talent Search projects to set specific
goals for the percentage of applicable participants expected to achieve
various outcomes during each program year.[^1] Each project's goals are
supposed to be challenging, taking into account the types of
participants served, the service context, and the project's track
record.

Table 7.1 displays the average goals across all Talent Search projects,
as reported in 1998--99 annual performance reports (APRs), and conveys
some of the variability in projects' goals. For example, while projects,
on average, expected 75 percent of participating seniors to be admitted
to a postsecondary institution, about a quarter of all projects set a
goal of 65 percent or lower, and another quarter set a goal of 84
percent or higher. Projects clearly expected to achieve the most success
with their objectives for secondary school retention and graduation.
Their lower goals for secondary and postsecondary re-entry reflect staff
views that high school dropouts and adults are more difficult
populations to serve. Although not shown here, analysis reveals that
projects' goals did not differ substantially or systematically by type
of host institution.

  ---------------------------------------- --------- ------------ ------------
  Table 7.1---Goals set by Talent Search                          
  projects for major participant outcomes:                        
  1998--99                                                        

  Outcome objective                        Average   25th         75th
                                           goal      percentile   percentile

  **Secondary school retention**\          89%       85%          97%
  (percentage of secondary school                                 
  participants who will continue in                               
  secondary school)                                               

  **Secondary school graduation**\         88        83           95
  (percentage of high school seniors and                          
  GED or alternative education students                           
  who will graduate or receive equivalency                        
  certificate)                                                    

  **Secondary school re-entry**\           64        50           80
  (percentage of secondary school dropouts                        
  who will re-enter secondary education                           
  program)                                                        

  **Postsecondary admissions**\            75        65           84
  (percentage of high school graduates and                        
  equivalency recipients who will enroll                          
  in postsecondary education program)                             

  **Postsecondary re-entry**\              65        50           80
  (percentage of postsecondary "stopouts"                         
  who will re-enter postsecondary                                 
  education program)                                              
  ---------------------------------------- --------- ------------ ------------

[Source]{.smallcaps}: Analysis of data from Talent Search Performance
Reports, 1998--97.

Talent Search projects do not necessarily limit their outcome objectives
to the five major ones discussed above. The project survey also asked
about objectives for a handful of other outcomes of potential interest.
As shown in table 7.2, about four-fifths of all projects (82 percent)
had a specific performance objective for the percentage of seniors
applying for financial aid, and about the same proportion had an
objective for participant grade-level progression.[^2] In contrast, less
than one-half of all projects (48 percent) had a goal pertaining to
participants' grades.

  ---------------------------- ---------- ------------- ---------- -------- -----------
  Table 7.2---Additional                                                    
  outcome objectives                                                        

                                          Host                              
                                          institution                       

                               All        Public 4-year Private\   2-year   Community
                               projects                 4-year              org.

  Percentage of projects with                                               
  specific performance                                                      
  objectives concerning the                                                 
  following outcomes:                                                       

  Percentage of seniors        82%        86%           79%        78%      84%
  applying for financial aid                                                

  Participant grade            81         79            87         80       85
  progression                                                               

  Percentage going to college  58         56            64         48       72
  full-time in fall after                                                   
  graduation                                                                

  College preparatory course   51         57            51         45       52
  selection (middle school)                                                 

  Participant grades           48         51            59         39       50

  Projects' average goal for:                                               

  Percentage of seniors        87         87            90         90       81
  applying for financial aid                                                

  Percentage going to college  72         71            73         74       68
  full time in fall after                                                   
  graduation                                                                
  ---------------------------- ---------- ------------- ---------- -------- -----------

[Source:]{.smallcaps} National Survey of Talent Search Projects,
1999--2000.

Overall, with a couple of exceptions, there was relatively little
variation between projects operated by different types of host
institutions. As for the exceptions, projects hosted by 2-year colleges
were substantially less likely than those hosted by community
organizations to have established a performance goal addressing
full-time college enrollment in the fall after graduation, and projects
hosted by 2-year colleges were substantially less likely than those
hosted by private 4-year colleges to have established a performance goal
addressing participants' grades. The reasons for these differences are
unclear, but, in the latter case, it may be worth noting that the two
groups of projects were, respectively, the least and most likely to
provide tutoring services (see chapter 6, table 6.1).

The survey also asked projects to report their goals for the following
two outcomes: percentage of seniors applying for financial aid and
percentage attending college full time in the fall after high school
graduation. For the former outcome, the average goal was 87 percent; for
the latter, 72 percent.

We conclude this section by summarizing information from the case
studies on desired outcomes. We asked project and target school staff
how students who had participated in Talent Search would be expected to
differ from similar nonparticipants as a result of their experiences in
the program. As might be expected, respondents consistently said that
participants ought to demonstrate higher rates of high school graduation
and postsecondary enrollment, which are the overarching objectives of
Talent Search. Other outcomes mentioned by respondents included better
knowledge of careers and their educational requirements; better
knowledge of financial aid; improved self-esteem or self-confidence;
better sense of direction in life; and greater comfort on college
campuses and more knowledge of assistance available to them there, such
as Student Support Services. Some comments reflected projects'
particular service emphases. For example, at projects that focused on
academic support, interviewees mentioned better secondary school grades
and test scores as expected outcomes. As for longer-term outcomes, many
respondents felt fairly certain that Talent Search alumni would do
better and stay longer in college than similar nonparticipants. But they
also felt strongly that Talent Search projects should not be held
accountable for such outcomes. As one project director stated, "We just
show them to the door" of college. In the opinion of this director and
other staff, once participants left Talent Search, many other factors
influenced their lives, none of which the program could affect.

# Participant Outcomes

We turn now to a discussion of how participants may have benefited from
Talent Search. First we consider information on participant outcomes
from APRs and the project survey.[^3] Then we summarize anecdotal
comments about outcomes from case study interviews.

## Project-Reported Information on Outcomes

An aggregate analysis pooling 1998--99 APR data from all Talent Search
projects showed that for three of the five major outcome objectives
discussed above, the program as a whole appeared to exceed the average
goal set by projects (table 7.3). About 97 percent of 6th through 11th
graders stayed in school from one academic year to the next, exceeding
the average goal of 89 percent for secondary school retention. About 94
percent of participating high school seniors (and GED or alternative
education students) graduated from high school or received a certificate
of high school equivalency, exceeding the average goal of 88
percent.[^4] And about 72 percent of postsecondary education stopouts
had re-entered a postsecondary education program during the program year
or reportedly would do so in the following fall. The rate exceeded the
average goal of 65 percent for postsecondary re-entry.

However, the program as a whole appeared to fall short of the average
goal set by projects in two areas. About 51 percent of high school
dropouts re‑entered a secondary education program during the program
year, short of the average goal of 64 percent. (APR data did not permit
calculation of an overall rate, including middle school dropouts.) And
about 71 percent of high school (and high school equivalency) graduates
had enrolled in a postsecondary education program during the program
year or were reportedly planning to do so in the following fall, short
of the average goal of 75 percent for postsecondary admissions.
(Analysis also revealed that for the two key outcomes---high school
graduation and postsecondary admissions---there was very little
variation by type of host institution in either projects' average goals
or their actual outcomes.)

APRs also contain data on two more outcomes, in this case for high
school seniors and high school (or equivalency) graduates served, also
known as "college-ready" participants. Nationally, 83 percent of these
participants applied for postsecondary admission and 82 percent applied
for financial aid.[^5]

Rather than pooling data across all projects, an examination of results
for individual Talent Search projects provides a more detailed
perspective on participant outcomes. Such an analysis reveals that the
large share of projects met their goals for secondary retention,
secondary graduation, and postsecondary re-entry and that just over half
of all projects met their goals for postsecondary admission (table 7.3).
However, a majority of projects failed to meet their goals for secondary
re-entry, a finding that may provide empirical evidence in support of
staff statements about the difficulty of serving dropouts.

  -------------------- ---------------- -------------- -- ---------------- ------------ ------------
  Table 7.3---Talent                                                                    
  Search projects'                                                                      
  success in meeting                                                                    
  goals for major                                                                       
  participant                                                                           
  outcomes: 1998--99                                                                    

  Outcome objective    Aggregated,                        Disaggregated,                
                       national-level                     project-level                 
                       data                               data                          

                       Average goal     Percentage of     Percentage of    Percentage   Percentage
                                        participants      projects that    of projects  of projects
                                        that achieved     met or exceeded  that missed  that missed
                                        the outcome       their goal       their goal   their goal
                                                                           by five      by more than
                                                                           percentage   five
                                                                           points or    percentage
                                                                           less         points

  Secondary school     89%              97%               84%              7%           9%
  retention                                                                             

  Secondary school     88               94                87               6            7
  graduation                                                                            

  Secondary school     65               51                38               3            59
  re-entry                                                                              

  Postsecondary        75               71                53               18           29
  admission                                                                             

  Postsecondary        65               72                81               4            15
  re-entry                                                                              
  -------------------- ---------------- -------------- -- ---------------- ------------ ------------

[Source]{.smallcaps}: Analysis of data from Talent Search Performance
Reports, 1998--99.

[Note]{.smallcaps}: The number of projects in this analysis varied by
outcome, ranging from 328 for postsecondary admissions to 113 for
postsecondary re-entry. Like other outcomes analyses performed on APR
data (U.S. Dept. of Education May 2002), the analyses followed a
two-part strategy. First, projects were included only if they reported
data on their outcome goal, number of relevant participants, and number
of participants achieving the outcome. Second, apparently erroneous data
were corrected. Specifically, when the number of participants reported
as achieving an outcome exceeded the relevant number of participants
reported earlier in the APR, we capped the outcome number as equal to
the participant number, resulting in a 100 percent success rate for
these cases. Data problems such as these should be eliminated with the
new, Internet-based APR form.

### Postsecondary Outcomes

The project survey data enabled us to look more closely at projects'
expected outcomes for one particular group---participants who had earned
either a high school diploma or a GED by spring 1997. These participants
are of special interest because they were in a position to fulfill
Talent Search's ultimate objective of enrollment in a postsecondary
education program. Results show that project staff expected the vast
majority (75 percent) of participants to enroll in a 2- or 4-year
college by fall 1999 (table 7.4). They reported that relatively few
would go on to a vocational or proprietary school. About 2 percent were
reportedly planning to enroll in other types of programs or
institutions, with the military specified most frequently (52 of 80
respondents who provided a written answer). On average, projects
reported that 10 percent of high school graduates and GED recipients
would not continue their schooling and that the status of 5 percent of
participants was unknown. Finally, participants at projects hosted by
private 4-year colleges were reportedly more likely to attend a 4-year
college, and participants at projects hosted by 2-year colleges were
reportedly more likely to enroll in that type of college.

  ------------------------ ---------- ------------- ---------- -------- -----------
  Table 7.4---Expected                                                  
  fall 1999 status of                                                   
  participants who had                                                  
  graduated from high                                                   
  school or received a GED                                              
  by spring 1999                                                        

                                      Host                              
                                      institution                       

  **Average percent who    All        Public\       Private\   2-year   Community
  would:**                 projects   4-year        4-year              org.

  Enroll in a 4-year       41%        47%           54%        31%      43%
  college                                                               

  Enroll in a community    34         25            23         46       33
  college                                                               

  Enroll in a vocational   7          9             7          5        7
  or proprietary school                                                 

  Enroll in a tribal       1          1             0          1        1
  college^a^                                                            

  Enroll in some other     2          2             1          2        2
  program or institution                                                

  Not continue their       10         10            8          11       10
  schooling                                                             

  Education status unknown 5          6             6          4        4
  ------------------------ ---------- ------------- ---------- -------- -----------

[Source:]{.smallcaps} National Survey of Talent Search Projects,
1999--2000.

^a^The survey noted that participants who would be entering a tribal
college that was also a community college should be listed in the tribal
college response category.

APR data provide similar details on the placements of high school
graduates and postsecondary re-entry students *who were going on to some
type of postsecondary education program*. Overall, about two-fifths were
reportedly planning to attend a public 4-year college, and an almost
equal proportion was reportedly planning to attend a 2-year institution
(table 7.5). Many fewer were headed for a private 4-year college or a
trade or vocational school. The pattern for projects hosted by community
organizations was nearly identical to that of the overall pattern for
all projects. But among projects hosted by postsecondary institutions,
there was an increased likelihood that participants would attend an
institution of the same type. For example, whereas 11 percent of the
students overall were reportedly admitted to a private 4-year college,
20 percent of students at projects hosted by private 4-year colleges
were reportedly admitted to a private 4-year college.

  -------------------------- ---------- ------------- ---------- -------- -----------
  Table 7.5---Types of                                                    
  postsecondary institutions                                              
  that high school graduates                                              
  and postsecondary re-entry                                              
  students planned to                                                     
  attend: 1998--99                                                        

                                        Host                              
                                        institution                       

  Percentage admitted or     All        Public\       Private\   2-year   Community
  readmitted to              projects   4-year        4-year              org.

  Public 4-year institution  41%        48%           44%        31%      41%

  Private 4-year institution 11         9             20         8        12

  Public or private          40         34            28         55       41
  nonprofit 2-year                                                        
  institution                                                             

  Proprietary school or      6          8             9          6        5
  public or private                                                       
  nonprofit vocational/                                                   
  technical institution                                                   
  -------------------------- ---------- ------------- ---------- -------- -----------

[Source]{.smallcaps}: U.S. Department of Education, Office of
Postsecondary Education, *A Profile of the Talent Search Program:
1998--99*, Washington, DC: May 2002.

### Outcomes for Dropouts

The project survey also collected information from directors on outcomes
for dropouts. Two potential short-term outcomes for secondary school
dropouts are to re-enter a secondary education program (for example,
regular or alternative high school) or to prepare for and receive a
GED.[^6] The survey found that 63 percent of all Talent Search projects
counted at least one participant preparing for the GED in 1998--99
(table 7.6). The number preparing for the GED averaged about 23
participants per project and was equivalent to about 68 percent of the
total number of secondary school dropouts served by projects. The number
of participants who received a GED averaged about 14 participants per
project and was equivalent to about 61 percent of the total number
preparing. (This figure can be interpreted only as a rough estimate of
the overall GED success rate because some of those who were preparing
may not have taken the GED examination during the same program year, and
some who took the examination may not have prepared during the same
year.) Some of the variation by type of host institution, particularly
for the private 4-year category, may be more attributable to the small
number of projects in this analysis rather than to real differences.[^7]

  --------------------------- ---------- ------------- ---------- -------- -----------
  Table 7.6---GED preparation                                              
  and outcomes: 1998--99                                                   

                                         Host                              
                                         institution                       

                              All        Public\       Private\   2-year   Community
                              projects   4-year        4-year              org.

  Percentage of projects with 63%        66%           46%        65%      68%
  participants preparing for                                               
  a GED                                                                    

  Average number of           23         25            11         21       25
  participants who were                                                    
  preparing                                                                

  Number preparing as a       68%        72%           68%        64%      67%
  percent of number of                                                     
  secondary school dropouts                                                
  served                                                                   

  Average number that         14         16            6          11       19
  received a GED                                                           

  Number of GED recipients as 61%        65%           51%        49%      71%
  a percent of the number who                                              
  were preparing                                                           
  --------------------------- ---------- ------------- ---------- -------- -----------

[Source:]{.smallcaps} National Survey of Talent Search Projects,
1999--2000.

### Exploring Outcome Differences

Using 1999-2000 APR data, we explored what factors might account for
differences in some of the outcomes discussed above. Specifically, we
used multivariate regression analysis to examine (1) what might account
for variation in the key project outcomes of secondary school
graduation, postsecondary admissions, and admission to 4-year
postsecondary institutions, and (2) why graduates from projects hosted
by postsecondary institutions were relatively more likely to enroll at
the same type of college that hosted the Talent Search project they
participated in. We controlled for factors such as the characteristics
of host institutions (type, location), total number of participants
served, percentage of new versus continuing participants, number of
years in operation, participants' demographic characteristics
(race/ethnicity, status on eligibility criteria, grade level), and the
percentage of participants receiving each of the ten services.
Unfortunately, however, the analyses provided little insight; the models
explained only about 20 to 30 percent of the variation in project
outcomes. Better data---especially measures of more factors that may
influence outcomes, such as staff models and tenure---may be needed to
provide more insight. As for differential enrollment rates at various
types of colleges, it may simply be that students' familiarity with
their host institutions may lead them to seek out similar types of
colleges when they finish high school.

## Subjective Information on Outcomes

People we interviewed during our site visits---project staff, school
staff, students, program alumni, parents---all viewed Talent Search as a
positive influence on participants. Current and former participants felt
that they were better off than they would have been without the program.
The outcomes mentioned by respondents varied considerably within
specific projects and even within specific target schools. In other
words, particular projects did not emerge as primarily affecting
students in any particular way. We also did not detect any systematic
differences in the opinions expressed by different types of
interviewees. Below we summarize numerous statements made by various
respondents, thereby illustrating the range of ways in which respondents
saw Talent Search affecting participants.

First, however, a note on respondents' comments. We met only with
students who were currently participating in the program and with alumni
who had gone on to college---individuals whose opinions on Talent Search
are likely to be more favorable than those of students who stopped
participating in the program or those of alumni who did not pursue a
college education. Furthermore, anecdotes do not constitute rigorous
evidence that the Talent Search program has positive effects on
participants. Separating outcomes "caused by" Talent Search from those
more attributable to other experiences of the same students requires
systematic comparisons of outcomes for participants and similar
nonparticipants.[^8]

The list of ways in which Talent Search was said to have helped students
was both long and diverse. One commonly mentioned benefit was *more
knowledge and information regarding postsecondary education*.

-   Participants at one high school felt that Talent Search had helped
    them answer some of the major questions they had about going to
    college, such as what GPA is needed, how much college costs, what
    support services are available, how big colleges are, what classes
    are required for admission, and, once at college, what type of
    scholarships and financial aid may be available.

-   Another student said that he did not even know that colleges offered
    tours to interested students. He had always just thought he would
    apply, be expected to pay tuition up front, and then just show up.

```{=html}
<!-- -->
```
-   A liaison to the program at one target high school opined that,
    compared with nonparticipants, Talent Search participants have the
    necessary information to make good decisions, such as what classes
    they need to take.

Some participants and alumni felt that the program had *improved their
access to and broadened their choice of colleges*. Beyond just providing
students with needed information, Talent Search project staff provided
other services to make sure that students got into college---in some
cases a college they had assumed was out of reach.

-   A high school student described taking a program-sponsored trip
    while in middle school to visit a particular college and realizing
    instantly that it was the place for him. A second student said that
    because of Talent Search he now knew he could go straight to a major
    university. A third student said that he had decided to go directly
    to college after high school instead of first joining the Navy.

-   An alumna claimed that without Talent Search she would have attended
    a community college; instead, she enrolled in a 4-year college.
    Furthermore, she said that if she had initially attended a community
    college, she doubted that she would have later transferred to a
    4‑year college. She said that her Talent Search advisor helped her
    prepare for the possibility of attending a 4-year college by making
    sure that she took algebra II and the SAT. She also saw the advisor
    as instrumental in securing a scholarship for her.

```{=html}
<!-- -->
```
-   Another alumnus said that in high school he did not think he could
    go to directly to a 4-year college but that his Talent Search
    advisor kept encouraging him to set his sights on a 4-year
    institution. When he reached his senior year in high school, the
    student still had not taken enough of the appropriate classes to
    attend a public 4-year college; nonetheless, his advisor said that
    enrollment in a 4-year institution was still possible. The advisor
    made some calls and succeeded in convincing a major public
    university to add the student's name to a special admission list;
    the student has since graduated from that university.

Yet another outcome mentioned by some participants was *doing things
sooner to prepare for college than they would have otherwise*. Officials
at one project said that Talent Search helps keep students on track for
important steps such as registering for and taking college entrance
examinations, applying for college admissions, and completing financial
aid applications. At another project, a high school student said that if
it were not for Talent Search, he probably would have "let it slide
until twelfth grade." Similarly, a middle school student said that if
there were no Talent Search, "I wouldn't *do* anything. I'd just worry
about it until I hit high school." And at a third project, all of the
alumni we interviewed concluded that the assistance that enabled them to
complete their college applications and financial aid forms early was
one of the best aspects of Talent Search.

Several interviewees described how Talent Search led to *expanded
horizons.* In other words, participants had developed broader
perspectives and saw more opportunities for themselves. As described
above, some participants saw Talent Search as expanding their horizons
with regard to college. Students who at first suspected that a college
education was beyond their reach came to see it as within their grasp;
students who initially thought they could attend only a certain type or
level of college came to see attendance at a higher-level, more
expensive, or more prestigious institution as a realistic option. We
also heard several comments about broader perspectives on careers.

-   A project staff member who worked with middle school students saw
    the project's career focus as raising participants' career goals. At
    the beginning of the school year, when asked about what careers they
    wanted to pursue, some students would name occupations such as hair
    stylist or mechanic. Toward the end of the year, though, those
    students were aiming higher for more advanced careers.

-   Two teachers whose classrooms were served by that staff member
    independently said that Talent Search taught their students a lot
    more about careers and college than they themselves could have and
    that the project helped students make an important connection
    between jobs and education much earlier than they otherwise would;
    it gave them a reason for the things they did in school.

```{=html}
<!-- -->
```
-   At another project, a high school official felt that, compared with
    nonparticipants, Talent Search participants tended to grow more in
    their career interests. The parent of a participant in that project
    described how the career-interest tests taken by her daughter had
    opened her daughter's eyes and helped lead her to an intended
    college major and career field. Furthermore, her son, a Talent
    Search alumnus, originally had professed aspirations of following
    his father by working in a mechanic-related job in the mining
    industry, but the program helped him discover an aptitude for
    mathematics and computers.

One outcome mentioned at a few projects, especially those we selected
for their emphasis on academic assistance, was *improved academic
performance* as reflected in higher grades or test scores. At projects
that featured substantial tutoring components, many interviewees were
quick to cite tutoring success stories.

-   *Grades.* At one project, an alumnus said that when he started in
    Talent Search, he was earning a D in algebra and felt unable to do
    any better on his own; with tutoring, however, he raised his grade
    to a high B. Tutors for the project told of a student who had
    improved his GPA from 0.8 to 3.0 over a two-year period and of a
    second student who had failed algebra twice but passed it after
    tutoring. At another project, students mentioned improving their
    grades in language arts as a result of the story writing they
    completed in Talent Search.

-   *Grade-level promotion and graduation.* At one target high school, a
    student speaking on behalf of himself and some other juniors said
    that, without Talent Search tutoring, "Some of us would still be in
    tenth grade." Another student said that some juniors would have
    dropped out. Similarly, the director of another project pointed to
    students' grade-to-grade promotion rates and high school completion
    rates as major indicators of the program's success.

-   *College entrance examinations.* A high school guidance counselor at
    one project said that students who spent a decent amount of time in
    Talent Search preparing for the ACT examination often raised their
    composite scores on a retake by at least two points and occasionally
    by much more.[^9] In addition, a counselor from a high school at
    another project that served all the juniors and seniors said that
    SAT scores had increased in the project school compared with
    nontarget schools in the same district.

-   *Other tests.* Staff at one project cited pre- to post-test
    improvements on teacher-prepared examinations as well as
    improvements on the standardized achievement test used by the school
    district. Staff from another project said that the program was
    motivating students to work harder and helping more of them pass the
    state's high school exit examination.

```{=html}
<!-- -->
```
-   *Study habits and test-taking skills.* At one target high school, a
    student said that her tutor helped her to feel prepared, calm, and
    in control when taking mathematics tests. He also showed her
    different ways of approaching mathematics problems; those methods
    surprised her own teacher when she used them on a test. A guidance
    counselor from one of the same project's target high schools
    mentioned that students learned how to study in groups. Students
    from another project mentioned that they improved their study skills
    such that they were less likely to forget what they had learned in
    school. Staff from a third project described their participants as
    growing more confident about taking the ACT and SAT.

Several comments focused on how Talent Search participants were *better
prepared to succeed in college* than were their peers. To a certain
extent, interviewees saw this outcome as the cumulative effect of the
various services provided by Talent Search, but sometimes they cited
specific activities or experiences, especially those that made students
more comfortable in a college environment and more familiar with support
services.

-   At one host college, officials viewed Talent Search alumni who
    enrolled there as demonstrating both a stronger determination to
    pursue college and a better understanding of the commitment needed
    to succeed in college than did similar students who had not been
    project participants. At another college, an administrator reported
    that when Talent Search alumni enrolled in that institution, they
    performed better than other students because of the academic support
    they had received, the general exposure to college they gained
    through field trips, and their ability to deal with the new freedoms
    offered by the college environment.

-   An alumnus from one project explained that Talent Search not only
    opened the door to college but also introduced him to a network of
    support services to help him succeed. Specifically, his Talent
    Search advisor in high school told him about some important
    resources at the university he was going to attend, such as tutoring
    assistance from a program similar to Student Support Services.

```{=html}
<!-- -->
```
-   An alumnus from a different project said a "bridge" program that she
    participated in during the summer after high school graduation
    helped with networking and gave her a connection to some of the
    "higher-ups" on campus. Another alumnus of the same project said
    that the connections he made through Talent Search and the bridge
    program helped him secure a work-study job at the host college when
    he enrolled there. He also credited Talent Search for helping him
    feel comfortable in speaking to professors during their office
    hours, which he thought had helped him do better in classes.

In addition to the above outcomes, people mentioned several other
perceived benefits of participating in Talent Search. Some of these
additional outcomes were tangible. For example, a high school counselor
pointed to the amount of scholarship money received by participants.
Students at one project spoke of improving their attendance at school so
that they would be eligible to have the Talent Search program pay part
of the cost of a school-sponsored trip to Disney World. Participants in
some projects reported benefiting from referrals to summer programs,
such as an Upward Bound Math/Science program. Two students at one target
high school felt that they had benefited from assistance in securing a
summer job.

Other outcomes were more intangible, such as changed attitudes.

-   Students at one project, for example, described *learning to control
    their temper* and walk away from certain situations in response to
    what they had learned about conflict resolution. A participant at
    another project said, "I have more patience."

-   Sometimes staff described students as *developing a "sense of
    direction"* or a clearer plan for their lives. A student speaking
    for several others said that they now believed that they had a goal
    for the future. Although we stressed earlier that most students
    already had a desire to attend college when they joined Talent
    Search, a few high school students and alumni said the program had
    changed their aspirations. Before joining Talent Search, they said,
    they mostly wanted to "get out of school and just live life";
    eventually, they developed a desire for a college education.

-   Another perceived intangible benefit was *increased confidence*. At
    one project, a high school student said that the program made him
    feel better about himself and the future, that he became more
    confident. Elsewhere, a student described how her participation in
    Talent Search had increased her self-confidence to the point where
    she was able to compete in speech and debate tournaments. Similarly,
    at another project, a student said, "I was shy and now I am more
    outspoken."

-   Interviewees at a couple of projects described participants as
    becoming *more organized* and better at time management.
    Participants also cited *improved communication and teamwork*
    skills. An alumna said that the program taught her how to *set goals
    and achieve them*.

-   A middle school counselor said that the program *builds self-esteem*
    and makes students feel special. Coincidentally, a middle school
    student from a different project said, "I learned how to respect
    myself."

```{=html}
<!-- -->
```
-   Talent Search also reportedly helped *motivate students to work
    harder,* according to staff from one project we visited. Elsewhere,
    an alumnus said that Talent Search helped build his motivation to
    work harder in school, helped him learn to enjoy school, and kept
    him focused on "doing good things" and staying away from bad
    influences. A current participant in that project said that he had
    been earning As and Bs but was not necessarily trying to do his
    best. With Talent Search, he made homework completion a high
    priority.

Did other groups besides participants experience positive outcomes?
Despite little or no talk of "spillover" effects, in which benefits
extend directly or indirectly to nonparticipants in target schools or
the wider community, we heard the occasional comment about how Talent
Search had benefited parents of participants. At one project, a
noncollege-educated parent said that through her children's involvement
in Talent Search she herself had "learned lots about college." A school
liaison to a second Talent Search project said that she sometimes sees
participants' parents become interested in pursuing postsecondary
schooling. At a third project, a parent said that the program
"empowered" her and helped her become more involved in her children's
education; it gave her more confidence to talk to her children's
teachers.

Many of the positive outcomes described above are closely interrelated.
Some of them may be seen as intermediate outcomes that lead to other,
subsequent outcomes. For example, attitudinal changes, such as greater
self-confidence, may lead to behavioral changes, such as better academic
performance; better academic performance, in turn, may lead to greater
postsecondary options; and so on. But some relationships between
outcomes may be more complex. In the view of one target school guidance
counselor, for instance, academic success is the best way to build
self-esteem and the confidence that leads to more success. A
longitudinal study would be necessary to explore such issues.

# Project Data, Record Keeping, and Evaluation

The ability of Talent Search projects to demonstrate empirically how
they help program participants depends on the participant information
they collect and maintain. Systematically collecting and analyzing
process and outcome information is also critical to individual projects'
evaluations. Both the project survey and the case studies shed light on
projects' practices in collecting, maintaining, and evaluating process
and outcome information.

Two questions in the survey directly addressed projects' data-collection
efforts.[^10] One asked about the types of items that projects had
attempted to measure. The outcome that the largest share (85 percent) of
projects tried to measure was completion of college financial aid forms,
followed closely by completion of college applications, which 82 percent
of projects had attempted to measure (table 7.7).[^11] Of the eight
items we asked about, a majority of projects had not measured three of
the items---participant self-esteem, completion of GED preparation
courses, and number of college preparation courses taken. There was
little variation between projects operated by different types of host
institutions. Overall, almost one-third (30 percent) of projects had
tried measuring seven or eight of the items, but about 23 percent had
tried measuring three or fewer; the mean number that projects had tried
to measure was 5.1.

  ----------------------------- ---------- ------------- ---------- -------- -----------
  Table 7.7---Participant                                                    
  information that Talent                                                    
  Search projects have                                                       
  attempted to measure                                                       

                                           Host                              
                                           institution                       

                                All        Public 4-year Private\   2-year   Community
                                projects                 4-year              org.

  Percentage of projects that                                                
  have attempted to measure:                                                 

  College financial aid form    85%        85%           84%        84%      87%
  completion                                                                 

  College application           82         82            78         80       87
  completion                                                                 

  College aspirations           70         73            70         67       68

  SAT/ACT test taking           68         67            64         67       73

  Financial aid awareness       62         60            62         64       63

  Participant self-esteem       49         48            54         50       45

  GED course preparation        44         42            46         43       47
  completion                                                                 

  Number of college preparatory 43         48            35         41       47
  courses taken                                                              

  Percentage of projects that                                                
  have attempted to measure:                                                 

  All eight of the above        12         10            14         12       13

  Seven of the above            18         22            14         18       15

  Six of the above              22         20            25         22       22

  Five of the above             14         17            14         12       10

  Four of the above             12         8             14         10       22

  Three of the above            8          10            3          8        8

  Two or fewer of the above     15         14            17         17       10
  ----------------------------- ---------- ------------- ---------- -------- -----------

[Source:]{.smallcaps} National Survey of Talent Search Projects,
1999--2000.

A second survey question asked about the types of participant
information that projects tracked or monitored and whether projects
tracked or monitored the information for some or all participants. All
but one of the eight items---hours of participation in the
program---referred to participant outcomes. All respondents reported
that they tracked or monitored high school graduation, and over 95
percent tracked or monitored progress through high school, college
enrollment, and completion of college applications (table 7.8). Only one
item was tracked by a majority of projects, but not typically for all
participants: participants' course selection. College graduation was the
only outcome that a majority of projects did not track or monitor at
all. Almost two‑thirds (63 percent) of all projects tracked or monitored
seven or all eight types of information, with the mean number of items
at 6.7.

  ------------------------------- -------------- -------------- --------------
  Table 7.8---Information that                                  
  Talent Search projects tracked                                
  or monitored on program                                       
  participants                                                  

                                  Yes, for all   Yes, for some  Not for any
                                  participants   participants   participants

  Percentage of projects that                                   
  tracked or monitored:                                         

  High school graduation          94%            6%             0%

  Year-to-year progression        91             8              2
  through high school                                           

  Enrollment in college           83             14             3

  Completion of college           74             22             4
  applications                                                  

  Contact hours or participation  70             13             17
  in program                                                    

  Grades                          65             26             10

  Course selection of             35             42             23
  participants                                                  

  Graduation from college         15             26             59

  Percentage of projects that                                   
  tracked or monitored:                                         

  All eight of the above          26                            

  Seven of the above              37                            

  Six of the above                24                            

  Five of the above               9                             

  Four or fewer of the above      3                             
  ------------------------------- -------------- -------------- --------------

> [Source:]{.smallcaps} National Survey of Talent Search Projects,
> 1999--2000.

A third survey question asked not just about the types of participant
data that projects collected, but also about how they maintained it---on
paper, in a computer database, or both. Virtually all projects (97 to 99
percent) maintained records on active participants' demographic
characteristics and the services they received---information that is
critical for completing the APR; the vast majority maintained both paper
and computerized records (table 7.9). In contrast, less than two-thirds
of projects maintained participants' scores on college entrance
examinations (ACT or SAT) or other standardized tests, and a majority of
those that did maintain such records kept only hard-copy records.
Projects often maintained records both on paper and in a computerized
format, but there were a few items (including transcripts and career
survey results) that were much more likely to be maintained on paper
only. Very small percentages of projects reported maintaining the
various types of information we asked about in a computer database only.

  -------------------------- ------------ ------------ ------------- ------------
  Table 7.9---How Talent                                             
  Search projects maintained                                         
  data on active                                                     
  participants                                                       

                             Maintained   Maintained   Maintained    Not
                             on paper     in a         both on paper maintained
                             only         computer     and in a      in either
                                          database     computer      form
                                          only         database      

  Records of services        26%          2%           70%           1%
  received                                                           

  Individual participant     30           2            67            2
  contact sheets                                                     

  Demographic information    10           1            86            3

  Project's assessment       46           2            38            14
  records                                                            

  High school or             64           1            20            15
  postsecondary transcripts                                          

  College or postsecondary   29           5            50            16
  school enrollment                                                  

  Career-survey results      56           2            20            21

  Recommendations or         63           1            12            24
  commendations                                                      

  Follow-up data on former   29           3            41            27
  participants                                                       

  Financial aid applications 42           3            27            28

  College or postsecondary   42           4            23            31
  school applications                                                

  ACT scores                 37           3            25            35

  SAT scores                 36           3            21            40

  Other standardized test    39           3            17            40
  scores                                                             

  Diagnostic test data       27           1            8             64

  Attitude scale profiles    21           1            5             73
  -------------------------- ------------ ------------ ------------- ------------

[Source:]{.smallcaps} National Survey of Talent Search Projects,
1999--2000.

A task closely related to data collection and record keeping is project
evaluation, and it is a task that all projects must undertake in one way
or another. Evaluation plans are one of the dimensions on which Talent
Search grant applications are scored. Information on participant
outcomes is obviously central to evaluating project effectiveness, but
information on project processes is also needed so that staff can
consider whether and where they may need to make improvements. Three
project survey questions addressed evaluation practices.

Most evaluations of Talent Search projects are internal evaluations. At
the time of the project survey, fewer than half (45 percent) of all
Talent Search projects had undergone an external evaluation (figure
7.1). Projects hosted by community organizations were somewhat more
likely than those hosted by postsecondary institutions to have undergone
an external evaluation. A previous review of 31 recent Talent Search
grant applications found that some external evaluations were to be
conducted by advisory committees composed of community members and that
sometimes professionals from other Talent Search projects participated
(Silva and Kim 1999).

Figure 7.1---Percentage of Talent Search projects that have had an
external evaluation conducted

![](media/image1.wmf){width="5.898611111111111in"
height="3.172222222222222in"}

[Source:]{.smallcaps} National Survey of Talent Search Projects,
1999--2000.

Overall, more than 90 percent of Talent Search projects reported that
their evaluations involved an ongoing assessment of program operation
and success---a type of assessment that is sometimes called a formative
evaluation (table 7.10). Grant applications commonly described plans
calling for primary staff to meet on a regular basis during the year to
assess a project's progress. Case studies confirmed that project staff
did in fact meet regularly to address a wide range of operational
issues, from recruitment and target school relations to positive and
negative aspects of particular activities. In addition, over 60 percent
of all projects reported that their evaluations involved a comprehensive
year-end study, sometimes called a summative evaluation. Such a study
provided an opportunity to make final determinations about project
success in meeting process and outcome objectives.[^12] There was almost
no variation in choice of evaluation by type of host institution.
Response patterns revealed that about 60 percent of projects undertook
both formative and summative evaluations and that about 30 percent
relied on only an ongoing (formative) assessment.

  --------------------------- ---------- ------------- --------- -------- -----------
  Table 7.10---Types of                                                   
  evaluations performed for                                               
  Talent Search projects                                                  

                                         Host                             
                                         institution                      

                              All        Public 4-year Private   2-year   Community
                              projects                 4-year             org.

  Percentage of projects                                                  
  using:                                                                  

  Ongoing assessment of       92%        89%           94%       90%      97%
  program operation and                                                   
  success                                                                 

  Comprehensive year-end      63         63            58        65       63
  study                                                                   

  Percentage of projects                                                  
  using:                                                                  

  Ongoing assessment only     30         27            38        28       35

  Year-end study only         2          1             0         3        2

  Both of the above           61         61            56        61       62

  Neither of the above        7          10            6         7        2
  --------------------------- ---------- ------------- --------- -------- -----------

[Source:]{.smallcaps} National Survey of Talent Search Projects,
1999--2000.

What types of information were used to evaluate projects' success in
meeting their goals and objectives? Of the 11 types of information that
our project survey asked about, the most common response was an analysis
of participants' school retention or graduation rates (table 7.11). This
result is not surprising in that these rates pertain to major outcome
objectives. The second- and third-most common categories of information
were written evaluations prepared by, respectively, students and staff.
One case study project, for example, distributed feedback forms with
open-ended questions after some events (such as workshops, college
visits, and other field trips) and used year-end evaluation forms that
asked students to rate the program on various dimensions. The least
commonly used type of information was a comparison of standardized test
scores for participants and nonparticipants (18 percent). Overall, about
15 percent of Talent Search projects used eight or more types of
information, but about 26 percent used three or fewer, with the mean at
5.

There were a few cases of notable (but nonsystematic) variation by type
of host institution. For example, projects hosted by community-based
organizations were much more likely than other projects to (1) follow-up
on participants who left the program but remained in school and (2)
compare the retention rates of participants with those of
nonparticipants. However, we have no insights to explain the differences
in the information collected.

  -------------------------------- ---------- ------------- --------- -------- -----------
  Table 7.11---Information used to                                             
  evaluate Talent Search projects'                                             
  success in meeting their goals                                               
  and objectives                                                               

                                              Host                             
                                              institution                      

                                   All        Public 4-year Private   2-year   Community
                                   projects                 4-year             org.

  Percentage of projects using:                                                

  Analysis of school retention or  94%        96%           86%       95%      92%
  graduation rates for those                                                   
  served                                                                       

  Written student evaluations of   82         84            83        85       76
  services                                                                     

  Written staff evaluations of     65         70            73        56       65
  project procedures                                                           

  Analysis of course completion    48         56            46        44       45
  rates for those served                                                       

  Analysis of retention rates for  40         38            30        38       52
  those served compared to                                                     
  nonparticipants                                                              

  Analysis of standardized test    38         40            41        32       43
  scores for those served                                                      

  Follow-up of those who left the  28         28            22        25       38
  program and the school                                                       

  External evaluations^a^          32         30            35        32       35

  Follow-up of those who left the  31         30            24        27       43
  program but remained in school                                               

  Analysis of course completion    22         18            19        22       28
  rates for those served compared                                              
  to nonparticipants                                                           

  Analysis of standardized test    18         18            19        17       20
  scores for those served compared                                             
  to nonparticipants                                                           

  Percentage of projects using:                                                

  10 to 11 of the above            6          6             6         5        7

  Eight to nine of the above       9          9             3         8        14

  Six to seven of the above        20         16            25        19       26

  Four to five of the above        39         47            42        35       33

  Two to three of the above        23         21            22        30       14

  None or one of the above         3          1             3         3        7
  -------------------------------- ---------- ------------- --------- -------- -----------

[Source:]{.smallcaps} National Survey of Talent Search Projects,
1999--2000.

^a^Respondents who said "yes" to this item were asked to specify the
type of external evaluation. Respondents most often mentioned feedback
and assessment from target school staff, followed by feedback and
assessment from the host institution.

The preceding tables and narrative may give the impression that many
Talent Search projects maintain a great deal of data on program
participants and use it extensively to assess program effectiveness. But
the situation is not that clear. For example, projects that reported
that they had attempted to measure various outcomes may not have
succeeded in measuring the outcomes, or may have stopped due to
associated costs or difficulties. In addition, the project survey did
not collect information on how projects selected nonparticipants for
comparison with participants.

The case studies provided a different perspective on data collection,
record keeping, and program evaluation. The overall impression that
emerged was that these projects did not place much emphasis on data
collection and analysis. Although most case study project staff had
fairly firm beliefs about the ways in which their participants benefited
from Talent Search, they did not have objective data that would support
those beliefs. The projects focused their record keeping and data
analysis on the elements required for the APR, such as high school
graduation and other major outcome objectives. By and large, projects
did not collect data on the other outcomes they claimed their
participants were achieving. A project with a major emphasis on
tutoring, for example, collected no objective information, such as
course grades, on students' academic performance. Similarly, a project
with a substantial test preparation component for high school students
did not collect data on students' SAT or ACT scores.

Projects also did not use a rigorous approach to evaluation. None of the
14 case study sites systematically compared outcomes for program
participants with a matched group of nonparticipants. Only one of 31
previously reviewed Talent Search grant applications mentioned a plan to
conduct such an evaluation (Silva and Kim 1999).

What might account for projects' practices regarding data collection and
evaluation? First, projects indicated they did not have sufficient
resources---such as time, funds, or expertise---to mount a serious
data-collection and analysis effort. As it was, staff in some projects
already spent one day each week engaged in paperwork, such as recording
which students had received which services that week. More time on
administrative tasks would have meant less time in the field working
with participants. Second, staff apparently operated under the
assumption that participants would stop using project services if such
services did not meet their needs. In other words, why track students'
grades to measure the effectiveness of a tutoring component when
voluntary participation rates in tutoring suggest that the service must
be producing the desired outcome? Third, project staff seemed to feel
that they were close enough to their participants and key target-school
staff to obtain a well-informed view of how students were benefiting
from the program. Fourth, in some cases, projects might not have had
easy access to needed data sources such as student transcripts.

Overall, evidence suggests that data collection and record-keeping
appeared to be among the greatest challenges faced by some Talent Search
projects. When the survey asked directors to list problematic aspects of
their projects, *collecting and maintaining student records* and
*student tracking and follow-up* were among the more frequently cited
problems.

[^1]: Outcome objectives differ from process objectives, which address
    steps that projects take in operating a program and serving
    participants. Talent Search projects are also required to set goals
    for two process objectives: the percentage of "college-ready"
    participants who will receive assistance in applying for
    postsecondary admission and the percentage of college-ready
    participants who will receive assistance in applying for financial
    aid. Providing assistance with applications is clearly a process
    undertaken by project staff, not an outcome experienced by
    participants as a result of their involvement in the program.

[^2]: Grade-level progression differs slightly from secondary school
    retention; the former refers to moving ahead (not repeating a
    grade), whereas the latter refers to staying in school (regardless
    of grade level).

[^3]: APR and survey data both represent projects' self-reported outcome
    data. Neither source has been independently verified. In addition,
    neither source compares outcomes for participants with those of
    similar nonparticipants.

[^4]: The project survey found that the graduation rate for 12th grade
    participants in 1998--99 was 91 percent overall, with little
    variability by type of host.

[^5]: These percentages cannot be compared directly with any outcome
    goals. The related goals set by projects pertain to the percentage
    of "college-ready" participants who will receive assistance with
    applications. But goals for providing assistance are process
    objectives, not outcome objectives. Furthermore, there is a
    difference between receiving assistance with an application process
    and actually applying.

[^6]: The desirable long-term outcome, of course, is that they
    subsequently enroll in a postsecondary education program.

[^7]: Only 37 private 4-year projects responded to the initial GED
    question; as a result, the subsequent means and percentages
    presented in table 7.6 are based on just 13 to 15 cases.

[^8]: Phase II of the National Evaluation of Talent Search will attempt
    to make such comparisons by using data from a handful states. For a
    description of the planned research, see Maxfield et al., 2000.

[^9]: The maximum composite score on the ACT is 36.

[^10]: A major purpose of these questions, and one discussed below on
    record-keeping, was to help assess the feasibility of using project
    data in an impact analysis during Phase II of the National
    Evaluation of Talent Search. The implementation study did not set
    out to describe projects' data-collection efforts. For an earlier
    detailed look at seven projects' data-collection efforts, see
    Decision Information Resources, 1994.

[^11]: Given that all Talent Search projects are required to report in
    their APRs the number of participants who applied for financial aid
    and the number who applied for postsecondary admission, it is
    unclear why the results were so far below 100 percent. In a related
    survey item, 96 percent of respondents reported monitoring or
    tracking college application completion (table 7.8).

[^12]: In addition, about 15 percent of projects indicated that some
    other type of evaluation was performed for their programs, but the
    responses did not provide any clear insight into the nature of the
    evaluations. Finally, about 3 percent of projects reported no
    evaluations undertaken for their programs.