# Presentation: 343160

## Education Longitudinal Study of 2002 

**Education Longitudinal Study of 2002**** **

**NSF Summer Data Policy Institute**

**July 22, 2007**

**Notes:**
Hello.

## Objectives of ELS: 2002

- Monitor the critical transitions made by a cohort of high school sophomores (10th graders) through college into their adult careers
- Maintain comparability with NLS-72, HS&B, and NELS:88 to measure trends in transitions and outcomes over time
- Link the assessment of achievement outcomes to other high school assessments (NAEP and PISA) for purposes of analysis
- Focus 2nd follow-up on issues of college access and choice

**Notes:**
Monitor the critical transitions made by a cohort of high school sophomores (10th graders) 
     Key term:    
        “critical transitions” means dropping out, attaining different levels of 
        education, forming a family
An important goal in the design of ELS was to maintain comparability with 
previous longitudinal studies
     To measure trends in critical transitions over time
       -  requires repetition of of questions over waves
       -  requires consistency in timing of follow-ups
The assessments included in ELS are linked other assessments for 
purposes of comparison. Linked to:
        - To earlier high school longitudinal surveys (equated to NELS: 88)
        - To NAEP (equated)     - To PISA  (concordance)

FOCUS OF 2nd FOLLOW-UP IS ON ISSUES OF COLLEGE ACCESS AND 
CHOICE
      - Have always had some data on college transition
      - ELS F2 has much more.

## ELS: 2002 Basic Survey Design

- Stage 1: School selection
- Target population: schools with 10th graders
- Probability of selection proportional to school size
- Stratification: region x  urbanicity x school control
- Total strata: 96
- Over sampled: Catholic schools, and other private
- Final sample size: 752 responding schools with a 10th grade
- Similar to NELS:88 but starts in spring of 10th grade

**Notes:**
ELS has a two-stage sample design  (as do many longitudinal surveys)
    - First schools are selected, then students within those schools

In Stage 1, target population is schools with 10th graders

Schools are selected with a probability of selection proportional to their enrollment size
    This is a key aspect of the sample design

The sample was stratified by region(8 Census), urbancity (3), school control (3).
    o for total sampling strata =  96

Oversampled: Catholic schools and other private schools

Final sample size:  752 responding schools with a 10th grade
        - 580 public, 172 private

## ELS: 2002 Basic Survey Design

- Stage 2: Student selection
- Random selection of about 26 students per school
- From student rosters provided by school
- Number of students per school was limited to minimize burden
- Rosters organized by four race/ethnicity groups
- Over-sampled: Asians/Pacific Islanders
- All ELL and special education students included (expanded sample)
- Sample size:  17,591 10th grade students selected representing 3.6 million students

**Notes:**
In stage 2, about 26 students per school are selected from rosters provided by the schools.
They are selected within the school, not by classroom.

The number of students per school was limited to minimize burden selecting of multiple students per school, but..

It also means that when combined with the PPS selection of schools, the result is  that students across the country are selected with roughly equal probability.

This sample design also means that ELS sample is CLUSTERED, that is, even though individual students are randomly selected, they are more likely to be similar to each other than if they were randomly selected from across the country.
  -  This has important implications for estimating the standard errors of estimates

In this process, Asians/Pacific Islanders are oversampled

Base Year sample size, 17,591 10th grade students.

## ELS: 2002 Basic Survey Design

- Contextual data sources:
- One parent per student
- Two teachers per student
- One a mathematics teacher of the student
- One an English/language arts teacher
- Principal of the school
- School characteristics
- School librarian/media center director
- Planned follow-ups:
- First follow-up, high school seniors (2004) (*done*)
- Second follow-up, college sophomores (2006)
- Third follow-up, eight years after high school (2012)

**Notes:**
Other sample members:   (don’t read bullets—or refer to previous slide)
    o One parent per student
         - Specification was the parent with the most knowledge about the 
           child’s education
         - For this reason, the sample of parents is not representative of the 
           parents of high school students
    o Two teachers per student
         - Students are randomly selected from the school
         - One is a mathematics teacher of the student
         - The other an English/language arts teacher of the student.
         - The teachers selected are not representative of teachers
         - Maximum number of students per teacher about 6
     o Principal of the school
         - Information about school characteristics is given by the principal of school
         - For example, the principal reports on,
             + type of school, crime level in the neighborhood,
             + percentage of students in college prep, voced, special ed, alternative, 
                bilingual education
             + types of curriculum standards and tests in use,
             + who influences assignment of students to course/programs
             + and so on.
Will describe what’s in the follow-ups later

## Sample Design and Weights

- Schools and students are assigned weights according to their probabilities of selection.
- Use of these weights in analyses is necessary to obtain accurate population estimates.
- Values of the weights for school and student sample members are inversely proportional to their probabilities of selection.

**Notes:**
Schools and students are assigned weights according to their probabilities of selection.

Use of these weights is necessary in analyses to obtain accurate population estimates.

Basic Principle for assigning weights: 

  - The value of the weights assigned for each individual school and 
     student member of the ELS samples are inversely proportional to 
     their probabilities of selection.
  - As indeed they are in any sample survey.

  - But I think it helps to know what the weights stand for.

  - They stand for the number of individuals in the population that each 
     sample member represents.

  - So, a weight of 400 means that the sample member in question 
    represents 400 other individuals in the population.

## Sample Design and Weights

- School Weights –
- E.g., Stratum H in the Pacific region contains 80 public, urban, schools
- If two schools are chosen (H1 and H2), the school weight for each one is 80/2=40
- So each sampled school in stratum H represents 40 schools with a 10th grade.

**Notes:**
Here is how school weights are assigned – 

Suppose one of the strata, call it Stratum H, which is in the in the Pacific region,  contains 80 public/urban/schools

If two schools are chosen (H1 and H2) in this stratum, the school weight for each one is 80/2=40

So each sampled school in stratum H represents 40 schools with a 10th grade.

The actual process is a little more complicated than this in order to achieve a probability of selection proportional to size but this simple example shows the basics.

## Sample Design and Weights

- Student weights –
- Roster of all 10th graders enrolled in spring 2004 was provided by the selected school
- If school H1 had 120 sophomores and 24 were selected, then the “within school weight” for each student would be 120/24 = 5.
- The overall weight for these students would be 40 x 5 = 200.

**Notes:**
Similarly for the student weights – 

They are assigned in the following manner:
   o  Each selected school is asked to provide a roster of all 10th graders
         enrolled in spring 2004
o  Suppose the roster for school H1 had 120 sophomores.
o  Suppose also that 24 of these students were randomly selected 
      from this roster for inclusion in the study.
o  Then the “student weight” for each student would be 120/24 = 5.
o  AND, the overall weight for each of these studenets would be 
       40 x 5 = 200.
o  There were any Asian students in this school, they would be selected 
    with a somewhat higher probability of selection and their student 
     weights would be accordingly lowered a small amount.

## Sample Design and Weights

- Base Year student weights:
- The result is a Base Year _*student weight*_ for every individual in the sample corresponding to the number of individuals in the population that person represents.
- The distribution of these weights is:

| Statistics | Base Year student weight, BYSTUWT |
| --- | --- |
| Mean | 224 |
| Median | 205 |
| Quartiles | 99, 205, 300 |
| Range | 5-978 |

**Notes:**
The result of this process is a weight for every individual in the sample corresponding to the number of individuals in the population that that person represents.

The distribution of these Base Year student weights is shown in the table on the screen.

All of these weights for each student are included in the ELS data files.

## Sample Design and Weights

- Base Year school weights:
- For purposes of analyzing schools, there is also a _*school weight*_* *for the Base Year schools.
- The distribution of these school weights is:

| Statistics | Base Year school weight, BYSCHWT |
| --- | --- |
| Mean | 32.7 |
| Median | 17.0 |
| Quartiles | 7.4,17.0,27.4 |
| Range | 0-395.7 |

**Notes:**
For purposes of analyzing schools, all of the base year school weights are also included in a separate file on the ELS CDRom.

These weights are potentially useful for analyzing differences among schools.

Do not REALLY NEED these weights because the school information is contextualized in student record

## Sample Design and Weights

- Between the Base Year and First Follow-up,
- Some student do not respond, and
- Some students migrate to new schools
- In ELS, the sample weights are adjusted to account for for sample member non-response
- Using model-based methods
- The total weight for Base Year cohort does not change.
- All students who transferred to another school were included in the First Follow-up data collection

**Notes:**
Now the situation gets more interesting.

Between the Base Year and First Follow-up, the student sample changes in two ways:
    o  Some students are nonrespondents; they don’t answer the questionnaire no 
        matter how hard we try.
    o  Some students move from their Base Year school to a new school, or 
        “transfer school,” as we call it.

In ELS, the sample weights are adjusted to account for sample member non-response.
    o ...using model-based methods
    o Consequently non-respondents disappear from the sample, but their place is 
       taken  by students like them who remain in the sample
    o The adjustment is done so that the weights are assigned according to propensity 
       and demographic  background variables.
        -- these adjusted weights are then trimmed

## ELS: 2002 Sample Distribution

| Change in the size of the ELS:2002 sample of schools and students, by data collection stage (unweighted) |  |  |
| --- | --- | --- |
|  | Base Year | First Follow-up |
| Original school |  |  |
|    Total number of schools | 752 | 747 |
|    Total number of  students | 16,202 | 13,988 |
|    Average number students per school | 21.5 | 17.8 |
| Transfer school |  |  |
|    Total number of schools | --- | 1,130 |
|    Total number of  students | --- | 1,248 |
|    Average number students per school | --- | 1.2 |

- Source: ELS 2002 data file

**Notes:**
The movement of students to other schools, means that the average number of students per school declines, but not by much, as shown in this table.
     - You can see that the average number of student per school declines from 
        21.5 students per school to about 17.8
     - The total number of students in the Base Year schools remained about the 
        same as in the Base Year
     - About 1,000 students moved to another school.
     - On average, there were only 1.2 Base Year students per school in these other, 
        or “TRANSFER,” schools.

     - Notice that the total number of students in the sample declined from 16, 202 
        to 15,118 (must add 13,988 and 1,248)
     - This was primarily due to the non-response of students who were respondents 
        in the Base Year that I mentioned.
     - However, some of this loss was replaced by 651 BY non-respondents who 
       responded in F1, which I will show you later.
     - Also, the “freshening” of the senior cohort added 171 students to the sample, 
        which I will also talk about later.

## Sample Design and Weights

- Additional measures to correct for non-response:
- Non-respondents are included in follow-ups
- Receive special questionnaires to obtain data otherwise collected in previous follow-up
- Universe variable, F1UNIV1, shows the F1 respondents who were non-respondents
- All of the main student background variables (sex, race, SES, etc.) and some status variables are imputed

**Notes:**
Some additional measures are taken in ELS to correct for student non-response.

One is that non-respondents are included in follow-ups:
   o This  increases sample sizes and helps to avoids bias across waves of the 
      survey.
   o 651 BY non-respondents in ELS became respondents in F1.
   o In order to fill-in information that these respondents 
      would have provided in the earlier follow-up, they are asked 
      to complete special questionnaires to provide this information
   o An IMPORTANT MESSAGE IS: The UNIVERSE VARIABLES show who 
      these non-respondents became respondents are.
   o I will illustrate this later.

Another step is that all of the main student background variables are imputed
   o using the information obtained from non-respondents who respond
   o using logical imputation
   o using model-based imputation as necessary
   o as little statistical imputation is done as needed
   o very little is needed for most variables.

## Sample Design and Weights

- After imputation and non-response adjustment, a _*panel weight*_*, *F1PNLWT,* *is formed from sample members for whom there is response data in both the Base Year and First Follow-up.
- This panel weight should be used whenever change  in a variable from BY to F1 is analyzed.
- This panel weight is zero for all students for whom BY-F1 information is not available
- The sample of cases still represents the distribution of the original Base Year Cohort

**Notes:**
After imputation and non-response adjustment, a PANEL WEIGHT is formed from sample members for whom both background data and data for key F1 status variable are available.

This panel weight should be used in all analyses

This panel weight is zero for all students for whom BY-F1 information is not available

The sample of cases still represents the distribution of the original Base Year Cohort

The number of cases included is smaller than the number with a Base Year weight but not by much.

## Sample Design and Weights

- For all students who completed a First Follow-up questionnaire, there is also a  _questionnaire weight_, F1QWT
- F1QWT includes the freshened sample and all other students in Base Year or Transfer schools who completed a First Follow-up questionnaire, including dropouts, etc.
- The cross-sectional sample of seniors is formed by crossing F1QWT with a cohort flag, G12COHRT, identifying all in-school seniors in the First Follow-up.

**Notes:**
For all students who completed a First Follow-up questionnaire there is a sample weight, F1QWT.
    o F1QWT includes the freshened sample
    o It also includes all other students in BY or F1 schools who completed a 
       questionnaire
    o All students, in or out of school are included.
    o The CROSS-SECTIONAL SAMPLE is formed by crossing F1QWT with a
        cohort flag, G12COHRT.
    o This flag identifies all in-school seniors in the First Follow-up.

## ELS Analysis Populations  (E4P)

| Sample | Population | Weight |
| --- | --- | --- |
| Base year students | Cross-section of sophomores | BYSTUWT |
| Expanded sample | Cross-section of sophomores, including ineligible students | BYEXPWT
(restricted) |
| First Follow-up | Cross-section of seniors (freshened)  | F1QWT  x
G12COHRT  |
| Longitudinal panel | Same group of sophomores followed for 2 years | F1PNLWT |
| Longitudinal panel | Same group of seniors followed for 4 years | F2PNLWT |
| Base year schools | Schools with 10th grade | BYSCHWT  |

**Notes:**
This chart shows the main analysis samples that are configured within the ELS sample design and the weights that are associated with each one.

They are:
    - Base Year students
    - The Expanded sample of students
    - The senior cohort (freshened sample)
    - the BY-F1 longitudinal panel 
         + students for which there is data (included imputed data) for both years

    - Also, there will an F1 to F2 longitudinal panel in the F2 data that will be released in about a year.

## Design Effects and Standard Errors

- Because of the selection of multiple students per school, the ELS sample is “clustered”
- Because of this clustering, the standard errors of population estimates are larger than they would be with simple random sampling (SRS)
- The extent of departure is measured by the “Design Effect” or DEFF
- Because of the “DEFF,” special software must be used to correctly estimate standard errors

**Notes:**
Sample Weights and Design Effects

Because of the selection of multiple students per school, ELS sample is “clustered”

Because of this clustering, the standard errors of population estimates are larger than they would be with simple random sampling (SRS)

The extent of departure is measured by the “Design Effect” or DEFF

Because of this “Design Effect,” standard errors of estimated variables must be estimated with special software

If you are not already on board, some understanding of these “Design Effects” and their implications for how standard errors should be estimated...
Should be two of your takeaways from this session,.

## Design Effects and Standard Errors

- In general, DEFF is defined as the ratio of,
- the square of the “correct” standard error by
- the square of the standard error of the estimate assuming simple random sampling (SRS).
-        DEFF =  (correct S.E.2 / (SRS S.E.)2

**Notes:**

In general, the design effect is defined to be the ratio of the square of the calculated standard error of an estimate TAKING CLUSTERING INTO ACCOUNT, compared to what the standard error would have been WITH THE SAME WEIGHTS but simple random sampling: 

       DEFF = (Design-Std.Err.)**2 / (SRS-Std.Err.)**2
    or
       DEFT = (Design-Std.Err.) / SRS-Std.Err. = sqrt(DEFF)

These Design Effects are specific to each variable to be estimated from the survey. It is not a constant for all variables that can be estimated from the survey.
---------------------------------------------------------------------
If you look in the back of the ELS Data File Documentation you will see long tables of design effects for student and parent variables.

    For example, the DEFT for the percentage students whose mother was born in the U.S. is:
         - 1.33 for all students
         - 1.60 for females
         - 1.82 for males 

Some surveys publish DEFFs/DEFTs for long, long lists of variables and recommend using them to “adjust” standard error estimated assuming SRS, which can be done with most statistical software. 

## Design Effects and Standard Errors

- Mean and root design effects for NELS:88 and ELS:2002 sophomore cohort panel estimates

**Notes:**
In ELS:2002 the average DEFT for Base Year student variables is 2.35, which means that the average DEFT is 1.5

Or, the correct standard error is about 1.5 times the standard error estimated with accounting for the clustering of sample members within schools.

In NELS, the root design effect was 1.9.

The design effects indicate that the ELS sophomore panel is more efficient than the NELS sophomore panel.

Main reasons are thought to be: (1) more analysis strata, (2) model-based non-response adjustment.

## Survey Components

-      Base Year survey
- _Spring 2002 sophomores_
- Student questionnaire
- Student cognitive tests
- Mathematics
- Reading
- Parent questionnaire
- Teacher questionnaires
- Principal questionnaire
- (school information)
- Library/media questionnaire
- School facilities checklist
- Geocoded data available

-      First Follow-up survey
- _Spring 2004 seniors_
- Student questionnaire
- Transfer student quest.
- Dropout questionnaire
- Early graduate quest.
- Home school quest.
- New student quest.
- Student cognitive tests
- Mathematics
- Principal questionnaire
- (school update)
- High school transcripts

**Notes:**
Now let’s shift gears and look in more detail at the components of ELS: 2002.
Base Year survey  (spring 10th grade)
Student questionnaire ( 45 minutes)
Student cognitive tests (1/2 hour each plus a 15 minute routing test)
Mathematics   AND Reading
Parent questionnaire
Teacher questionnaires
Principal questionnaire (school information)
Including a question on the degree of influence difference school personnel have in assigning courses or programs.
Curriculum and testing standards
Percentage of students on FRPL, IEPs, etc.
Numbers of teachers, etc.
Library/media questionnaire
School facilities checklist
The school facilities checklist was completed by the survey interviewer, as he or she traveled around the school building rating it for,
   - the condition of the building, 
   - graffiti on the walls, 
   - trash in the rest rooms, and so on.

## ELS Results: Educational Expectations

- Bachelor’s degree

- Graduate or professional degree

- Percentage of 12th graders who expected to attain a bachelor’s degree or a graduate or professional degree: 1981-82, 1991-92, and 2003-04.

**16**

**44**

**51**

**32**

**63**

**67**

**64**

**86**

**86**

**Low SES**

**Middle SES**

**High SES**

**Notes:**

This chart takes advantage of the three most recent high school longitudinal studies supported by NCES.
      ELS: 2002
      NELS: 88, which Jeff will talk about in a minute.
      and
      High School and Beyond, class of 1982.

  Between the high school class of 1982 and 2004:

      o  The gap between the percentage of high and low SES students who expected to earn a bachelor’s, graduate, or professional degree declined from 48 percent in the high school class of 1982 (64 percent – 16 percent) to 19 percent (86 percent – 67 percent) for the class of 2004.

      o The gap between the percentage of high and low SES students who expect to earn a graduate or professional degree has declined by a smaller amount. In the class of 2004, high SES students were 31 percent more likely to expect to earn a graduate or professional degree than low SES students compared to 26 percent in the class of 1982.

## Survey Components

- Second Follow-up
- _Two years after high school_
- Young adult questionnaire
- High school completion update
- College choice
- College enrollment
- College major
- Employment (and military service)
- Life course (living situation, volunteer work, expectations
- Administrative records
- Federal student aid received
- SAT/ACT scores
- GEDs received

- Third Follow-up
- _Eight years after high school_
- Adult questionnaire
- College enrollment
- College completion status
- Employment
- Life course (living situation, volunteer work, expectations)
- College transcripts

**Notes:**
Survey Components for the Second and Third Follow-ups to ELS: 2002.
Data collection for the Second Follow-up is completed. Data in will be released in about a month.

Data collection for the Third Follow-up is scheduled to begin in six years or eight years after the sample members graduated from high school.

The Second Follow-up is new for NCES’s longitudinal studies.

Upon the recommendation of the TRP for ELS: 2002, the draft questionnaire was rewritten to focus on issues of access to higher education, while still collecting information from students who do not go onto college. 

Some of the college access information will be collected retrospectively; that is, students who are currently enrolled in college are being asked about: 
   o The specific colleges they applied to
   o The colleges that offered them aid and the types of aid they were offered
   o and, which college they chose to enter and why.

THERE ARE MULTIPLE TRACKS THROUGH the 2nd Follow-up questionnaire:
   o Standard enrollees
   o Delayers      o Leavers
   o Non-enrollees

Third Follow-up

  o Data collection will occur in 2011. 

## ELS High School Transcripts 

- Sample:
- Collected from Base Year school, and
- If in another school, from last school of attendance
- Provides a complete record of high school coursework
- Course titles, grades earned, credits earned, year taken, grades 9-12
- Coded according to course classification schemes
- Includes updated high school completion status
- Diploma or certificate of attendance
- GED
- Still in school

**Notes:**
Pass this over. Robert will talk about it.

## ELS High School Transcripts

- Ninth and eventual 12th grade mathematics course completed : 2004

- Percent completing specific course by end of  high school

- Among students who complete geometry in the ninth grade, 84% take Advanced mathematics (trigonometry-pre-calculus or calculus) by the 12th grade

- Among students who take general or applied mathematics in the ninth grade, 10% take Advanced mathematics

## Trend analysis with ELS, NELS, HS&B

| Survey and follow-up | High school sophomores | High school seniors | College sophomores | College Graduates |
| --- | --- | --- | --- | --- |
| ELS: 2002/12   3rd FU  |  |  |  | 2012 |
| ELS: 2002/06   2nd FU  |  |  | 2006 |  |
| ELS: 2002/04   1st FU  |  | 2004 |  |  |
| ELS: 2002   BY  | 2002 |  |  |  |
| NELS: 88/2000  4th FU  |  |  |  | 2000 |
| NELS: 88/2000  3rd FU |  |  | 1994 |  |
| NELS: 88/2000  2nd FU |  | 1992 |  |  |
| NELS: 88/90  1st FU  | 1990 |  |  |  |
| HS&B: 1980 Soph.  4th FU  |  |  |  | 1992 |
| HS&B: 1980 Soph.  2nd FU  |  |  | 1984 |  |
| HS&B: 1980 Soph.  1st FU  |  | 1982 |  |  |
| HS&B: 1980 Soph.  BY  | 1980 |  |  |  |

**CT**

**CT**

**HT**

**HT**

**HT**

**CT**

**Notes:**
This chart shows the three main high school longitudinal studies that have been created by NCES and all of the follow ups include in each one.
     - As you can see, the studies have followed at similar intervals (about 10 years) and included followups at the same points in time.

The regularity and consistency of data elements across these surveys and follow-ups 
is a strength of the NCES longitudinal studies. 

Because of this regularity and consistency it is possible to do several kinds of 
studies with the data:
   o Cross-sectional study (in one year)
   o Cross-sectional trend study
   o Longitudinal change study
   o Trend in longitudinal change study

The chart shows where transcripts are or will be available.
   o “CT” college transcript
   o “HT” high school transcript

## Availability of Restricted Data

- Restricted data files
- Use requires a licensing agreement with NCES
- “Restricted” files contain information not on the “Public Use” data files
- Linking variables
- Course records from student transcripts
- No top or bottom coding
- Released on CD-Roms with the same ECB and kinds of documentation as Public Use files.
- http://nces.ed.gov/pubsearch/licenses.asp

**Notes:**
NCES survey data files come in two forms: “Restricted” and “Public Use.”

The Restricted contain information that is not on the Public Use date file because it could potentially be used to identify individual sample members.

This information includes linking variables that are not on the Public Use data files, course records from student transcripts, more detailed race/ethnic classifications, and no top or bottom coding, which is used in the Public Use files to obscure sample member’s identities.

In order to obtain these Restricted data sets, users must obtain a licensing agreement with NCES. The instructions for doing so can be found on the NCES website at the address shown.

Or just type “Restricted” in the search box at the top of any NCES web site page. 

## Education Longitudinal Study: 2002

*For additional information:*
- Go to ELS: 2002 web site on the NCES web site:
-       nces.ed.gov/surveys/els2002
- To obtain Public Use data files:
-    click on “Order CDs” at the URL above
- Sign up for the “High school longitudinal studies” _News Flash_ on the NCES web site