This document responds to the U.S. Department of Education (USDOE)
questions regarding the District of Columbia (DC) growth model. When
appropriate, we refer to documents submitted alongside the original
proposal. Those include Appendix A, a white paper that describes the
growth model, and Appendix B, a white paper that describes an
implementation of the Levenshtein algorithm used for matching students
over time. The District of Columbia would be happy to provide additional
information or clarification for any of these questions or those that
may arise during the peer review process.

**[Principle 1: Universal Proficiency]{.underline}**

**USDOE Question #1:** Please provide additional information or examples
regarding how the growth would be attributed for students who transition
between schools (i.e., how scores of elementary students projected to be
proficient in middle school would be included.)

**DC response**: With status scores, the attribution of scores to a
specific school can be easily determined. However with growth, a policy
decision must be made as to what schools receives credit for the growth,
or in our case, the projected scores.

In the DC model, we form a projected score for each student. This
projection indicates the probability that student *i* in grade *g*
attending school *j* will be proficient the next school year. A
particular complexity arises when a probability is formed for, say, an
elementary student that will transition into a middle school. The policy
question is whether the elementary school or the middle school should
receive credit for this projection.

For example, assume an elementary school configuration is K to 5 and the
middle school configuration is 6 to 8. From a purely technical
viewpoint, the statistical model assigns a success probability (i.e.,
the probability that the student will be proficient) to all grade 5
students denoting the likelihood that they will be proficient in grade
6.

However, the likelihood of grade 6 proficiency conditional on grade 5
performance and school attended is a function of the middle school's
instructional program and not a function of the elementary school's
effectiveness. Therefore, it would be unfair and technically unsound to
credit the elementary school for what occurs within a middle school.

Given this issue, DC plans to use a conjunctive accountability system
that will only allow a school to make AYP if they meet two conditions.
The conditions are as follows:

-   the projected scores for the "inner" or non-transitional grades plus

-   the NCLB status scores for the "highest" or transitional grade must
    be equal to or greater than the AMO for the current year.

Assume a school is a K to 5 school and tests are administered to
students in grades 3, 4, and 5. The "inner" grades are grades 3 and 4
and the highest grade is grade 5. In order to make AYP under our
proposed growth design, the number of students projected to be
proficient in the inner grades (i.e., grades 3 and 4) plus the number of
students scoring at or above proficient in the highest grade (i.e.,
grade 5) must be equal to or higher than the AMO.

For example, assume the AMO is 50% for reading. Sample data for the
example are provided in the table below. This table shows that of the 33
students currently in grade 3, 15 are projected to be proficient in
grade 4. Of the 33 students in grade 4, 20 are projected to be
proficient in grade 5. Of the 34 students in grade 5, 19 have currently
scored at or above proficient on the test. The total number of students
either projected or at or above is 54, or 54%. Because 54% \> 50% (the
AMO) then this school would be considered as making AYP under growth.

  --------------- --------------------------------------------- ----------
  **Grade**       **Projected/At or Above**                     **N**

  3               15                                            33

  4               20                                            33

  5               19                                            34

  **Total**       **54**                                        **100**
  --------------- --------------------------------------------- ----------

This model is conjunctive because it faces a problem that regular status
models do not face. If we were to only include the inner grades that
would systematically exclude the highest grade in the school. If we were
to use the grade 5 to 6 projections, that would be inherently unfair as
it would credit (or punish) the elementary school for what occurs in the
middle school. Our view is that this method ends up including all grades
within a school and only uses the growth model estimates in a way that
credits the school for what it may be responsible for and not what
another school may be responsible for.

**USDOE Question #2**: Please provide additional examples of how AYP
will be calculated for 2-3 schools with the addition of the growth
model. Specifically, include information regarding how the probabilities
of individual students becoming proficient are aggregated and impact the
of the school random effect.

**DC Response**: Appendix A attached to the original proposal outlines
an assumption (Assumption 3) that indicates that some schools are more
effective in terms of growth than others. The addition of the random
effect in the linear predictor has the effect of shifting the logistic
curve to the right or to the left of the state curve, which will give
very different success probabilities depending on which school a student
attends.

The graph below shows the probability of attaining proficiency in grade
4 mathematics conditional on grade 3 performance and the school
attended. The curve in the middle is the statewide conditional
probability. The curve furthest to the left is the school that is the
most effective school in the District and the school furthest to the
right is the least effective school in the District. Scores on the
x-axis are centered on the proficient cutscore so a score of 0
represents a scaled score of 360.

As can be seen in this scenario, assumption 3 is met given that there is
apparent variability in the probability of proficiency depending on
which school the student attends. For example, if a student has a score
of 0 in grade 3 (centered, so this is actually a score of 360), and he
attends the lowest performing school in the District, the probability
that the student will be proficient is less than 20%. However, if a
student has a score of 0 but attends the highest performing school in
the state, the student would have about a 90% probability of being
proficient. If the school random effect were ignored and not
incorporated into the model, all students with a grade 3 score of 0
would be assigned the state average probability, which is about 60%.

Clearly, this is unfair. In doing so, students in low performing schools
would be assigned success probabilities that are much higher than are
likely to occur in practice and, at the same time, high performing
schools would be punished as their students would be assigned success
probabilities that are lower than would be likely to occur in practice.
The random effect accounts for differences in the instructional programs
across schools, which in turn affects the probability of future success.

![](media/image1.png){width="3.3756944444444446in"
height="3.370138888888889in"}

USDOE has also asked how the probabilities of individual students are
aggregated. The method by which this is done is provided by equation 5
for the school overall and equation 6 for the subgroups in Appendix A.
Because probabilities can be used to form expectations, the
probabilities can be summed across students within schools or within
subgroups to form these expectations.

For instance, assume a school has only five students each with the
following probabilities: Student 1 = .32, Student 2 = .70, Student 3 =
.43, Student 4 = .86, Student 5 = .54. The sum of these probabilities is
2.85. This is also what statisticians refer to as an expectation.
Consequently, we would denote that 2.85 out of 5 students are projected
to be proficient next year.

USDOE has also asked for 2 to 3 examples of how this model will be
applied to schools. We extend the example given in our answer to
question #1 to show how this would apply to two different school
configurations: a k to 5 school and a 6 to 8 school. The method is
actually quite straight forward and simple to implement once the
probabilities are generated. The first table is for a K to 5 school with
students tested in grades 3 to 5 and the second table is an example of a
6 to 8 school with students tested in all grades. In the elementary
school, grades 3 and 4 are the inner grades and grade 5 is the highest
grade. In the middle school, grades 6 and 7 are the inner grades and
grade 8 is the highest grade.

The tables contain the same values because it is the concept we intend
to convey here and that concept doesn't depend on the values in the
cells. All computational details as well as a complete example are also
provided in Appendix A. In both tables, the values in the column
**Projected/At or Above** for the inner grades are the sums of the
individual probabilities as described above. For example, the first
table shows that 15 grade 3 students are expected to be proficient in
grade 4. The value in this same column for the highest grade (either
grade 5 or 8) is the actual number of students scoring at or above
proficient currently.

AYP decisions are made by computing the marginal sum of the
**Projected/At or Above** and comparing this to the total number of
students in the school. In this example, 54% of the students are either
on track or currently proficient. This percentage is compared to the
AMO.

  --------------- --------------------------------------------- ----------
  **Grade**       **Projected/At or Above**                     **N**

  3               15                                            33

  4               20                                            33

  5               19                                            34

  **Total**       **54**                                        **100**
  --------------- --------------------------------------------- ----------

  --------------- --------------------------------------------- ----------
  **Grade**       **Projected/At or Above**                     **N**

  6               15                                            33

  7               20                                            33

  8               19                                            34

  **Total**       **54**                                        **100**
  --------------- --------------------------------------------- ----------

**[Principle 2: Establishing Appropriate Growth Targets at the Student
Level.]{.underline}**

**USDOE Question #3**: How will the District of Columbia handle students
who are currently proficient but may fall below the proficient cut point
in the following year?

**DC Response**: Every student is included in our growth model analyses
whether they are currently scoring above proficient or whether they are
currently scoring below proficient. This is somewhat different than the
"status plus" approach which only includes students that score below
proficient and combines them with the number of students that are
currently scoring at or above proficient.

As discussed in the white paper (Appendix A) of our original proposal,
every student has a non-zero probability of being proficient in the
subsequent school year. As explained in the section on assumptions, and
is demonstrated in the data analysis, the likelihood of being proficient
in the subsequent school year depends on the student's prior year score.
Students with low scores in grade *g* are less likely to be proficient
than students with high scores in grade *g*. However, that probability
is never 1, although it may approach 1 depending on the student's school
(Assumption 3) and their prior score (Assumption 2).

Because we form a success probability for each student, irrespective of
their current proficiency level, we can simply apply the AYP estimation
methods outlined in section 4.2 of Appendix A. That is, the sum of the
individual probabilities forms the expected number of students likely to
be proficient in the subsequent school year. In this respect, these
students are treated in the same way as any other student in the data.

**[Principle 4: Inclusion of All Students]{.underline}**

**USDOE Question #4**: Please clarify how the growth model will factor
in students who are new, have missing data, or are unmatched.

**DC Response**: The DC growth model operates very differently from most
other models. While methodologically different, our proposed growth
model most resembles the philosophy (and in some respects the estimation
of the fixed effects) of the model used in both Tennessee and Ohio.

In the DC model (as in Tennessee and Ohio), estimates of the fixed
effects (Equation 1 in Appendix A) as well as the transformation of
log-odds to the probability scale are garnered from a prior cohort and
applied to the current cohort as illustrated in section 2.4---Generation
and Application of the Conditional Probabilities in Appendix A.
Consequently, a student only needs to have a single test score in order
for us to generate a probability -- there is no need to match records
for this student (for the current year).

In fact, a student is included in the growth model calculations for AYP
if they meet the following conditions:

-   The student has one test score on the DC-CAS and

-   The student meets our definition of full academic year

Our methodology allows for every student that is included in the status
calculations to be included in the growth calculations. If a student has
a missing score, then that indicates they were not tested in which case
they are also excluded from the status model calculations.

**USDOE Question #5**: Please provide further data regarding the impact
of low school-level match rates on the school level random effects.

**DC Response**: The school level random effects are what statisticians
refer to as *empirical bayes* estimates, or the best linear unbiased
predictors (BLUP). They are "best" in the sense that no other minimum
variance unbiased predictor exists. If a particular school were to have
a low level of matches, the random effect would "borrow strength" from
the data and shrink the estimates of the school random effects towards
the population means in order to correct for some of the apparent
unreliability that may occur as a function of the small number of
students used to estimate the random effects.

**USDOE Question #6**: Please clarify whether the growth model will be
applied to all students in every school in the state.

**DC response**: The growth model will only be applied to students in
grade 3 to 8. It will not be applied to students in high school. We
believe that it would be inappropriate to form high school projections
from the grade 8 data. It is statistically possible to generate the
probability that a grade 8 student will be proficient in grade 10. But
the chasm between grade 8 and 10 is large and forming probabilities over
such a large period of time ignores many important educational factors
that could change in the interim.

This does not indicate that high schools are excluded form the state
accountability system. It only means that these schools will not have
the additional benefit of using the growth model results, but they are
included in status and safe harbor calculations.

**USDOE Question #7**: Please clarify whether the growth model applies
to students taking the alternate assessment based on alternate
achievement standards.

**DC response**: The model does not currently include these students.

**[Principle 6: Tracking Student Progress]{.underline}**

**USDOE Question #8**: Please provide additional information on the
match rates of students by all subgroups included in AYP determinations
including economically disadvantaged, students with disabilities, and
limited English proficient.

**DC Response**: The match rates for English language learners, students
with disabilities, and students that are economically disadvantaged are
provided below. The match rates for all racial and ethnic subgroups were
provided in the original submission.

*Reading Tables*

  -------------------------------------------------------------------------------------------------------
  **Table of mergeflag by                                                                     
  SPECIAL_EDUCATION_STATUS2**                                                                 
  ----------------------------- ------------------------------------- ----------------------- -----------
  **mergeflag**                 **SPECIAL_EDUCATION_STATUS2(Special                           **Total**
                                Education Status)**                                           

  **Frequency\                  **N**                                 **Y**                   
  Col Pct**                                                                                   

  **0**                         1930\                                 241\                    2171
                                10.06                                 5.80                    

  **1**                         17263\                                3911\                   21174
                                89.94                                 94.20                   

  **Total**                     19193                                 4152                    23345
  -------------------------------------------------------------------------------------------------------

  -----------------------------------------------------------------------------------
  **Table of                                                              
  mergeflag by                                                            
  ECONOMY_STATUS2**                                                       
  ------------------- --------------------------- ----------------------- -----------
  **mergeflag**       **ECONOMY_STATUS2(Economy                           **Total**
                      Status)**                                           

  **Frequency\        **N**                       **Y**                   
  Col Pct**                                                               

  **0**               962\                        1209\                   2171
                      12.63                       7.69                    

  **1**               6656\                       14518\                  21174
                      87.37                       92.31                   

  **Total**           7618                        15727                   23345
  -----------------------------------------------------------------------------------

  ------------------------------------------------------------------------
  **Table of mergeflag                                         
  by ESL_STATUS2**                                             
  -------------------- ------------------- ------------------- -----------
  **mergeflag**        **ESL_STATUS2(ESL                       **Total**
                       Status)**                               

  **Frequency\         **N**               **Y**               
  Col Pct**                                                    

  **0**                1957\               214\                2171
                       8.92                15.22               

  **1**                19982\              1192\               21174
                       91.08               84.78               

  **Total**            21939               1406                23345
  ------------------------------------------------------------------------

*Mathematics Tables*

  -------------------------------------------------------------------------------------------------------
  **Table of mergeflag by                                                                     
  SPECIAL_EDUCATION_STATUS2**                                                                 
  ----------------------------- ------------------------------------- ----------------------- -----------
  **mergeflag**                 **SPECIAL_EDUCATION_STATUS2(Special                           **Total**
                                Education Status)**                                           

  **Frequency\                  **N**                                 **Y**                   
  Col Pct**                                                                                   

  **0**                         1941\                                 242\                    2183
                                10.11                                 5.83                    

  **1**                         17261\                                3911\                   21172
                                89.89                                 94.17                   

  **Total**                     19202                                 4153                    23355
  -------------------------------------------------------------------------------------------------------

  -----------------------------------------------------------------------------------
  **Table of                                                              
  mergeflag by                                                            
  ECONOMY_STATUS2**                                                       
  ------------------- --------------------------- ----------------------- -----------
  **mergeflag**       **ECONOMY_STATUS2(Economy                           **Total**
                      Status)**                                           

  **Frequency\        **N**                       **Y**                   
  Col Pct**                                                               

  **0**               965\                        1218\                   2183
                      12.66                       7.74                    

  **1**               6655\                       14517\                  21172
                      87.34                       92.26                   

  **Total**           7620                        15735                   23355
  -----------------------------------------------------------------------------------

  ------------------------------------------------------------------------
  **Table of mergeflag                                         
  by ESL_STATUS2**                                             
  -------------------- ------------------- ------------------- -----------
  **mergeflag**        **ESL_STATUS2(ESL                       **Total**
                       Status)**                               

  **Frequency\         **N**               **Y**               
  Col Pct**                                                    

  **0**                1957\               226\                2183
                       8.92                15.92               

  **1**                19978\              1194\               21172
                       91.08               84.08               

  **Total**            21935               1420                23355
  ------------------------------------------------------------------------

**USDOE Question #9**: Please clarify the universe for the match rates
reported on pages 13-17.

**DC Response**: We are unclear what is meant by "universe" for the
match rates. The total N sizes per subgroup are the marginal sums of the
columns in the tables provided on pages 13-17 of the original. For
example, the mathematics table showing the merge rates by ethnicity
indicates that 264 Asian students (A) are merged and 46 Asian students
could not be merged. The marginal sum is 310 as reported in the table.

If further information is needed DC would be happy to provide that
information.

**USDOE Question #10**: Please provide additional information on how the
current tracking system is accurate in matching student information
across multiple years.

**DC Response**: In the original proposal, Appendix B provides a
comprehensive study showing how the Levenshtein algorithm operates and
how it is applied to accurately match and merge student records over
time. We believe that DC is the only state education agency that has
achieved this level of accuracy to ensure accurate merging of student
records.

In addition to the study in Appendix B, a list of business rules were
provided in the original proposal narrative that describes how the
Levenshtein algorithm and our "salvage" efforts will be implemented. DC
would be happy to provide additional information as needed.

**USDOE Question #11**: Please provide additional information on the
match rates of proficient versus non-proficient students.

**DC Response**: These match rates are provided for reading and
mathematics in the tables below. The merge rates are better than 90% for
proficient and non-proficient students.

  ------------------------------------------------------------------------
  **Table of mergeflag by                                  
  rdpl2**                                                  
  -------------------------- -------------- -------------- ---------------
  **mergeflag**              **rdpl2**                     **Total**

  **Frequency\               **N**          **Y**          
  Col Pct**                                                

  **0**                      1241\          930\           2171
                             9.30           9.30           

  **1**                      12102\         9072\          21174
                             90.70          90.70          

  **Total**                  13343          10002          23345
  ------------------------------------------------------------------------

  ------------------------------------------------------------------------
  **Table of mergeflag by                                  
  mtpl2**                                                  
  --------------------------- -------------- ------------- ---------------
  **mergeflag**               **mtpl2**                    **Total**

  **Frequency\                **N**          **Y**         
  Col Pct**                                                

  **0**                       1278\          905\          2183
                              9.30           9.42          

  **1**                       12468\         8704\         21172
                              90.70          90.58         

  **Total**                   13746          9609          23355
  ------------------------------------------------------------------------

**USDOE Question #12**: Please provide additional information regarding
how the current data system collects information on student demographic
characteristics, disability status, and socio-economic status and how
this information will be used in reporting academic growth.

**DC Response**: The schools in the District of Columbia use one of two
student information systems. The public charter schools (PCS) all use a
system called OLAMS and the District of Columbia Public Schools (DCPS)
use a system called DC Stars. Both systems collect and maintain all the
demographic data required under NCLB including: race/ethnicity, gender,
disability status, ELL status, and economically disadvantaged status, as
well as daily attendance and enrollment status. In additions to annual
enrollment audits in October of each year, the state conducts periodic
audits to identify duplicate records and to identify any students
enrolled without a state approved USI.

In the fiscal year 2008 budget, the District of Columbia invested \$3
million to create an integrated state-wide longitudinal data warehouse.
Overall, this investment is \$19 million over five years in addition to
a three year \$5.7 million grant from the US Department of Education.
The first step toward this end was to create a new state quality control
system for reviewing, cleaning, and maintaining student records
including achievement and demographic data for all students in the
District. Phase I has been completed and includes a process for ensuring
the integrity of students' individual identifiers or unique student
identifiers (USI). This process eliminates any duplicate records or
incorrect IDs in the public charter school system (OLAMS) and the
District of Columbia Public School (DCPS) student information system.
The target for the new system is for the USI process to ensure a unique
identifier with greater than 98% accuracy.

However, the DC model is based only on student achievement scores. It
does not use student demographic or background characteristics to
produce predicted scores. The proposed model is based on students' most
recent scores at the same school to determine predicted scores.

**USDOE Question #13**: Please clarify how scores will be tracked across
schools and whether and how the growth trajectory follows students
across schools and LEAs.

**DC Response**: A growth trajectory is not estimated for each student.
We only estimate the conditional probability of future proficiency for
each student.