**Part B**

**Supporting Justification Request for**

**OMB Clearance of Information Collection Forms for:**

**The Effects of a Hybrid Secondary School Course in Algebra I**

**on Teaching Practices, Classroom Quality and Adolescent Learning**

***September 28, 2007***

Submitted to: Submitted by:

U.S. Department of Education REL Appalachia / CNAC

Institute of Education Sciences 4825 Mark Center Drive

555 New Jersey Ave., NW, Rm. 506 Alexandria, VA 22311

Washington, DC 20208 703-824-2828

202-219-1597

Project Officer: Project Director:

Sandra Garcia Linda Cavalluzzo

**CONTENTS**

### INTRODUCTION [5](#introduction) {#introduction-5 .TOC-1}

### 1. Respondent Universe and Sampling Procedures [8](#respondent-universe-and-sampling-procedures) {#respondent-universe-and-sampling-procedures-8 .TOC-1}

### 2. Statistical Methods for Sample Selection and Degree of Accuracy Needed [9](#number-range-chapterstatistical-methods-for-sample-selection-and-degree-of-accuracy-needed) {#statistical-methods-for-sample-selection-and-degree-of-accuracy-needed-9 .TOC-1}

### 3. Methods to Maximize response rates and Deal with Nonresponse [13](#methods-to-maximize-response-rates-and-deal-with-nonresponse) {#methods-to-maximize-response-rates-and-deal-with-nonresponse-13 .TOC-1}

### 4. Tests of Procedures or Methods to be Undertaken [14](#tests-of-procedures-or-methods-to-be-undertaken) {#tests-of-procedures-or-methods-to-be-undertaken-14 .TOC-1}

### 5. Individuals Involved [14](#individuals-involved) {#individuals-involved-14 .TOC-1}

###   {#section .TOC-1 .unnumbered}

### TABLES {#tables .TOC-1}

### Table 1. Schedule of Activities [7](#__RefHeading___Toc178749306) {#table-1.-schedule-of-activities-7 .TOC-1}

### Table 2. Potential Respondent Universe (*N*) and Treatment (T)/Control (C) Sample (*n*) and expected number of respondents (*R*)\* [9](#__RefHeading___Toc178749307) {#table-2.-potential-respondent-universe-n-and-treatment-tcontrol-c-sample-n-and-expected-number-of-respondents-r-9 .TOC-1}

### Table 3 Primary Hypotheses, Data Sources, Data Collection Procedures and Analyses [10](#__RefHeading___Toc178749308) {#table-3-primary-hypotheses-data-sources-data-collection-procedures-and-analyses-10 .TOC-1}

APPENDICES

### Appendix A [17](#__RefHeading___Toc178749313) {#appendix-a-17 .TOC-1}

Informational Meeting Schedule Template for Kentucky Educational
Cooperatives [17](#__RefHeading___Toc178749314)

**[The Effects of a Hybrid Secondary School Course in Algebra I on
Teaching Practices, Classroom Quality and Adolescent
Learning]{.smallcaps}**

### INTRODUCTION

This submission is a request for approval of data collection instruments
in support of an evaluation of an Algebra I intervention the applicant
proposes to implement in 60 high needs high schools located primarily in
rural areas across the Commonwealth of Kentucky (KY). The project is
sponsored by the Institute of Education Sciences within the US
Department of Education and will be conducted by the Regional
Educational Laboratory Appalachia (REL-A) administered by the CNA
Corporation (CNAC).

This intervention involves implementation of a hybrid instructional
approach in all high school Algebra I classrooms in intervention
schools. The hybrid approach combines online instruction with
face-to-face classroom instruction for students. Intervention teachers
will be supported by extensive and ongoing professional development
focused on implementing effective hybrid classes and on research-based
instructional practices for Algebra I.

Algebra I has emerged in recent years as a critical gatekeeper course,
necessary to prepare students for the rigorous mathematics curriculum
required for high school graduation and successful post-secondary
experiences. Therefore, providing Algebra I teachers with the very best
resources and professional development to ensure effective instruction
has become a priority in Kentucky and across the nation. This research
study is designed to test, through a randomized control trial,
experimental design, an approach that combines online and technology
enhanced instruction with face-to-face classroom instruction to address
this need. This hybrid or "blended" approach has shown promising results
in Kentucky and in research elsewhere.

CNAC, as the lead organization for the research study, has comprised a
team supported by researchers at Education Innovations (EI) and the
University of Virginia (UVA). In addition, the Collaborative for
Teaching and Learning (CTL) and Kentucky Virtual High School (KVHS) will
implement the intervention and support recruitment efforts. Teachers
receiving the intervention will apply the hybrid approach using the
online course curriculum in Algebra I selected by the Kentucky
Department of Education (KDE) and KVHS. This curriculum is an
off-the-shelf product created by university faculty affiliated with the
University of California College Prep Online and the Center for Digital
Innovation, UCLA and was chosen for its quality and content and because
it is customizable, allowing educators to modify the content as needed.
This last feature has particular appeal for going to scale if the
intervention is shown to be effective, as the content can be tailored to
meet the unique needs of different school systems. The courseware has
gone through an external quality control protocol by the National
Repository of Online Courses and has been reviewed by curriculum
specialists at the KDE for quality and alignment with national and state
standards for Algebra I instruction. The results on improved
instructional practices, classroom quality, and student learning will be
compared to those in control sites in which Algebra I instruction will
continue as it has with normal classroom instruction.

Teachers will receive professional development through face-to-face
training led by a master teacher in mathematics and distance training
sessions supported by an online curriculum. Spotlight On Algebra I,
developed by the Southern Region Education Board with funding from the
AT&T Foundation, is the online courseware that provides the framework
for professional development. Training and support for teachers will
begin in the summer and continue throughout the intervention school
year. Use of "Spotlight" is expected to improve teacher skills and
instructional methods in Algebra I, and is expected to work
synergistically with the hybrid curriculum for the following reasons:
(1) the professional development experience provides models of
instructional methods for teachers using online content; (2) it focuses
on Algebra I, allowing teachers to focus on and improve their
subject-specific instructional methods; and (3) it provides a vehicle
(framework) for regular sustained activities and discussions for
participating teachers (the community of learners). Further, coupling
professional development in research-based instructional practices with
hybrid instruction is expected to change the classroom environment,
helping teachers break old habits of instruction and facilitating
adoption of improved, research-based practices. Finally, the hybrid
model provides tools to adopt more effective instructional approaches,
including flexible instruction and formative assessments to help
teachers meet the needs of different learners.

**Research Hypotheses**

This study poses five hypotheses focusing on the impact of the Algebra I
intervention on teaching practices, classroom quality, and student
learning.

*The Hybrid Algebra I approach will:*

> **Hypothesis 1**: *Increase the use of research-based best practices
> for Algebra I as documented in the Algebra I instructional
> standards.*[^1]
>
> **Hypothesis 2**: *Improve classroom quality as seen in increased
> levels of student interest and engagement and academically focused
> class time.*
>
> **Hypothesis** **3**: *Increase student achievement in learning
> Algebra I content and skills.*
>
> **Additional hypotheses:**
>
> **4:** *Achievement gains associated with the online curriculum may
> vary among students with different characteristics (gender,
> race/ethnicity, income as measured by free or reduced-price lunch
> status, LEP, LD/SPED, age).*
>
> **5:** *The beneficial effects of the intervention on student outcomes
> will extend to the post-intervention year, as measured in terms of
> improved performance on the 10th grade PLAN assessment, mathematics
> course-taking, mathematics course grades, and improved high school
> continuation rates.*

**Methodology**

To address the hypotheses in this evaluation, the research team will:
(1) identify the universe of schools in Kentucky that meet the criteria
for inclusion in the study; (2) recruit eligible schools; (3) randomly
assign schools to treatment and control conditions; (4) collect
administrative and survey data, conduct classroom observations,
administer a post-treatment assessment of Algebra I knowledge and
skills; and (4) analyze the data and report the findings from our
analyses. A summary of key activities appears in Table 1.

[]{#__RefHeading___Toc178749306 .anchor}Table 1. Schedule of Activities

+------------------------------------+---------------------------------+
| Activity                           | Schedule                        |
+------------------------------------+---------------------------------+
| Create District and School Pool    | Spring - Fall 2007              |
| for Site Selection                 |                                 |
|                                    |                                 |
| Recruitment                        |                                 |
+------------------------------------+---------------------------------+
| District and School Recruitment    | Winter-Spring 2008              |
| (pending OMB approval)             |                                 |
+------------------------------------+---------------------------------+
| District IRB                       | Spring - Fall 2008              |
+------------------------------------+---------------------------------+
| District and School MOUs           | Spring 2008                     |
+------------------------------------+---------------------------------+
| Random Assignment                  | Spring 2008                     |
+------------------------------------+---------------------------------+
| Start Intervention                 | Summer 2008 (Teachers)          |
+------------------------------------+---------------------------------+
| Start Collection of administrative | Fall 2008                       |
| data                               |                                 |
+------------------------------------+---------------------------------+
| Collect Classroom observations,    | Spring 2009                     |
| Surveys and Algebra I Post-test    |                                 |
+------------------------------------+---------------------------------+
| Collect administrative data on     | Winter 2010                     |
| longer-term outcomes               |                                 |
+------------------------------------+---------------------------------+
| Final Report of Findings           | Fall 2010                       |
+------------------------------------+---------------------------------+

**SUPPORTING STATEMENT B -- COLLECTION OF INFORMATION EMPLOYING
STATISTICAL METHODS**

### 1. Respondent Universe and Sampling Procedures 

The study focuses on the impacts of hybrid instruction in high needs
schools in which Algebra I is taught in the 9^th^ grade. For the
purposes of this study, we define "high needs" in poor performance in
mathematics. Schools that are considered to be poor performers in
mathematics are identified as those in which fewer than 65 percent of
students are proficient in mathematics, as indicated by either the
nationally norm-referenced CTBS/5 exam in mathematics given to 9^th^
graders in school year 2005-2006, or by the criterion-referenced
Kentucky exam (the KCCT) for 8^th^ or 11^th^ graders that share the
school with the 9^th^ graders. These criteria result in a preliminary
list of 210 eligible schools that will be contacted to determine their
interest and ability to participate in the study.

In addition to the criteria described above, schools that express
interest in participation in the study will be asked about their
capacity to support the technology-based intervention. The availability
of technology in schools is not expected to be a limiting factor for the
applicant pool. Rural and low-income schools have amassed considerable
technology resources through the federal e-rate program. That program
has had a direct effect on the almost universal availability of
technology in high-needs schools. [^2] The widespread availability of
adequate technology in high schools has been further corroborated by
KDE. In addition KVHS, which offers online courses throughout KY, will
provide technical assistance to schools in support of the intervention.

*Recruiting eligible schools.* Based on our power analysis (described
below), and the estimated response rate, we established an initial
target sample of 60 schools.

[]{.mark}

The full initial sample of 60 participating schools will include
approximately 120 Algebra I teachers and about 13,500 students who are
enrolled in 9^th^ grade Algebra I classes. After accounting for
attrition/non-response, we anticipate a minimum sample size from which
we will collect data for analysis of 50 schools, 100 teachers and 10,800
students. This sample size reflects an 83 percent response rate for
teachers and schools and an 80 percent response rate for students. These
response rates provide us with a conservative estimate of our final
sample and give us a cushion to help to ensure that we reach the desired
levels of statistical power for our analyses.

[]{.mark}

Study researchers will collect administrative data from the Kentucky
Department of Education (KDE), the school districts, and the schools,
depending on where specific files are stored. In particular, KDE will
provide demographic data for students; districts will provide 8^th^ and
10^th^-grade test scores, and schools will provide enrollment data. The
administrative data will be collected over the period of Fall 2008 to
Winter 2010.

In addition to the administrative data described above, surveys, site
observations, and a post-test of student knowledge and skills will be
collected from study participants. These data will be collected in
Spring 2009. Table 2 summarizes information about the respondent
universe and anticipated sample sizes by data source.

[]{.mark}

[]{#__RefHeading___Toc178749307 .anchor}Table 2. Potential Respondent
Universe (*N*) and Treatment (T)/Control (C) Sample (*n*) and expected
number of respondents (*R*)\*

  ------------------------ ------------- ------- -------------- ------- -------------- --------
                           **Schools**           **Teachers**           **Students**   

  **Universe**             *N* = 210             *N* = 420              *N* = 47,250   

  **Initial Sample**       *n* = 60              *n* = 120              *n* = 13,500   

  **Respondents**          *R=50*                *R=100*                *R=10,800*     

   **Data Source**         T             C       T              C       T              C

  Administrative records                                                5,400          5,400

  Researcher Site          25            25      50             50      5,400          5,400
  Observations (SOM/AQA)                                                               

  Treatment Teacher Survey                       50                                    

  Control Teacher Survey                                        50                     

  Algebra Posttest                                                      5,400          5,400
  ------------------------ ------------- ------- -------------- ------- -------------- --------

\*Source of Estimates below:

Schools: *N* = all regular Kentucky schools with 9^th^ grade Algebra I
classes and meeting eligibility requirements for student proficiency
rates in mathematics; *n* = initial sample of schools; *R*= number of
schools remaining after attrition.

Teachers: *N* = total schools (from above) x 2 Algebra 1 teachers per
school; *n* = total initial sample schools x 2 Algebra 1 teachers per
school (rounded up), *R*= number of teachers remaining after attrition.

Students: *N* = total number of Algebra 1 students in universe, based on
an average enrollment of 225 students per school; *n* = total in initial
sample, *R* = number of students remaining after attrition.

### 

### 2. Number range CHAPTERStatistical Methods for Sample Selection and Degree of Accuracy Needed

######### 

######### a. Stratification and Sample Selection

Schools will be recruited from the full list of eligible schools, to
apply to participate in the study. Recruiting from the list will
continue until we reach our target of 60 schools. The schools will be
recruited in the winter and spring of school year 2007-2008 for
participation during 2008-2009 implementation year. Participating
schools will be randomly assigned to treatment or control groups, with
an equal chance of assignment to either. Within each participating
school, all Algebra I teachers, classrooms and students will participate
in the study. Based on characteristics of schools that are in the
eligible universe, we project a mean of: 2 Algebra I teachers per school
(120 teachers, with 60 per treatment condition), with an average of 225
students per school, for a total of 13,500 students in the initial
sample.

######### b. Estimation Procedures

The Table 3 summarizes the hypotheses, data sources, data collection
procedures and analyses that will be conducted to determine impacts of
the intervention on teaching practices, classroom quality, and
adolescent learning. In each case, we will use a two-tailed test of
statistical significance, and alpha value of .05 to determine
statistical significance. Where multiple indicators are collected, a
Bonferroni-Hochberg correction also will be used.

Primary hypotheses:

The use of KVHS' online courseware as a central instructional device for
adolescent students of Algebra I in a technology-enhanced traditional
secondary classroom setting, coupled with professional development for
teachers in research-based practices in Algebra I and hybrid instruction
to support its effective use will:

[]{#__RefHeading___Toc178749308 .anchor}Table 3 Primary Hypotheses, Data
Sources, Data Collection Procedures and Analyses

+--------------+-----------------+-------------------+---------------+
| **Primary    | **Data          | **Data Collection | **Analysis**  |
| Hypotheses** | Sources**       | Procedure**       |               |
+==============+=================+===================+===============+
| Increase the | **1) School     | Observations of   | Observation   |
| use of       | Observation     | full              | data from     |
| re           | Measure**       | (approximately 1  | the 1) SOM    |
| search-based | (SOM*^©^*). The | hour) Algebra 1   | and 2) AQA    |
| best         | SOM is used to  | classes will be   | will be       |
| practices    | collect data    | conducted in up   | analyzed      |
| for Algebra  | regarding       | to 5 classrooms   | using         |
| I as         | overall         | during 1-day      | Analysis of   |
| documented   | classroom       | visits to each of | Variance      |
| in the       | activities.     | the 30 Treatment  | (ANOVA) in    |
| Algebra I    |                 | and 30 Control    | which         |
| i            | **2) Algebra I  | schools. The 1)   | between-group |
| nstructional | Quality         | SOM and 2) AQA    | (hybrid vs.   |
| standards.\* | Assessment**    | will be used to   | control)      |
|              | (AQA). The AQA  | conduct the 300   | differences   |
| Improve      | is used to      | classroom         | will be       |
| classroom    | record more     | observations,     | investigated  |
| quality as   | detailed        | which will        | for each of   |
| seen in      | information     | involve observing | the           |
| increased    | about observed  | nearly every      | strategies    |
| levels of    | use of the      | Treatment and     | assessed.     |
| student      | Algebra I       | Control teacher   | Effect sizes  |
| interest and | instructional   | for two or more   | will also be  |
| engagement   | standards.\*    | full class        | computed      |
| and          |                 | periods.          | using Cohen's |
| academically | **3) Teacher    | Education         | *d*. To       |
| focused      | Surveys** will  | Innovations will  | protect       |
| class time.  | be used to      | conduct classroom | against the   |
|              | collect         | observations and  | increased     |
|              | Treatment and   | administer        | likelihood of |
|              | Control teacher | teacher surveys   | Type I error, |
|              | perceptions of  | under direction   | the           |
|              | the Algebra I   | of Dr. Deborah    | Bonfer        |
|              | approach they   | Lowther during    | roni-Hochberg |
|              | use (Hybrid vs. | spring 2009. 3)   | step-down     |
|              | district        | Teacher surveys   | procedure to  |
|              | curriculum) and | will be collected | adjust        |
|              | use of the      | by classroom      | statistical   |
|              | Algebra I       | observers on the  | significance  |
|              | instructional   | day of their      | will be used. |
|              | standards.\*    | school visit in   |               |
|              |                 | spring 2009.      | 3\) Teacher   |
|              |                 |                   | survey data   |
|              |                 |                   | will be       |
|              |                 |                   | analyzed by   |
|              |                 |                   | means of      |
|              |                 |                   | ANOVA with    |
|              |                 |                   | Bonfer        |
|              |                 |                   | roni-Hochberg |
|              |                 |                   | step-down     |
|              |                 |                   | procedure to  |
|              |                 |                   | adjust        |
|              |                 |                   | statistical   |
|              |                 |                   | significance. |
+--------------+-----------------+-------------------+---------------+
| Increase     | **1) 8^th^      | 1\) Researchers   | A 2-level     |
| student      | grade scores on | will collect test | *HLM* model   |
| achievement  | the KCCT**.     | scores from       | will be used  |
| in learning  |                 | electronic        | as a baseline |
| Algebra I    | **2) Algebra I  | administrative    | model to      |
| content and  | Assessment**    | files stored by   | assess the    |
| skills.      | **scores**      | districts. 2)     | overall       |
|              |                 | Trained external  | effect of     |
|              | **3) Student    | proctors will     | treatment     |
|              | enrollment      | conduct onsite    | while         |
|              | data**          | administration of | controlling   |
|              |                 | the Algebra 1     | for the       |
|              |                 | Assessment in May | school        |
|              |                 | 2009 in each of   | average       |
|              |                 | the Treatment and | performance   |
|              |                 | Control Schools.  | on the        |
|              |                 |                   | pretest       |
|              |                 | 3\) Student       | (school-level |
|              |                 | enrollment data   | covariate).   |
|              |                 | will be collected |               |
|              |                 | by researchers    |               |
|              |                 | from              |               |
|              |                 | participating     |               |
|              |                 | schools at the    |               |
|              |                 | end of each       |               |
|              |                 | marking period.   |               |
|              |                 | These data will   |               |
|              |                 | be used to        |               |
|              |                 | control for       |               |
|              |                 | amount of         |               |
|              |                 | exposure to the   |               |
|              |                 | intervention in   |               |
|              |                 | the analysis.     |               |
+--------------+-----------------+-------------------+---------------+
| *Additional  | **1)) 8^th^     | 1), (2), (3) Same | A             |
| Hypotheses*  | grade scores on | as above.         | comprehensive |
|              | the KCCT**.     |                   | *HLM* model   |
| Achievement  |                 | 4 ) Researchers   | will be used  |
| gains        | **2) Algebra I  | will collect      | to evaluate   |
| associated   | Assessment      | administrative    | if the        |
| with the     | scores**        | data in           | treatment has |
| online       |                 | electronic files  | different     |
| curriculum   | **3) Student    | from KDE using    | effects for   |
| may vary     | enrollment      | procedures that   | subgroups of  |
| among        | data**          | ensure protection | students      |
| students     |                 | of student        | (e.g.,        |
| with         | **4) Student    | information.      | students with |
| different    | ch              |                   | different     |
| cha          | aracteristics** |                   | math          |
| racteristics | (e.g., gender,  |                   | proficiency;  |
| (gender,     | race/ethnicity, |                   | ethnic        |
| rac          | income as       |                   | majority vs.  |
| e/ethnicity, | measured by     |                   | underserved   |
| income as    | free or         |                   | minority      |
| measured by  | reduced-price   |                   | students,     |
| free or      | lunch status,   |                   | students with |
| r            | LEP, LD/SPED,   |                   | different SES |
| educed-price | and age).       |                   | background).  |
| lunch        |                 |                   |               |
| status, LEP, |                 |                   |               |
| LD/SPED,     |                 |                   |               |
| age).        |                 |                   |               |
+--------------+-----------------+-------------------+---------------+
| The          | **1) 8^th^      | 1), (3), (4) Same | A             |
| beneficial   | grade scores on | as above.         | comprehensive |
| effects of   | the KCCT**.     |                   | *HLM* model   |
| the          |                 | 5\) Researchers   | will be used  |
| intervention | **3) Student    | will collected    | to evaluate   |
| on student   | enrollment      | student           | evidence of   |
| outcomes     | data**          | administrative    | systematic    |
| will extend  |                 | records from      | differences   |
| to the       | **4) Student    | districts or      | in student    |
| post-        | ch              | schools for use   | outcomes      |
| intervention | aracteristics** | in statistical    | between       |
| year, as     | (e.g., gender,  | analyses of       | treatment     |
| measured in  | race/ethnicity, | longer-term       | versus        |
| terms of     | income as       | student outcomes  | control       |
| improved     | measured by     | of the            | groups.       |
| performance  | free or         | intervention.     |               |
| on the 10th  | reduced-price   |                   |               |
| grade PLAN   | lunch status,   |                   |               |
| assessment,  | LEP, LD/SPED,   |                   |               |
| mathematics  | and age).       |                   |               |
| co           |                 |                   |               |
| urse-taking, | **5) Indicators |                   |               |
| mathematics  | of longer-term  |                   |               |
| course       | student         |                   |               |
| grades, and  | outcomes,       |                   |               |
| improved     | including       |                   |               |
| high school  | 10^th^ grade    |                   |               |
| continuation | PLAN Math       |                   |               |
| rates in the | Score,          |                   |               |
| post-        | po              |                   |               |
| intervention | st-intervention |                   |               |
| school year. | math course     |                   |               |
|              | taking, math    |                   |               |
|              | grades in first |                   |               |
|              | 2 marking       |                   |               |
|              | periods of the  |                   |               |
|              | po              |                   |               |
|              | st-intervention |                   |               |
|              | school year, HS |                   |               |
|              | enrollment in   |                   |               |
|              | January of      |                   |               |
|              | 10^th^ grade.** |                   |               |
+--------------+-----------------+-------------------+---------------+

######### 

######### 

######### c. Degree of Accuracy Needed

We conducted a power analysis under a range of assumptions in order to
determine appropriate sample sizes for this study. These assumptions are
conservative and incorporate a cushion for participant attrition to
ensure that the resulting data will yield reliable results. In
particular, we assume that on average, 180 students are clustered within
each of the 50 to 60 schools that comprise the final (respondent)
sample.

The power analysis for our key research question (impact on Algebra I
student learning) follows. We assume that individual students account
for 90 percent of variation in the outcome variable (test score), while
the remaining 10 percent of variation is accounted for by clustering at
the school level. This means that the *ICC* is estimated to be 0.10.
This estimate is based on the consideration that most of our targeted
schools are high-needs high schools in rural areas of Kentucky. As such,
there is some uniformity across these targeted schools. So compared with
a situation where ALL high schools in a state are considered, the
variation among these targeted high-needs schools should be smaller than
that among high schools in general.

We plan to use *at least* one aggregated 2nd level (i.e., school level)
covariate in the model. The cluster level covariate is the 8^th^ grade
math scores aggregated at the school level (i.e., the average pre-test
math score for all the students in a school). Research on school
academic achievement has repeatedly shown that pre-test scores are
highly correlated with post-test scores, thus substantially correlated
with cluster (i.e., school) means of the post-test scores. We assume
that this relationship is R^2^~L2~=0.49.

We assume two levels of effect size:  δ=0.20 and δ=0.25. These two
effect size levels represent small effect sizes as typically discussed
in social and behavioral science in general (e.g., Cohen, 1988). Because
the hybrid approach is rather new, there is limited information about
the magnitude of its effectiveness. In studies that have been reported,
the results appear to be large, but do not convert readily to effect
sizes. For example, Cincinnati Public Schools' Virtual High School, an
alternative school serving drop out recovery students, and over age
students used a hybrid approach in the 2005-2006 school year. The school
reported that the percentage of 10^th^ grade students passing the
statewide graduation exam grew substantially in each of five tested
subject areas, reducing the gap in performance between the Virtual High
School and other schools in the district. In mathematics, for example,
the pass rate increased 22.9 percentage points, from 35.9 to 58.8
percent.
([www.ode.state.oh.us/reportcard](http://www.ode.state.oh.us/reportcard)).
A meta-analysis that included 42 studies of the effects on students of
the use of computer assisted instruction and online activities in
conventional educational settings reported a weighted mean effect size
of .448 for cognitive outcomes. Waxman, Lin and Michko (2003) Thus, the
MDE's used for determining sample sizes for the current study seem
reasonable. Given these parameters and assumptions as detailed above, we
can see that the proposed design and analysis has sufficient power for
detecting the hypothesized effect sizes. Without attrition at the school
level, the minimum power level is about **[0.90]{.underline}** (as
indicated by the upper dashed arrow imposed on the lower power curve,
where the effect size is assumed to be .20). Even with the planned 83
percent response rate at the school level (60 schools dropped to 50
schools), we can still expect a power level of about
**[0.83]{.underline}** for the smaller effect size. These power levels
are considered sufficient in applied research and should be considered
as realistic and conservative estimates.

**Figure 1: Power Curves for Cluster Randomized Trials**

The lower curve shows power for an effect size of .20. The upper curve
shows power for an effect size of .25.

![](media/image1.wmf){width="4.290972222222222in"
height="3.6145833333333335in"}

Power curves are based on a hypothesized effect size of .20 and analytic
approach described in, "Optimal Design for Longitudinal and Multilevel
Research-V1.55", Raudenbush, S. W., Spybrook, J., Liu, X.F., & Congdon,
R., 2005.

######### d. Unusual Problems Requiring Specialized Sampling Plans

None.

######### e. Use of Periodic Data Collection Cycles to Reduce Burden

Not applicable.

### 3. Methods to Maximize response rates and Deal with Nonresponse

We will use data from the most recent Common Core of Data to identify
all schools in KY with a 9^th^ grade. We will merge these data with data
from the Kentucky Department of Education website on school performance
in mathematics and use the resulting data file to select for possible
inclusion in the study all regular high schools in Kentucky in which (1)
Algebra I is taught (and will continue to be taught for the duration of
the study) to 9^th^ graders and (2) fewer than 65 percent of students
are proficient in mathematics, as indicated by either the CTBS/5 exam
for 9^th^ graders or the KCCT exam for 8^th^ or 11^th^ graders, using
the most recent available data.

[Recruiting strategy]{.underline}

Successful recruiting of randomly selected schools is critical to the
sample design. The research team has set aside ample resources to ensure
participation. The Collaborative for Teaching And Learning (CTL) will
have a critical role in implementation of the recruiting strategy under
the direction of the PI.

-   CTL will develop awareness for the study by organizing sessions at
    professional conferences widely attended by Kentucky teachers of
    mathematics, including the fall 2007 Kentucky Council of Teachers of
    Mathematics and the winter 2008 Kentucky Teaching and Learning
    Conference (KTLC).

-   CTL will draft a letter for signature from the Chief State School
    Officer to qualifying districts and schools informing them of their
    eligibility for the study and encouraging them to participate (see
    Exhibit A).

-   Letters will be mailed to the 210 building administrators that are
    eligible for participation in the study, as well as their district
    superintendent. The letters will arrive in a large KDE envelope with
    a brochure describing the intervention and promoting its value, and
    a "commitment statement" for interested teachers and schools to
    complete (Exhibit A). A pre-stamped and addressed envelope will be
    included in the package for applicants.

-   Commitment statements will be screened by CTL to ensure that schools
    meet minimum requirements for participation.

-   Follow-up reminders will be sent by e-mail from the commissioner,
    KDE, directly to all 210 eligible schools (Exhibit A).

-   CTL will contact district leaders and school principals by phone to
    answer questions and further encourage participation.

-   CTL will arrange for presentations at each of 7 regional cooperative
    meetings in spring 2008. Superintendents from all districts in each
    region attend these meetings. This forum will be used to familiarize
    superintendents with the intervention and the study. Two members of
    the research team (a researcher and a practitioner involved in
    implementation) will give the presentation and answer questions. A
    template for scheduling that lists contacts for each regional
    cooperative appears in Appendix A.

-   CTL will arrange for similar follow-up meetings for principals and
    teachers, following the cooperative meeting in each region.

Our goal is to recruit 60 schools from the 210 schools that meet our
eligibility criteria. The Collaborative for Teaching and Learning (CTL)
is an excellent choice for ensuring successful recruiting of our
selected sites. CTL is headed by a former associate commissioner of
education in the state, and has the contacts and reputation that will
help her recruit the selected schools for the study. In addition, CTL
has several projects that has put them in direct contact with schools
that would be eligible for participation in the experiment. In
particular, CTL worked directly with a number of networks of schools and
practitioners through GEAR UP and Western KY Partnership Gear Up
schools. These are schools with more than 50 percent of their students
on free and reduced lunch, and an overall accountability score in dire
need of improvement.

[Response Rate]{.underline}

Because school participation requires teachers and their principals and
superintendents to volunteer to participate, the anticipated response
rate among participating schools is expected to be high for this
collection. Additional intervention and research design elements that
will contribute to a high response rate include: (1) an intervention
that incorporates frequent contacts throughout the intervention with a
master teacher and a learning community of other participating teachers
for professional development; (2) a professional development program
that is built around subject matter and pedagogy to improve teachers'
current classroom practices; (3) technology supports from KDE to reduce
the impact of technical glitches that could impede program success, or
reduce teacher enthusiasm for the intervention; (4) orientation meetings
for treatment and control teachers; and (5) each participating teacher
will be asked to review an information sheet describing their
responsibilities under the study. (Please see Exhibit B.)

While the above elements will help to reduce attrition from the program,
data collection has also been designed specifically to capture as many
responses as possible. In particular, both teacher and student
questionnaires will be administered during researcher site visits to
schools. Student assessments will be given during the regular school
day. As a result, there is no need to mail/e-mail responses, and it will
be a specific responsibility of the visiting researcher to collect data
from all measurement instruments.

### 4. Tests of Procedures or Methods to be Undertaken

Classroom observation tools (SOM, and a form similar to the AQA) have
been used in multiple research studies and have validation and
reliability information in addition to norms. The Algebra 1 teacher
questionnaires have been pre-tested by nine Algebra 1 teachers in
Kentucky schools. The average completion time for the teacher survey was
9 minutes. The teachers who completed the surveys for the pretest were
asked to indicate any items that were unclear and offer suggestions for
improving the surveys. The pretest participants indicated that the
surveys were clear and easy to understand. No teachers provided
suggestions for revisions.

### 5. Individuals Involved

  ---------------- ---------------- ------------------------ --------------
  Name             Role             Title                    Telephone

  Linda Cavalluzzo PI               Senior Researcher, The   703-824-2197
                                    CNA Corporation          

  Deborah Lowther  Co-PI            Assoc. Professor,        901-678-5645
                                    University of Memphis,   
                                    and Senior Researcher,   
                                    Education Innovations    

  Xitao Fan        Methodologist    Professor, Research,     434-243-8906
                                    Statistics and           
                                    Evaluation, The Curry    
                                    School of Education,     
                                    University of Virginia   

  Michael Puma     External         Senior Evaluator,        410-897-0968
                   evaluator        Chesapeake Research      
                                    Associates, LLC          

  Johannes Bos     Technical        President and CEO,       510-465-7884
                   Working Group    Berkeley Policy          
                   (TWG)            Institute                

  Laura M.         TWG              Professor of Public      615-322-5521
  Desimone                          Policy and Education,    
                                    Peabody College of       
                                    Education and Human      
                                    Development Vanderbilt   
                                    University               

  Barbara Goodson  TWG              Senior Researcher, Abt   617-349-2811
                                    Associates               

  Rebecca Maynard  TWG              University Trustee       215-898-3558
                                    Chair, Professor of      
                                    Education and Social     
                                    Policy, University of    
                                    Pennsylvania             

  Samuel           TWG              Nystrand Center of       502-852-0615
  Stringfield                       Excellence\              
                                    in Education, University 
                                    of Louisville            
  ---------------- ---------------- ------------------------ --------------

**References**

Virtual High School, School Year Report Card (2004-2005 and 2005-2006).
http://[www.ode.state.oh.us/reportcard/](http://www.ode.state.oh.us/reportcard/)
\[report card not cited in part A, but plenty of references to the
virtual high school\]

Waxman, H.C., Lin, M., & Michko, G. (2003). A Meta-analysis of the
Effectiveness of Teaching and Learning with Technology on Student
Outcomes. Learning Point Associates http://www.ncrel.org/tech/effects/

[]{#__RefHeading___Toc178749313 .anchor}Appendix A

[]{#__RefHeading___Toc178749314 .anchor}Informational Meeting Schedule
Template for Kentucky Educational Cooperatives

**Kentucky Educational Cooperatives -- Mar-April Board Meetings**

+----------------+----------------+-------------+--------------------+
| **Educational  | **Executive    | **Phone/    | **Hybrid Algebra   |
| Cooperative**  | Director**     | Fax/email** | Study              |
|                |                |             | Presentation**     |
|                |                |             |                    |
|                |                |             | **10-15            |
|                |                |             | minu               |
|                |                |             | tes\--presenters** |
+----------------+----------------+-------------+--------------------+
| Central        | Donald Pace    | Ph:         | TBD                |
| Kentucky       |                | 8           |                    |
| Education      |                | 59-257-3244 |                    |
| Cooperative    |                |             |                    |
|                |                | Fax:        |                    |
| University of  |                | 2           |                    |
| Kentucky, 43   |                | 70-745-6892 |                    |
| Dickey Hall    |                | <dwpace@p   |                    |
|                |                | op.uky.edu> |                    |
| Lexington, KY  |                |             |                    |
| 40506-0017     |                |             |                    |
+----------------+----------------+-------------+--------------------+
| Green River    | Liz Storey     | Ph:         | TBD                |
| Regional       |                | 2           |                    |
| Educational    |                | 70-745-2451 |                    |
| Cooperative    |                |             |                    |
| (GRREC)        |                | Fax:        |                    |
|                |                | 2           |                    |
| Western KY     |                | 70-745-5199 |                    |
| University     |                |             |                    |
|                |                | <Li         |                    |
| 1906 College   |                | z.Storey@gr |                    |
| Heights Blvd.  |                | rec.ky.gov> |                    |
| #21031         |                |             |                    |
|                |                |             |                    |
| Bowling Green, |                |             |                    |
| KY 42101-1031  |                |             |                    |
+----------------+----------------+-------------+--------------------+
| Kentucky       | Stan Riggs     | Ph:         | TBD                |
| Educational    |                | 6           |                    |
| Development    |                | 06-928-0205 |                    |
| Corporation    |                | x 2201      |                    |
| (KEDC)         |                |             |                    |
|                |                | Fax:        |                    |
| 904 W. Rose    |                | 6           |                    |
| Road           |                | 06-928-3785 |                    |
|                |                | [Stan.Riggs |                    |
| Ashland, KY    |                | @kedc.org]{ |                    |
| 41102-7104     |                | .underline} |                    |
+----------------+----------------+-------------+--------------------+
| Kentucky       | Jeff Hawkins   | Ph:         | TBD                |
| Valley         |                | 6           |                    |
| Educational    |                | 06-439-1119 |                    |
| Cooperative    |                | x 26        |                    |
|                |                |             |                    |
| Hazard         |                | Fax:        |                    |
| Community      |                | 6           |                    |
| College        |                | 06-439-1322 |                    |
|                |                | [Jeff       |                    |
| 1 Community    |                | .Hawkins@]{ |                    |
| College Drive  |                | .underline} |                    |
|                |                |             |                    |
| Hazard, KY     |                | [kentuckyv  |                    |
| 41701          |                | alley.org]{ |                    |
|                |                | .underline} |                    |
+----------------+----------------+-------------+--------------------+
| Northern       | Dawn Tackett   | Ph:         | TBD                |
| Kentucky       |                | 8           |                    |
| Cooperative    |                | 59-442-8600 |                    |
| for            |                | x 16        |                    |
| Educational    |                |             |                    |
| Services       |                | Fax:        |                    |
|                |                | 442-7015    |                    |
| 5516 East      |                | [Da         |                    |
| Alexandria     |                | wn.Tackett@ |                    |
| Pike           |                | nkces.org]{ |                    |
|                |                | .underline} |                    |
| Cold Spring,   |                |             |                    |
| KY 41076       |                |             |                    |
+----------------+----------------+-------------+--------------------+
| Ohio Valley    | Leon Mooneyhan | Ph:         | TBD                |
| Educational    |                | 5           |                    |
| Cooperative    |                | 02-647-3533 |                    |
| (OVEC)         |                | x 251       |                    |
|                |                |             |                    |
| P.O. Box 1249  |                | Fax:        |                    |
|                |                | 5           |                    |
| Shelbyville,   |                | 02-647-3581 |                    |
| KY 40066       |                | [LMooneyhan |                    |
|                |                | @ovec.org]{ |                    |
|                |                | .underline} |                    |
+----------------+----------------+-------------+--------------------+
| West Kentucky  | John Settle    | Ph:         | TBD                |
| Educational    |                | 2           |                    |
| Cooperative    |                | 70-809-6978 |                    |
|                |                |             |                    |
| Murray State   |                | Fax:        |                    |
| University     |                | 2           |                    |
|                |                | 70-809-2485 |                    |
| 420 Wells Hall |                |             |                    |
|                |                | <John.Settl |                    |
| Murray, KY     |                | e@wkec.org> |                    |
| 42071-3340     |                |             |                    |
+----------------+----------------+-------------+--------------------+
| South          | William Thames | Ph:         | TBD                |
| East/South     |                | 8           |                    |
| Central        | Tom            | 59-622-2581 |                    |
| Educational    | Bonny-Asst.    |             |                    |
| Cooperative    | Director       | Fax:        |                    |
|                |                | 8           |                    |
| Eastern KY     |                | 59-622-6526 |                    |
| University     |                |             |                    |
|                |                | <W          |                    |
| 417 Bert Combs |                | illiam.Tham |                    |
| Building       |                | es@eku.edu> |                    |
|                |                |             |                    |
| Richmond, KY   |                | [Tom.Bonn   |                    |
| 40475          |                | y@eku.edu]{ |                    |
|                |                | .underline} |                    |
+----------------+----------------+-------------+--------------------+

[^1]: Based on National Council of Teachers of Mathematics, North
    American Council for Online Learning (NACOL), ISTE NETS-T and
    NETS-S, and KY Algebra I standards.

[^2]: In all four states in this region, 99 percent of the elementary
    and secondary schools have Internet access. In Kentucky and
    Tennessee, over 86 percent have broadband access. Last year, the
    four states received a total of \$170 million in E-Rate grants to
    support their own expenditures for wiring classrooms, Internet
    access and telecommunications services. (Technology Counts, 2005, as
    reported in Ed Week 5 May 2005).