------------------- ------------------------------------------------------------------
  **ESEA: Teaching    
  American History**  

  **FY 2009 Program   
  Performance Plan**  

  Strategic Goal 1    

  Discretionary       

  ESEA, Title II,     
  Part C-4            

  CFDA                84.215X: Teaching of Traditional American History
  ------------------- ------------------------------------------------------------------

  -------------- --------------------------------------------------------
  **Program      **To improve student achievement by providing
  Goal:**        high-quality professional development to elementary and
                 secondary-level teachers of American history.**

  -------------- --------------------------------------------------------

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

  ------------------------- --------------------------------------------------------
  ***Objective 1 of 2:***   *Demonstrate the effectiveness of professional
                            development activities for secondary level teachers of
                            American history through the increased achievement of
                            their students.*

  ------------------------- --------------------------------------------------------

+-------+----------------+------------------+-------------------------+
| *     |                |                  |                         |
| *Meas |                |                  |                         |
| ure 1 |                |                  |                         |
| .1 of |                |                  |                         |
|  2:** |                |                  |                         |
|       |                |                  |                         |
| Stu   |                |                  |                         |
| dents |                |                  |                         |
| in    |                |                  |                         |
| ex    |                |                  |                         |
| perim |                |                  |                         |
| ental |                |                  |                         |
| and   |                |                  |                         |
| qua   |                |                  |                         |
| si-ex |                |                  |                         |
| perim |                |                  |                         |
| ental |                |                  |                         |
| st    |                |                  |                         |
| udies |                |                  |                         |
| of    |                |                  |                         |
| e     |                |                  |                         |
| ducat |                |                  |                         |
| ional |                |                  |                         |
| eff   |                |                  |                         |
| ectiv |                |                  |                         |
| eness |                |                  |                         |
| of    |                |                  |                         |
| Tea   |                |                  |                         |
| ching |                |                  |                         |
| Ame   |                |                  |                         |
| rican |                |                  |                         |
| Hi    |                |                  |                         |
| story |                |                  |                         |
| pro   |                |                  |                         |
| jects |                |                  |                         |
| will  |                |                  |                         |
| d     |                |                  |                         |
| emons |                |                  |                         |
| trate |                |                  |                         |
| h     |                |                  |                         |
| igher |                |                  |                         |
| a     |                |                  |                         |
| chiev |                |                  |                         |
| ement |                |                  |                         |
| on    |                |                  |                         |
| c     |                |                  |                         |
| ourse |                |                  |                         |
| co    |                |                  |                         |
| ntent |                |                  |                         |
| mea   |                |                  |                         |
| sures |                |                  |                         |
| a     |                |                  |                         |
| nd/or |                |                  |                         |
| stat  |                |                  |                         |
| ewide |                |                  |                         |
| U.S.  |                |                  |                         |
| hi    |                |                  |                         |
| story |                |                  |                         |
| a     |                |                  |                         |
| ssess |                |                  |                         |
| ments |                |                  |                         |
| than  |                |                  |                         |
| stu   |                |                  |                         |
| dents |                |                  |                         |
| in    |                |                  |                         |
| co    |                |                  |                         |
| ntrol |                |                  |                         |
| and   |                |                  |                         |
| compa |                |                  |                         |
| rison |                |                  |                         |
| gro   |                |                  |                         |
| ups.  |                |                  |                         |
|       |                |                  |                         |
|   (De |                |                  |                         |
| sired |                |                  |                         |
| direc |                |                  |                         |
| tion: |                |                  |                         |
| incr  |                |                  |                         |
| ease) |                |                  |                         |
|       |                |                  |                         |
|  1432 |                |                  |                         |
+-------+----------------+------------------+-------------------------+
| **Y   | **Target**     | **Actual\        | **Status**              |
| ear** |                | (or date         |                         |
|       |                | expected)**      |                         |
+-------+----------------+------------------+-------------------------+
| 2004  | Set a Baseline | (February 2008)  | Pending                 |
+-------+----------------+------------------+-------------------------+
| 2005  | BL+1PP         | (February 2008)  | Pending                 |
+-------+----------------+------------------+-------------------------+
| 2006  | BL+1PP         | (February 2008)  | Pending                 |
+-------+----------------+------------------+-------------------------+
| 2007  | BL+2PP         | (February 2009)  | Pending                 |
+-------+----------------+------------------+-------------------------+
| 2008  | BL+2PP         | (February 2010)  | Pending                 |
+-------+----------------+------------------+-------------------------+
| 2009  | BL+2PP         | (February 2011)  | Pending                 |
+-------+----------------+------------------+-------------------------+

**Source**. U.S. Department of Education, Teaching American History
Grantee Performance Report.

**Frequency of Data Collection**. Annual

**Explanation**. 

Measure 1.1-Baseline data cannot be set at this time due to the lack of
reliable data provided by Teaching American History grantees funded from
2003-2005 on their 2006 Annual Performance Reports. \
\
To resolve these data discrepancies, a contract to perform a National
Study of Teaching American History Grants will be awarded in September
2007. This contract will cover the 2004, 2005, and 2006 cohorts of TAH
grants. Two analyses of grantee data (see below) will be available to be
inputted in August 2008 when final reports on both analyses are
completed.\
\
**Analysis of Grantee Evaluations through Annual Performance Reports
(APRs):** The contractor shall compile and analyze grantee APRs from
grantees that have conducted rigorous evaluations using national or
state tests to document changes in teacher content knowledge and student
achievement for inclusion in a meta-analysis.\
\
**Meta-Analysis/Empirical Synthesis of High Quality Grantee Evaluations
of Teacher Content Knowledge and Student Achievement:** The contractor
shall perform a meta-analysis/empirical synthesis of those grantee
evaluations using national and state tests of teacher content knowledge
and student achievement that are determined to be sufficiently rigorous
in order to determine an overall pattern and effect size. \
\
[Further information:]{.underline}\
The difficulties in collecting data on both measures is described in the
following report sections below.\
**Teaching American History**\
Review and Synthesis of the 2006\
Annual Performance Reports\
Prepared By:\
Patricia Muller, PhD (Pages 7-10)\
\
**Student Achievement Issues** - An attempt was also made to determine
the extent to which TAH grantees included measures of change in student
achievement as a GPRA indicator. In general, approximately 41% of TAH
grantees specifically included the GPRA measure related to student
achievement (32% for the 2003 cohort; 29% for the 2004 cohort; and 65%
for the 2005 cohort). However, grantees that included this GPRA
indicator did not necessarily actually provide valid or reliable data
related to this indicator. Table 3 provides details related to TAH
grantees' measurement of student achievement as reported on the 2006
APR. This table highlights the disparities between what TAH grantees
report they are measuring in their project objectives or performance
measures, versus what they appear to be measuring in actuality. For
example, although 76% of TAH grantees included a project objective or
performance measure that specifically focuses on change in student
achievement (see Table 2 above), a closer review of the APR and
supplemental data indicates that only 54% of these grantees stating that
they measured changes in student achievement (or 41% of the total number
of grantees) actually included using a student content knowledge exam to
measure the change. As noted in the table below, other TAH grantees used
the following to measure changes in student achievement: teacher
self-reports of impact on student achievement, student course grades,
evaluations of student work products, and surveys of student
attitudes/interest in American History. Some grantees also included
measures such as the level of student participation in TAH-related
events (e.g., History As noted above with regards to student
achievement, an attempt was also made to determine the extent to which
TAH grantees included measures of change in teacher content knowledge as
a GPRA indicator. In general, approximately 45% of TAH grantees
specifically included the GPRA measure related to teacher content
knowledge (29% for the 2003 cohort; 44% for the 2004 cohort; and 65% for
the 2005 cohort). The sharp increase in percentage of grantees including
teacher content knowledge across grantee cohort years (e.g., 65% for
2005 grantees versus 29% for 2003 grantees) likely reflects the addition
of a formal GPRA indicator in later grant years, as well as the TAH
program office's increased emphasis on measuring teacher content
knowledge. However, as with student achievement, grantees that included
this GPRA indicator did not necessarily actually provide valid or
reliable data related to this indicator.\
\
**Evaluation Methodology and Measurement Issues**. Many of the problems
cited (approximately 25) specifically related to evaluation methodology
and/or measurement. Most commonly, these issues focused on the following
categories: \
\
• Problems with measuring impact on student achievement and/or teacher
content knowledge was noted by grantees. Some grantees noted the lack of
a statewide standardized achievement test for students, and/or changes
that resulted in previously used student achievement tests no longer
being administered (e.g., state stopped using a social studies
standardized text; end-of-course test was revised, postponing the
administration of the test until the following year). Other grantees
also noted difficulties identifying a valid and reliable measure of
change in teacher content knowledge, and noted problems associated with
using nationally validated tests of teacher content knowledge for TAH
programs. \
\
• Difficulties implementing experimental and/or quasi-experimental
designs were noted by various grantees, particularly problems with
developing an experimental design with a control group, and problems
identifying careful and valid comparison groups for matching purposes. \
\
• Lack of buy-in and/or cooperation from teachers or school
administrators to the evaluation contributed to issues with the both the
reliability of data and the actual data collection process itself. For
example, grantees noted "unwillingness" of teachers to complete
portfolios, and the reluctance of teachers or school administrators to
allow classroom observations. \
\
• Other issues noted by grantees including problems associated with
self-report data, and problems with measuring the impact of one-time
forums or professional development activities.

  ----------------------- ----------------- ------------------- --------------------------
  **Measure 1.2 of 2:**                                         
  Teachers will                                                 
  demonstrate an                                                
  increased understanding                                       
  of American history                                           
  through the use of                                            
  nationally validated                                          
  tests of American                                             
  history that can be                                           
  directly linked to                                            
  their participation in                                        
  the Teaching American                                         
  History program\                                              
  \                                                             
    (Desired direction:                                         
  increase)   1768                                              

  **Year**                **Target**        **Actual\           **Status**
                                            (or date            
                                            expected)**         

  2005                    BL+1PP            (February 2008)     Pending

  2006                    BL+1PP            (February 2008)     Pending

  2007                    BL+1PP            (February 2009)     Pending

  2008                    BL+1PP            (February 2010)     Pending

  2009                    BL+2PP            (February 2011)     Pending
  ----------------------- ----------------- ------------------- --------------------------

**Source**. U.S. Department of Education, Teaching American History
Grantee Performance Report.

**Frequency of Data Collection**. Annual

**Explanation**. 

Measure 1.2- Actual performance data on teacher content knowledge is
unavailable at this time due to the lack of reliable data from TAH
grantees on their Annual Performance Reports.\
\
To resolve these data discrepancies, a contract to perform a National
Study of Teaching American History Grants will be awarded in September
2007. This contract will cover the 2004, 2005, and 2006 cohorts of TAH
grants. Two analyses of grantee data (see below) will be available to be
inputted in August 2008 when final reports on both analyses are
completed.\
\
**Analysis of Grantee Evaluations through Annual Performance Reports
(APRs):** The contractor shall compile and analyze grantee APRs from
grantees that have conducted rigorous evaluations using national or
state tests to document changes in teacher content knowledge and student
achievement for inclusion in a meta-analysis.\
\
**Meta-Analysis/Empirical Synthesis of High Quality Grantee Evaluations
of Teacher Content Knowledge and Student Achievement:** The contractor
shall perform a meta-analysis/empirical synthesis of those grantee
evaluations using national and state tests of teacher content knowledge
and student achievement that are determined to be sufficiently rigorous
in order to determine an overall pattern and effect size.\
\
The difficulties in collecting data on both measures are described in
the following report sections below.\
**Teaching American History : Review and Synthesis of the 2006 Annual
Performance Reports**\
Prepared By:\
Patricia Muller, PhD (Pages 7-10)\
\
Teacher Content Knowledge-Table 4 provides details related to TAH
grantees' measurement of teacher content knowledge as reported on the
2006 APR. This table highlights the disparities between what TAH
grantees report they are measuring in their project objectives or
performance measures, versus what they appear to be measuring in
actuality. For example, although 85% of TAH grantees included a project
objective or performance measure that specifically focused on changes in
teacher content knowledge (see Table 2), a closer review of the APR and
supplemental data indicates that only 34% of these grantees stating that
they measured changes in teacher content knowledge (or 29% of the total
number of grantees) actually included using a teacher content knowledge
exam to measure the change. As noted in the table below, almost a third
of TAH grantees reported using teacher self-report surveys to measure
impact on teacher content knowledge (i.e., measuring teachers'
perception of impact on their content knowledge). Other grantees
reported using surveys of teachers' satisfaction with the project,
evaluations of teacher materials/portfolio, or surveys of teachers'
perceptions of impact on their teaching strategies. Interviews and/or
focus groups, self-report surveys of teachers' attitudes, classroom
observation and the completion of graduate/certification were also used
by some grantees as indicators of changes in teacher content knowledge.
Approximately one-third of grantees also noted the numbers of teachers
participating in professional development as a measure of change in
teacher content knowledge.\
\
**Evaluation Methodology and Measurement Issues**. Many of the problems
cited (approximately 25) specifically related to evaluation methodology
and/or measurement. Most commonly, these issues focused on the following
categories: \
\
• Problems with measuring impact on student achievement and/or teacher
content knowledge was noted by grantees. Some grantees noted the lack of
a statewide standardized achievement test for students, and/or changes
that resulted in previously used student achievement tests no longer
being administered (e.g., state stopped using a social studies
standardized text; end-of-course test was revised, postponing the
administration of the test until the following year). Other grantees
also noted difficulties identifying a valid and reliable measure of
change in teacher content knowledge, and noted problems associated with
using nationally validated tests of teacher content knowledge for TAH
programs. \
\
• Difficulties implementing experimental and/or quasi-experimental
designs were noted by various grantees, particularly problems with
developing an experimental design with a control group, and problems
identifying careful and valid comparison groups for matching purposes. \
\
• Lack of buy-in and/or cooperation from teachers or school
administrators to the evaluation contributed to issues with the both the
reliability of data and the actual data collection process itself. For
example, grantees noted "unwillingness" of teachers to complete
portfolios, and the reluctance of teachers or school administrators to
allow classroom observations. \
\
• Other issues noted by grantees including problems associated with
self-report data, and problems with measuring the impact of one-time
forums or professional development activities.

  -----------------------------------------------------------------------

  -----------------------------------------------------------------------

  ------------------------- --------------------------------------------------------
  ***Objective 2 of 2:***   *Improve the operational efficiency of the program.*

  ------------------------- --------------------------------------------------------

  ----------------------- ---------------- ----------------- -----------------------------
  **Measure 2.1 of 1:**                                      
  Cost per teacher                                           
  participant.   (Desired                                    
  direction: decrease)                                       
    89a0of                                                   

  **Year**                **Target**       **Actual\         **Status**
                                           (or date          
                                           expected)**       

  2007                                     4,925.59          Measure not in place

  2008                    BL-1             (October 2008)    Pending

  2009                    BL-2%            (October 2009)    Pending
  ----------------------- ---------------- ----------------- -----------------------------

**Source**. U.S. Department of Education, Teaching American History
Grantee Performance Report.

**Frequency of Data Collection**. Annual

**Data Quality**. Baseline FY2006 grantee data was reported in 2007.\
\
Analysis of FY2007 grantee data is pending and validated data is
expected to be available in October 2008 .