Deliverable 4.3.1(Final) OY 3, NPD GPRA IndicatorsPage 21

National Professional Development Program:

GPRA Indicators and Contextual Summaries

Projects First Funded in 2007, FY 2010-11 Reporting

Task 4.3.1, Option Year 3

Prepare Discretionary Program Reports (Final)

Due: July 3, 2012

Submitted:  June 22, 2012

Prepared for:

Cynthia Ryan, Division Director, Discretionary Grants

OELA, ED

Prepared by:

Judith Wilde, PhD, Executive Director

Keira Ballantyne, PhD, Associate Director

Marilyn Hillarious, Research Assistant

Krystyna Sonnenberg, Research Assistant

NCELA, GW

**Table of Contents**

**Introduction**1

**Part I:  Contextual Summaries**1

**Program descriptors**1

**Minority-serving institutions**2

**IHE Carnegie classifications**2

**Part II:  GPRA Analysis**4

**Introduction**4

**GPRA Indicator #1.1**5

**GPRA Indicator #1.2**5

**GPRA Indicator #1.3**5

**GPRA Indicator #1.4**6

**GPRA Indicator #1.5**6

**GPRA Indicator #1.6**6

**Part III: Executive Summaries**7

**Participants' reactions**8

**Participants' learning**9

**IHE support and change**10

**Participants' use of new knowledge**12

**Participants' effectiveness**13

**Challenges**15

**Successes**18

**Summary/Conclusions**20

**Appendices**

A:Coding sheet for CDRsSeparate attachment

B: Coding sheet for Contextual SummariesSeparate attachment

**National Professional Development Program: GPRA Indicators and Contextual Summaries (Draft Report)**

**Introduction**

This is the draft report on the Government Performance and Results Act (GPRA) Indicators for the 2007 cohort of National Professional Development (NPD) program grantees.  It summarizes fourth-year (FY 2010-11) reports provided by projects first funded in 2007 under the NPD program.  A total of 139 projects, in 37 states, the Commonwealth of Puerto Rico, the District of Columbia, and the Territory of Guam, were funded in that year.  All projects are located within institutions of higher education (IHEs); all collaborate with, or have partnerships with, local school districts.

Three types of data inform this report:  (1) Complete Data Reports written by the NPD projects to provide information on their 2010-2011 accomplishments (see Appendix A for these data); (2) contextual information collected by the staff of the Office English Language Acquisition (OELA) and the National Clearinghouse for English Language Acquisition (NCELA) that describe specific program features and the institutions in which projects are located (see Appendix B for this information); and (3) Executive Summaries developed by the NPD grantees to provide an overview of their accomplishments across their first four project years (i.e., 2007-2011).  To date, staff from OELA have provided CDRs from 124 NPD projects and executive summaries from 125 NPD projects – a total of 132 projects (95.0 percent of all projects) have provided CDRs and/or executive summaries.

**Part I:  Contextual Summaries**

Three sets of contextual data were collected this year.  These data reflect all 134 projects, whether or not they had provided CDRs and/or executive summaries.  First, OELA staff created a spreadsheet that identified specific aspects of each project such as focus of the project (e.g., type of personnel - teachers-in-training, paraprofessionals, and/or in-service teachers - and school type - elementary, middle, and/or high school).  Second, NCELA identified the context of the IHEs, focusing on minority-serving institutions and Carnegie classifications of the size and focus of each IHE.   See Appendix B for this information.

**Program descriptors**

****The OELA staff coded six specific types of information about the projects being implemented by these grantees.  These are summarized below; information is presented hierarchically, that is, the information about the greatest number of grantees is listed first, the least number of grantees is listed last.

· Grantees are working primarily with in-service teachers (*n*=113), pre-service teachers (*n*=72), and college of education faculty (*n*=36), as well as school administrators, counselor, paraprofessionals, parents, faculty from schools/departments of arts and sciences, and school psychologists.

· In programs leading to certification or endorsement, the primary focus area was English-as-a-Second Language instruction (*n*=96) or bilingual instruction (*n*=22); special education, early childhood education, and "other" types of endorsements also were sought.  No projects reported participants seeking an administrative certificate or credential.

· A total of 22 projects reported that participants were Spanish-speakers; one project reported Portuguese-speaking participants and one project reported that participants spoke Polish, French, Malayalam, Korean, Romanian, Urdu, German, Tagalog, Gurati, and Hindi.

· The greatest number of projects resulted in participants receiving a bachelor's degree (*n*=38) or master's degrees (*n*=37); a few projects assisted students with associate degrees or doctoral degrees.

· Program completers will be (or are) working in secondary (*n*=117), elementary (*n*=93), and middle (*n*=75) schools.

· Some projects reported special features including offering distance education or online education (*n*=12), having a tenured or tenure-track project director (*n*=12), working with special education, offering a Career Ladder program for paraprofessionals, working with rural schools, focusing on early education, and developing Native American Teachers.

**Minority-serving institutions**

****There are four types of minority-serving institutions (MSIs):  Hispanic Serving Institutions (HSIs), Historically Black Colleges and Universities (HBCU), Asian American-Native American-Pacific Islander Serving Institutions (AANAPISIs), including tribal colleges.  Of the total 134 grantees, 33 (23.7%) are located at IHEs representing one or more of the three designations.  The HBCUs and AANAPISIs have been identified by White House Initiatives; the membership of the Hispanic Association of Colleges and Universities (HACU) was used to identify Hispanic-serving IHEs.  HACU's membership categories allow a finer breakdown of information:  enrolling 25 percent of more Hispanic students or enrolling 10-25 percent Hispanic students.  There were 23 HSIs, 7 AANAPISIs (including 1 tribal college), and 1 HBCU.

**IHE Carnegie classifications****

**As described by the Carnegie Foundation, all accredited, degree-granting colleges and universities in the United States represented in the National Center for Education Statistics IPEDS system are eligible for inclusion in the Carnegie Classifications (as of the year a classification is issued, and subject to the availability of required data). Accreditation status is based on information provided by the US Department of Education, Office of Postsecondary Education.  While the various Carnegie Classifications provide a great deal of detail about the IHEs, here we report only information from Carnegie's "Basic Classification" which describes IHEs based on the type(s) of degrees that the institution offers.  Of the six generally described types of IHEs, NPD projects were located in all but the special focus institutions (those that award degrees in a single field or set of related fields (e.g., Law, Medicine, faith-related.)

*Associate's colleges*are those that offer only associate degrees, or for which bachelor's degrees represent no more than 10 percent of all undergraduate degrees offered, and do not include IHEs that fall into the "tribal college" or "special focus" categories.  Within the 134 projects, there were 5 located at community colleges – one project at each college.  Of these colleges,

· All are public IHEs,

· 3 serve rural populations,

· 1 serves urban populations, and

· 1 is a 4-year school that offers primarily associate's degrees.

*Baccalaureate colleges* are those where baccalaureate degrees represent at least 10 percent of all undergraduate degrees and where relatively few graduate degrees are awarded, and do not include IHEs that fall into the "tribal college" or "special focus" categories.  Within the 134 projects, there were 12 located at baccalaureate colleges – one project at each college.  Of these colleges,

· 6 offer degrees in arts and sciences,

· 5 offer degrees in several diverse fields, and

· 1 offers both baccalaureate and associate's degrees.

*Master's colleges and universities*include those that award at least 50 master's degrees and fewer than 20 doctoral degrees, and do not include IHEs that fall into the "tribal college" or "special focus" categories.  Within the 139 projects, there were 43 located at 40 unique master's colleges and universities.  Of these colleges and universities,

· 30 are larger programs (housing 32 projects),

· 9 are medium-sized programs (housing 10 projects), and

· 1 is a smaller program.

*Doctorate-granting universities*include IHEs that award at least 20 research-oriented doctoral degrees, but exclude both professional degrees (e.g., JD and MD) and special focus or tribal colleges.  Within the 134 projects, there were 78 located at doctorate-granting universities.  Of these colleges and universities,

· 24 are IHEs with very high research activity (housing 39 projects) – of those with IHEs with multiple projects, the highest numbers are located at

· University of Colorado at Boulder – 6 projects,

· University of Illinois – 4 projects

· Pennsylvania State University – 3 projects, and

· Brown University – 3 projects;

· 24 are IHEs with high research activity (housing 32 projects) – of those IHEs with multiple projects, the highest numbers are located at Kansas State University (6 projects); and

· 7 are IHEs that are considered doctoral and research universities (each housing 1 project).

*Tribal colleges* are controlled and operated by Native American tribes; they have become part of American Indians' institution-building in order to pass on their own cultures. The first was founded by the Navajo Nation in 1968 in Arizona; there are now 33 such IHEs plus 3 "associate" IHEs.  Of the 134 NPD projects, one is located within a tribal college.

**Part II:  Analysis of GPRA indicators from the Complete Data Reports**

All projects are to provide Complete Data Reports in the fall of their second through fifth funding years.  However, some of the 2007 cohort reported that they had no completers on whom to report and some were research-oriented projects to which the GPRA indicators did not apply.  Thus the 124 reports, which represent nearly 90 percent of the entire cohort, are considered to represent the outcomes with regard to the GPRA indicators.  

**Introduction**

**When project staff developed their proposals for funding, they generally anticipated providing educational opportunities for pre-service teachers (undergraduate students or paraprofessional educators seeking a teaching license, certificate, or endorsement to work with ELs), paraprofessionals seeking to meet their state's requirements to be "highly qualified," and/or in-service teachers seeking further knowledge and skills to improve their ability to work with EL students.  Grant applicants are not required to address services to all three types of participants.

The GPRA indicators now are somewhat different from those that were in-place when these IHEs wrote their proposals for funding.  There now are six GPRA indicators; projects report on those that apply to their projects.  Of these GPRA indicators, 3 are related to pre-service students who plan to become teachers of EL students, 1 is related to educational preparedness of paraprofessionals to work with EL students, and 2 are related to activities for in-service teachers (i.e., professional development leading to, or not leading to, certification to work with EL students).  See Appendix A for the data.

A total of 124 projects submitted reports; because projects could provide services to three different populations (paraprofessionals, pre-service teachers, and in-service teachers), counts are duplicative.  Thus across the 124 reports submitted, 61 reported a focus on pre-service teacher participants, 9 reported a focus on paraprofessional educator participants, and 107 reported a focus on in-service teacher participants.  In looking across all the projects, the minimum numbers of participants served (i.e., reported as participating in, though not necessarily completing or receiving any type of certification) each type of program were:

· 8,253 pre-service teacher education students, 

· 172 paraprofessionals,

· 3,174 in-service teachers in programs designed to provide some type of certification for work with ELs, and

· 4,087 in-service teachers in programs designed to improve their knowledge and skills but not to provide certification.

**GPRA Indicator #1.1:  The percentage of pre-service program graduates who are state and/or locally certified, licensed, or endorsed in EL instruction *during the reporting year*.**

Of the 61 projects reporting that they worked with pre-service teachers, all reported the number of students who participated, to some extent, in their NPD program during the 2010-2011 school year; fewer reported number of completers or number of completers with certification.

· The percentage of completers who received certificates during 2010-11: **42.9 percent**. 

· The number of *completers receiving certificate -* the total number who received a certificate was 1,354.

· The number of *program completers*- the total number reported was 3,154.

· The number of *program**participants* - the total number reported was 8,253.

**GPRA Indicator #1.2:  The percentage of pre-service program completers who are placed in instructional settings serving EL students *within 1 year of program completion.***

There were 58 projects that provided information about pre-service participants who completed their program during the year prior to the current reporting period – i.e., during the 2009-2010 school year; fewer project reported on the number of those who were followed and/or placed with EL students.

· Percentage of completers who were placed in programs serving EL students within one year of program completion: **81.3 percent**.

· The number of *2009-10**completers who were placed with EL students within one year* after completing their programs – the total number reported was 1,549.

· The number of *2009-10**completers followed by NPD projects for one year* - the total number reported was 1,906.

· The number of *completers during the 2009-10 school year* (58 projects reporting) varied from 0 to 640; the total number reported – 2,163.

**GPRA Indicator #1.3:  The percentage of pre-service program completers who are providing instructional services to EL students *3 years after program completion.***

NPD projects first funded in 2007 are not required to report on this GPRA indicator.

**GPRA Indicator #1.4:  The percentage of paraprofessional program completers who meet state and/or local qualifications for paraprofessionals working with EL students.**

Of the 9 projects reporting that they work with paraprofessional educators, all reported the number who participated, to at least some extent, in their NPD programs during the 2010-2011 school year; fewer reported on the number of completers and/or the number who met state or local qualifications to work with ELs.

· Percentage of paraprofessional completers who met state/local qualifications to work with EL students:  **50.0 percent**.

· The number of paraprofessional *completers who met state/local requirements* - the total number reported was 16.

· The number of paraprofessional *completers* - the total number reported was 32.

· The number of *paraprofessionals who participated* in some part of the program - the total number reported was 172.

· *Note*: One project (University of Indiana-T195N070021) stated that there are no state or local requirements for paraprofessional educators.

**GPRA Indicator #1.5:  The percentage of in-service teacher completers who complete state and/or local certification, licensure, or endorsement requirements in LEP instruction as a result of the program.**

A total of 107 projects indicated that they worked with in-service teachers during the 2010-2011 school year; of these, 83 reported that their program was designed to lead to certification or endorsement to work with EL students.  Fewer projects reported the number of program completers and/or the number who received certification.

· Percentage of in-service teachers who completed a program and obtained a certificate as a result of the NPD project:  **27.1 percent**. 

· The number of in-service teacher *completers who obtained certification* - the total number of in-service teacher-completers obtaining certification was 1,337.

· The number of in-service teacher *completers* - the total number of in-service teachers who completed a program, regardless of whether it led to certification to work with ELs was 5,077.

· The number of in-service *teachers who participated* in some part of a program that can lead to certification to work with ELs - the total number of in-service teacher-participants was 3,174.

· The number of in-service *teachers who participated in some part of a program*, regardless of whether it lead to certification to work with ELs – the total number of in-service teacher-participants was 7,261.

**GPRA Indicator #1.6:  The percentage of in-service teacher completers, *regardless of whether or not the program was designed to result in certification to work with ELs*, who are providing instructional services to LEP students.**

A total of 107 projects reported that they had developed programs for in-service teachers to expand their knowledge and skills.  Of these, 83 projects developed programs that were designed to lead to certification to work with ELs (see GPRA 1.5) and 58 projects developed programs that expanded teachers' knowledge and skills for working with ELs but were not designed to lead to certification; some projects reported developing both types of programs.  This GPRA indicator combines the results from both types of programs in order to determine how many in-service teachers completed a program and now are working with EL students (i.e., it combines data from programs designed to lead to certification and *not* designed to lead to certification).

Percentage of in-service teachers who completed a program offered within an NPD project and were in classrooms instructing ELs during 2010-11: **91.4 percent**.

· The number of *program completers placed with EL students*regardless of program type and regardless of whether they had obtained a certificate - the total number of in-service completers was 4,638.

· The total number of *program completers*was calculated by summing the number of program completers provided by 81 projects that were designed to lead to certification and the number of program completers provided by 58 projects that were not designed to lead to certification - the total number of completers was 5,077.

· The number of in-service *teacher-participants* in both types of programs - the total number of in-service participants was 7,261.

**Part III:  Executive Summaries**

This year, for the first time, each project was asked to provide an overall description of the first four years of their project in an executive summary.  In an effort to codify the information provided, each was given a template.  In following the template, projects reported information in specific categories:  Program description, cumulative numbers of each type of project completer, project outcomes, assessment of participant learning, challenges, success stories, and so on.  A total of 125 completed an executive summary; some projects provided detailed information, some provided very little, many listed challenges they had faced and provided success stories.

In order to review the executive summaries in a more structured manner, the work of Thomas Guskey was reviewed.  Guskey has developed a five-level model for the evaluation of professional development programs.  The five levels are as follows.

1. Participants' reactions to the program – did they like the program, will the program be useful to them in their careers?

2. Participants' learning – did they acquire the intended knowledge and skills?

3. Organizational support and change – what was the effect of the program on the IHE, was implementation of the program advocated and facilitated?

4. Participants' use of new knowledge and skills – did participants effectively apply the new knowledge and skills that they gained as a result of the program?

5. Student learning outcomes – was there an effect on the performance or achievement of participants' students?

A coding protocol was developed based on these five evaluation levels, and looking for the questions addressed and how the information was gathered.  Data were coded as written by grantees, and interpreted by NCELA staff with regard to Guskey's evaluation levels.  For instance, it was not necessarily clear whether the Praxis was used to assess participant learning or to demonstrate participants' use of knowledge or skills.  The information obtained is summarized, according to the five evaluation levels, in this section.

Also in this section are reported example grantee comments that support the numeric data.  These comments have been selected as being "typical" (i.e., representing the most frequently reported response) or as being more unusual in the program's approach to an area.  NCELA staff have selected these comments from as many different grantee reports as possible in order to provide an overall "feel" for these NPD projects.  The last four digits of the grant award numbers are provided at the end of each comment as well as the type(s) of participants served – pre-service and/or in-service, the latter divided into certificate-seeking, and/or not-certificate-seeking if possible.

**Participants' reactions**

A total of 49 projects (39.2%) commented upon participants' reactions to the program.  Of these projects, 41 developed and administered surveys (some of which were administered pre/post) to determine whether participants felt the program had met their needs, increased their knowledge and skills related to ELs, and/or overall felt the program was worthwhile.  Most participants provided positive feedback; 24 of these projects indicated that at least 75 percent of the participants enjoyed the program and felt that they had gained valuable knowledge and skills.  Of the seven projects that indicated they had assessed participants' reactions using other means, most had used focus groups or one-on-one interviews.  It appears that projects are seeking participants' reactions and have data to support the value of the program to participants.

The following comments were taken directly from grantees' executive summaries.  

· Satisfaction surveys completed after every PD session were consistently strong with all sessions rated excellent or good by all levels of participants (IHE faculty & LEA in-service [certification-seeking] teachers).  ...  Focus groups of IHE faculty and LEA in-service teachers conducted in Spring 2009 and Spring 2011 indicated high satisfaction with knowledge gained from TESOL for ALL PD.  IHE faculty especially valued the learning community established over their 2-year PD sessions. (0291, serving pre-service and in-service certificate-seeking)

· One of the unique elements of the CPELL model involves the use of an iterative process.  This process is used within each of the partner school districts as well as within each leg of the program.  The process begins with listening to the administration, teachers, and parents within the district.  ...  Feedback forms and evaluations are collected from the participants at each administrator, teacher, and parent session, providing CPELL with real-time data for shaping the next session.  (0359, serving in-service teachers both seeking and not seeking certificates)

· All [IHE faculty] fellows indicated that they were satisfied with the program and could offer no substantive ways to enhance its quality.  Three respondents from different universities especially enjoyed the collaborative aspects of the program, the group meetings and bi-weekly study groups, as the best way to develop and enhance their knowledge.  (0249, serving IHE faculty who will be teaching pre-service teachers)

**Participants' learning**

****A total of 74 projects (59.2%) included information about participants' ongoing learning experiences and how they were assessed or reported.  Of these projects,

· 31 described assessments administered to students 

· 28 stated that self-reflections in journals or responses to surveys or questionnaires demonstrated gains in knowledge and skills,

· 17 listed demonstrations (videos, classroom observations, and other means of actively demonstrating gains in knowledge) that showed improved participant skills when working with ELs in a classroom,

· 15 described portfolios of some type that participants created to demonstrate their increased skills and knowledge, and

· 25 projects listed other means of demonstrating skills and knowledge such as GPA, participants' grades in relevant classes, coaching notes, tutors' or mentors' comments, self-assessments, classroom observations, and pre/post changes in attitudes or beliefs about ELs and language acquisition.

As over half of the projects provided information on measuring participants' learning, it is clear that this is an area of great importance.  Several projects described in some depth the progress participants were making.  Also, while projects typically used one or two methods for determining whether participants were learning in their programs, some projects used as many as five different methods.  A sample of these descriptions follows.

· ALAS monitors the development and success of the program through formative and summative evaluation.  The program collects data via both direct and indirect measures and in a cyclical manner throughout the two years of the program assessing participants' skill acquisition and teaching practices.  **Formative evaluation**provides information regarding the progress of pre-service teachers in meeting required certification skills, and their dedicated engagement in the ALAS program.  Ongoing collection of information using required teacher performance expectations (TPA's) provides feedback needed to intervene, enhance or further provide assignments, and make program curriculum changes and alterations during implementation over the grant period.  Assessment reports are monitored each semester.  (0205, serving pre-service teachers seeking dual certificate for ELs and Special Education)

· In Project Year 2, the evaluator conducted qualitative analysis of student work products and reflections to assess student growth and measure the complex knowledge gained.  It was learned that: (1) All students were able to discuss course concepts fluently demonstrating expanded expertise.  Topics included comparisons between SIOP and CALLA models for supporting ELLs in the classroom, incorporation of the WIDA Standards into classroom learning objectives, and implementing the "Collaborative Assessment Conference" format for analyzing student work samples.  (2) Teachers actively engaged with the content and discussed their learning as bringing about significant changes in their practice, but also in their orientation to their students.  (0248, serving in-service teachers)

· The Academic Literacy for All project has collected data on participant learning outcomes in a number of ways, the primary means being the reflections of participants after they have been engaged in an ALA activity, whether it be the ALA seminar or summer institute, the ALA workshops, the ALA protocol in the secondary education classes, or the ALA study groups.  Using NVivo software we have analyzed and continue to analyze the participants' reflections and have created a database that we have used to write articles and give presentations.  This information is also used to shape ongoing ALA activities.  (0259, serving non-certificate-seeking in-service teachers) 

Participant learning is one area in which Guskey himself suggests several different ways to collect information.  It is clear that these projects have done just that.  They report a variety of methods for determining participant growth.  The means for gathering information, however, is less important that the fact that data are being gathered, and often used to modify and improve the content and/or delivery of knowledge to participants.

**Organizational support and change**

****The area in which the greatest number of projects provided responses was IHE support and change – 83 projects (66.4%) provided information about ways in which their own institutions supported the project and made changes because of the projects.  Of these projects,

· 61 reported changes to the curriculum (major revisions to coursework as well as new course offerings),

· 39 promoted additional training and professional development for their own faculty,

· 39 described a strong partnership or a working relationship with one or more local school districts,

· 30 listed syllabus changes (including minor adjustments to coursework, additional assignments or field work, new instructors, and so on),

· 18 reported increased field practice for participants, with attendant supervision by faculty members;

· 12 focused on faculty's follow-up activities with program participants, and

· 9 indicated other activities such as a partnership with one or more SEAs, increased faculty collaboration, financial support for student participants, course evaluations, and cost-sharing with the project.

The projects that reported on their work within their own IHEs most often reported changing curricula and syllabus changes.  These changes do not necessarily reflect a change in the organization as a whole, but does indicate support by their own unit within the IHE.  The typical project provided two or three examples of IHE support for the NPD project, but some projects listed as many as five different areas in which the IHE was supportive.  Selected explanations or comments are provided below.

· Our first major project goal has been to improve the [Boston College] LSOE Teacher Education Program in order to better prepare all teachers to provide effective instruction to English Language Learners (ELLs).  Activities in this area have included:

· Faculty development in the area of literacy and ELLs

· Development of an undergraduate course on bilingualism and ELL student issues

· Development and refinement of required pre-practicum activities for all licensure candidates in Elementary Education, Early Childhood Education and Secondary Education participating in a pre-practicum experience each semester

· Collaboration with practicum office in development of lesson plan templates requiring explicit content and language objectives

· Training for supervisors to support the teaching of academic language in practicum sites

· Infusion of theory and strategies to teach academic language in secondary History and Math methods courses as well as the Secondary Curriculum and Instruction class for undergraduate secondary program students  (0133, serving pre-service teachers)

· The objective of improving [the University of Missouri-Kansas City]'s teacher education programs in order to better prepare all faculty, teachers and teacher candidates to provide effective instruction to LEP students was responsive to the program's invitational priorities at the time of funding.  As [the next section shows], the university has addressed this objective by revising nearly all teacher education syllabi to include topics related to ELL education, has involved all teacher education faculty in seminars regarding ELL issues during each year of the project, and has added new courses related to ELL education.  [The next section] details the current year's activities, including faculty meetings on the newly adopted state English proficiency assessment, the WIDA ACCESS, and a panel of refugee parents.  (0316, serving certificate-seeking in-service teachers)

· As originally proposed, the program planned to deliver its services to four cohorts of 15 candidates each.  However, the delivery model was revised to include seven cohorts with fewer candidates in the first few cohorts.  This enabled the program faculty to focus on refinement of the course presentation and content during the early phases of the project.  (0189, serving certificate-seeking in-service teachers)

· Ongoing meetings were completed with representatives from the [three] participating school districts, representatives from Kansas State University and individuals from the Kansas Department of Education.  Two meetings occurred to incorporate inservice, professional development program refinements for [in-service certificate-seeking] teachers' instructional practices with English language learners.  (0349, serving certificate-seeking in-service teachers)

· Faculty cohort members report that they have made considerable changes to their coursework, adding modules focusing on the needs of English language learners, changing the textbooks they use to make them more authentic and/or explicit regarding ELLs, service learning components have also been added or adapted for several courses to more adequately address ELLs in the schools.  Additionally, the assessments for the courses have also been adapted to include a focus on the ELL content that has been introduced, thus making it a more permanent part of their teacher preparation curriculum.  (0324, serving in-service teachers)

**Participants' use of new knowledge**

****A total of 54 projects (43.2%) described means of measuring participants' summative gains in knowledge and skills and, more importantly, whether they were using these newly learned skills in the classroom.  Thus while the *participants' learning*looks at what the participants were learning in their IHE classes, *participants' use of new knowledge*looks at whether the new knowledge is put to use during field experience or in participants' own classrooms.  Of these projects,

· 23 reported that classroom observations had been completed,

· 21 indicated that completers had "passed" a state-required test,

· 16 listed some type of self-report measure such as a questionnaire, a survey, or a "self- assessment,” often used in a pre/post fashion (e.g., at the beginning of the student's classes and after completion of all the student's classes, or at the beginning of a semester and end of a semester),

· 10 projects interviewed completers, 

· 7 indicated that completers had "passed" a locally-developed (by the project or by the school district) assessment, and

· 11 reported other types of measures such as student work, summative GPA, receiving a teaching credential or ESL/bilingual endorsement, principal evaluation of the participant's teaching, videotaping of a teaching session, and becoming certified by the National Board for Professional Teaching Standards.

While fewer projects provided information regarding participants' use of their new knowledge, those that did provide the information typically listed two or three ways in which they could demonstrate participant skills.  Some examples are provided below.

· Pre-service teachers are assessed in their student teaching experience for four semesters (two under biliteracy and two under special education) using the CSTP standards that include: (1) making subject matter comprehensible/meaningful to students, (2) assessing students learning, (3) engaging and supporting student learning, (4) planning instruction and designing learning experiences for students, (5) creating and maintaining effective environments for student learning, (6) developing as a professional educator, and (7) linking with the school community and serving as a mediator of culture.  (0205, serving pre-service teachers seeking dual certificate for ELs and Special Education)

·  [At the end of the Year 3 and Year 4,] the workshop participants were also asked to evaluate how these workshops have helped their professional development. Over 90% of the participants agreed or strongly agreed that the workshops increased their knowledge and skills in their areas of certification, endorsement, or teaching assignment and the relevance of the workshops to ISBE teaching standards was clear.  (0300, serving pre-service teachers, and in-service teachers both seeking and not seeking certificates)

· Data on participant learning:

· 100% passed CTEL and CSET: LOTE exams by end of their four-year sequences

· 100% completed each program course with grade of B or better

· 100% passed PRAXIS II national school psychology exam

· 100% of graduates endorsed as Nationally Certified School Psychologists (NCSP)  (0280, serving those seeking certification as school psychologists and EL professionals)

**Participants' effectiveness**

A total of 47 projects (37.6%) reported various means for measuring completers' effectiveness in the classroom.  This area looked at whether student outcomes were improving, presumably as a result of the effective use of strategies, knowledge, and skills learned during the NPD project and implemented in the EL classrooms.  Of these projects,

· 19 projects reviewed student records to determine teacher effectiveness in the classroom,

· 13 projects developed questionnaires to ascertain what the teacher, parents, school administration, or students believed the students' performance had improved,

· 8 looked at school records to determine whether specific classrooms had shown improvement from previous the year(s),

· 1 used an interview format, usually with structured interviews, to determine the types of changes the in-service completers saw in their students, 

· 1 reviewed portfolios prepared by in-service completer-teachers of their students' work, and

· 15 reported other measures of effectiveness, or described the types of instruments used, to determine teacher effectiveness.  These projects reported that effectiveness was measured through the use of classroom observations (including multiple observations across more than one year), looking at Title III AMAOs for the school, surveys completed by students, higher graduation rates, using completers as trainers in a trainer-of-trainers model, information from the state education agency, whether in-service completers had received an endorsement to work with ELs, focus groups of stakeholders, meetings with the in-service completer(s), and recognizing school(s) no longer in probationary status.

While fewer projects reported on teacher effectiveness, those that did provided some level of detail and give a view of how these projects defined teachers' "effectiveness."  A few examples are provided below.  In looking at these examples, "effectiveness" has a range of definitions from obtaining the necessary endorsement to work with EL students to continuing with professional development activities to improving student outcomes and receiving teaching awards.

· ALAS has followed 60% (17 of 28) of completers who are teaching in the regional service area of San Diego County and 40% (11 of 28) by phone and mailings.  They have provided the project with data using a SWOT Process (Strengths, Weaknesses, Opportunities, Tensions) that they have experienced as they engage in their work with ELLs in both bilingual and special education settings.  Focus groups have been undertaken to seek clarity on what they two-credential program needs to enhance to address the professional needs of graduates from ALAS.  Participants have also provided direct feedback on how each department (bilingual and special education) can strengthen its curriculum and training.  California Dataquest is used to identify the growth of ELL student outcomes in Language Arts, Mathematics and Language Acquisition via the CELDT state assessment.  (0205, serving pre-service teachers seeking dual certificate for ELs and Special Education)

· All participants were followed up by Project STELL Director through surveys, school visits, and emails.  The purpose for the follow-up was to see whether the pre-service completers have obtained the professional teaching certification and endorsement in teaching ELLs and in-service completers have received the endorsement.  Also, the follow-up was to see whether they are providing instructional services to English Language Learners

Among 15 pre-service completers, 14 received a Type 09 secondary teaching certification and one a Type 10 special K-12 certification in Spanish.  They all obtained the bilingual/ESL endorsements or approvals.  13 out of 15 pre-service completers are providing instructional services to LEP students although not all of them have secured a full-time teaching position due to the tight job market.  ...  All in-service teachers obtained professional teaching certifications before their participation in the project and are still licensed to teach.  Among 34 completers, 28 have received bilingual/EL endorsements or approval from the state of Illinois.  All completers work in schools with a high concentration of ELLs ranging from 8.4 to 47.6% with an overage of 16.5%.   (0300, serving pre-service teachers, and in-service teachers both seeking and not seeking certificates)

We have several ways to track our graduates.  In what follows, we will share information that has emerged from follow-up surveys with participants, their mentees, and their campus administrators; monthly reunion meetings; internet professional group interactions (e.g., Facebook, email list serves); and ongoing collaboration on research and curriculum projects between graduates and UT faculty and staff.

We hold monthly follow-up meetings with program graduates.  Meetings are held in a range of settings after school, including graduates' classrooms, on campus at UT, and in local cafés or restaurants.  Monthly topics, selected collaboratively and designed specifically to support graduates' leadership work, have included:  "mentoring challenges"; "moving into leadership roles"; "practicing for presentations at TABE, NABE, and Literacy Research Association"; "The Role of Advocacy for Bilingual Teacher Leaders"; "developing dual language classrooms and program;" "Supporting positive identities for bilingual children in school."  When meetings are held in graduates' classrooms, hosts have done tours of their schools and shared the work they are dong on the campus.  These monthly meetings build bridges across the cohorts and provide a space for continued professional sharing and interaction with university faculty.  They have also allowed us to document and evaluate our graduates' participation in leadership activities.  Attendance has ranged from 2 to 11 of the 32 Proyecto Maestría program graduates, with 20 of the 32 attending at least one meeting.

All of our graduates are taking on leadership roles in a wide variety of ways. [Ten examples are provided.]  For each cohort, approximately one school year after they graduated we surveyed their principals.  ...  Mentoring has been a particular strength of our graduates.  ...  Graduates have continued to collaborate with UT facult7y and staff on a number of exciting projects. ...  It is a tribute to the powerful work our [project] teacher leaders are dong that a number of our teachers have won significant teaching awards.  (0272, serving in-service teachers both seeking and not seeking certificates)

· Data for the Colorado Student Assessment Program (CSAP), given each year to students, is recorded for each teacher who has completed the program.  Since 2007 longitudinal data are tied to the endorsed teacher. From the four yeas of longitudinal data, in the areas of reading, math, and writing, an analysis of a 5 point movement of progress in **Meeting the Standard** [original emphasis] revealed that ELL students, whose teachers received instruction on language acquisition and obtained the Linguistically Diverse Certification, did better showing a steady, positive growth pattern.  It is anticipated that the skills acquired by these grant teacher-participants will continue to impact future learning achievement for ELLs in their care.  For the current granting period, data from the August 2011 CSAP revealed that the participating teachers 421 ELL students maintained the growth trend of 53% reading growth, 38% math growth, and 53% writing growth.   (0091, serving in-service teachers both seeking and not seeking certificates)

**Challenges**

****Well over half of these projects (74, 59.2%) reported that they had faced challenges, with 71 identifying one or more challenges and 28 reported that the challenge(s) had been met and "fixed."  There were several general areas that challenged projects, as well as some very project-specific topics. The three largest areas of concern were recruitment of qualified applicants, funding issues, and changes in project staff; other challenges included geographic issues, retention of participants, issues related to partnering school districts, changes in state certification requirements, evaluation/evaluator issues, and problems with determining teachers' effectiveness.  A list of challenges, and the number of projects reporting each, is reported in Table 1.

Table 1:  Ten challenges most frequently listed by 71 projects

By far the most frequent challenge, listed by more than one-third of those who indicated they had met challenges, was **funding** or a more generally stated "resources."  In some cases (*n*=6), funding itself was mentioned:  delays in funding, cuts to funding, limited funding, and identifying on-going funds.  In a greater number of cases (*n*=21) the challenge was stated as "economic."  These **economic challenges** included tuition increases at the IHE; budget cuts at the state, the IHE, the school district, and/or the school; and economic challenges faced by the project participants.  As one project stated, the "economic crisis reduced [our] capacity to achieve stated goals."  On the other hand, this is one area in which there was some success in meeting the challenge.  Projects commented that they had solved issues by adding a pre-service component to their project, moving completers into leadership roles at their schools, identifying additional funding, creating partnerships within their communities, using "community-based pedagogies," demonstrating success with the project as a whole, and newspaper coverage of the project.

The second largest set of challenges (*n*=19) dealt with **recruiting**(listed by 13 projects) and **retaining** (listed by 6 projects) participants.  Most did not explain this more fully, although some indicated that the problem was with recruiting qualified applicants, recruiting students who "never thought they could have the financial means to become teachers," "in-service teachers committing to 3 courses" at an IHE, difficulty recruiting principals, and "recruiting and retaining teachers in program is hard because of the economic downturn."  On the other hand, some projects commented on their success in recruiting participants and indicated that they were having difficult "**meeting teacher demand**."

Another set of issues dealt with **changing****school staff**(*n*=15).  These included recruiting staff at the individual schools; teacher, principal, administrator, and "leadership" turn-over; "RIFs" at the local school; changing teacher assignments; and visa problems for some teachers.  Some projects noted that while these personnel changes were challenges, they were addressed by working with the new personnel.  Others included their work with partnering schools as a success for the project, stating that "partnerships were created," they had "increased collaboration despite limited funding," and that there were "positive on-going relationships with public schools" and "increased school-university collaboration."

While the first three sets of challenges are issues that projects might expect to deal with on a fairly regular basis, the next challenge is one that is much less under their control:  **state-level changes in certification requirements**(*n*=10 comments).  This challenge to projects included issues such as changes in certification requirements, the test used for certification purposes, ESOL standards, and ELP assessment as well as changing state policies affecting enrollment of students and changes to the required teacher-to-student ratio. 

The next set of challenges related to project-based issues:  **IHE issues** and **changing project staff** (*n*=8 comments for each of these two challenges).  The issues with the universities included lack of faculty buy-in to the NPD project as well as needing to provide professional development for the faculty, courses that were not available when needed for project participants, departmental (or college) reorganization, and lack of time with the university-assigned principal investigator.  As the projects are located within IHEs, we include in this general rubric those that are related to project staff.  The primary areas of concern were changing staff (project director, project evaluator, and issues related to conducting the evaluation - e.g., under-funded evaluation, consistency of the evaluation across project years, and a need for "more qualitative information" in the evaluation).  Also related to the evaluation of the project is the next challenge:  difficulties in **measuring participant performance and effectiveness** (*n*=6 comments).  This challenge included issues such as the timing of the state test, test results not being available to the grantee, and difficulties in comparing participants who taught during the regular school year with those who taught in a summer school session.  Four projects indicated that **maintaining contact with graduates** was difficult and two projects mentioned that **completers had not submitted the required paperwork to receive certification**.  These two issues also may be related to measuring participant performance.

In addition to project staff changes and school-site staff changes, these NPD projects also were affected by **school district issues**(*n*=6) including staffing changes, administrative changes, collaboration with the school district, redistricting, and reaching the "saturation" level for administrative professional development.

Finally, there were some **geographic issues** noted (*n*=5).  These primarily dealt with the remoteness of the project (e.g., internet access in the Polynesian islands) and/or of the participants being served (e.g., participants requesting weekend or teleconference courses).

Other, less commented upon (by 1-2 projects each), challenges included the following:

· Teachers resisting efforts to change,

· Data collection problems,

· Problems using Blackboard,

· Administrative structure of the grant,

· Teachers moving to the suburbs rather than staying in Chicago (where the project is located) and teachers who do not want to move to a new location to find a position,

· New guidelines and policies for implementing the NPD project,

· Lack of continuity of professional development activities,

· Non-traditional (older) students,

· Creating teacher teams and coaching partnerships, and

· Time constraints.

A few examples of projects' direct descriptions of challenges are provided below, with the last four digits of the grant award numbers and the type of participants served at the end of each example.

· Although the main focus of the grant was to recruit in-service teachers in secondary content areas the number of these teachers applying for the grant was less than anticipated.  During the operation of the grant period teachers and administrators in the field reported that secondary content teachers were reluctant to pursue the ESOL endorsement because they did not want to be the school's ESOL teacher and have all ELL students placed in their classes.  During the operation of the grant pre-service teachers seeking entry in the teaching profession, were more likely to be recruited than older in-service teachers less interested in improving their practice.  For this reason, the grant continued to recruit in-service teachers finding we were more successful, however, in recruiting pre-service and other content-area in-service teachers.  (0178, serving pre-service and both types of in-service teachers)

· It should be noted that since 2009, the recession has greatly burdened PHUSD teachers and teacher families financially.  For some cohort teachers as well as perspective applicants to the program, completing the ACCESS program competed with family finances; and for this reason, it has become more challenging to recruit and retain participants to complete the program.  This had a direct result on program recruitment.  Recruitment activities were expanded and it was decided that a target goal of 18 participants or 75% would be a more realistic baseline with the intention of reaching the grant's pre-established 24 participant target.  (0142, serving in-service teachers seeking certificates)

· The California economy continues to decline (unemployment and housing) while college tuition continues to increase, on an annual basis in the California University System.  While tuition costs are rising significantly, savings have been accrued due to participating ALAS candidates receiving federal student aid.  Yet, the condition of the economy and the labor intensive emphasis of the program, the pre-service candidates have been pressed by the cost of living conditions (e.g., housing, gas, food) and California credentialing exit tests and performance requirements (RICA & PACT).

Given the cut-backs in higher education, over 25% over the last three years, courses during the summer have been eliminated holding ALAS students back in meeting entry course (six courses) requirements necessary as foundational for the bilingual and special education credential.  Thus it has taken longer for students to enter the program.

Our College of Education is being reorganized and is challenging both the Policy Studies Department responsible for the bilingual credential and the Special Education Department responsible for the Mil-Moderate Specialists credential to have positional influence to assure that the ALAS dual credential program is institutionalized. (0205, serving pre-service teachers seeking dual certificate for ELs and Special Education)

**Successes**

****Many projects offered success stories (46, 40.4%).  There was quite a range of these from general information such as the number served to a statement that the project was meeting its objectives and goals, to specific examples of an individual's experience or longer-term effects of the project.  Most were quite brief – three to four sentences.  A few examples are provided below, with the last four digits of the grant award numbers and the type of participants served at the end of each example.

· As a result of TESOL for ALL, we have established positive, on-going relationships with local public schools, non-profit organizations, and community colleges.  Through such collaborations, our teacher candidates have more and more practicum opportunities to interact with English learners through volunteer work, part-time jobs, and full-time teaching positions at K-16 levels.  These practicum opportunities better prepare our teacher candidates to be career-ready, and they also strengthen the quality of ESL programs offered in the local community.

    Strong university-school collaboration with schools in our partner districts, Smith High School, and other schools in the region, also leads to a more interactive, sustainable, and reciprocal ESL PD model because university teacher educators facilitate in-service teachers' development of knowledge and skills in working with English learners and their families.  At the same time, in-service teachers apply what they learn and serve as role models for pre-service teachers.  The support of LEAs for the teacher leadership projects also leads to better preparation of ESL teacher leaders in K-12 settings.  (0291, serving pre-service and in-service teachers seeking certificates)

· A Teacher Talks about LDCD:  http://www.youtube.com/watch?v=svNkufBdKMI

Other teacher comments:

http:www.youtube.com/watch?v=EQ_pIs8DlbE&feature=related

One of our major successes has been with the math and science teachers who have completed the sequence of three LDCD courses.  Through interviews, class discussions, and final portfolio projects they have demonstrated an increase in their implementation of effective classroom practices for English language learners.  Following are some selected quotes from math and science teachers from Dr. Carol Bearse's LDCD classes over the course of the project.

"Although it is a math class, there is so much language.  Words that seem basic such as *evaluate, solve, equation, operation, less than, more than*, etc., are actually words that are key to doing mathematics."  Math Teacher

"During my coursework, I began to see very clearly the importance of direct language instruction, both academic and social, in my classroom. ... I also have committed myself to focusing on vocabulary in my classroom without watering down the mathematics.  This has required a lot more planning and a great deal of differentiation with the mathematics at hand and the instruction for each activity. ... The results are very positive.  My last class of beginning ELL students outperformed most of the ELL and non-ELL peers on the state math exam."  Math Teacher

(0255, serving in-service teachers not seeking certificates)

· Data collected to date across the four years of the program clearly indicates that Loyola has demonstrated how to be a good partner with community school districts.  The strength of the program has been the partnering of solid instruction and resources mixed with tailored plans and adjustments.  Also seen within the program is the ownership within the most successful partner districts.  Administrators and instructors believe in the approach and the program and how it helps their community schools. ...  CPELL will seek future opportunities to implement this model in additional districts and maximize impact at current schools.  Additionally, the CPELL team is interested in sharing challenges and best practices, and training on this proven model in order to informer [sic] other institutions and community programs interested in developing effective school-university partnerships.  (0359, serving in-service teachers both seeking and not seeking certificates).

· Over the course of the project, changes in school or system level personnel and resulting changes in curriculum and assessments have been an enormous challenge to the project.  A major budget crisis resulted in many excellent teachers losing their jobs and support for teacher morale was very important.  READwELL's consistency during this time, the program's ability to support programming and encourage teachers was very important.  Providing materials for a project, recognizing the time and effort of teachers and developing informal, collegial relationships with teachers help to keep key elements of the program going.

This groundwork of relationship building was key when CPS changed Mather's entire school structure to small learning communities in 2010.  The 2010-11 school year was an intensive year of planning for all teachers and administrators as they developed the curricula and focus for each of these communities.  READwELL participated and advised in this process.  The planning year was extremely collaborative around a shared set of goals.  This climate of collaboration and group work was extremely different than was seen in earlier years with the school.  In each curriculum development group, addressing the needs of ELL students was considered to be a high priority by teachers.  This is a marked change in teacher expectations from our early work with the school.  (0288, serving in-service teachers both seeking and not seeking certificates)

**Summary/Conclusions**

A total of 132 projects, of 139 funded projects, provided a Complete Data Report and/or an executive summary for the 2010-2011 school year.  Projects were asked to provide executive summaries.  One of the common themes in the executive summaries was IHE-project interaction.  These projects often reported change in curricula or syllabi due to the project's work, and indicated that the IHE supported the project in a number of ways.  One issue commented upon by few grantees, but which did affect all grantees, has been the changing GPRA indicators themselves.  These grantees are reporting on GPRA indicators that have changed since they wrote their applications for funding.  Given this, the number of grantees that are able to provide all the information requested is impressive.

Of the 139 funded NPD projects, 32 projects were located in minority-serving institutions enrolling large numbers (at least 10%-25%) of Hispanic, African-American, Asian-American/Pacific Island, and/or Native American participants.  As described by the Carnegie classification codes, these IHEs (1) have both undergraduate and graduate colleges, schools, and programs, (2) serve more undergraduate students than graduate students, (3) are selective four-year institutions with larger master's programs, and (4) are research-oriented.

A total of 17,849 pre-service students, paraprofessionals, and in-service teachers participated in various programs designed, or not designed, to lead to licensure and/or certification – but all designed to increase the knowledge and skills of the participants in order to serve EL students more effectively.  Because of the length of many of the courses, it was not anticipated that all participants could or would complete their programs within this one reporting year.  The percentages of completers, variously defined as completing coursework, obtaining a license or certification, and/or being placed with EL students, do not provide an accurate reflection of the projects' accomplishments because (1) these reports reflect one year of the project (plus one year of follow-up with pre-service teacher completers) and (2) the percentages are based on raw numbers.  As an example, a project might report 100 participants, but not mention completers at all.  Although this is different from reporting 100 participants with no completers, both reports result in the same calculation:  0% completers.  A more accurate percentage would be based only on projects that reported actual data in each category needed for a particular GPRA indicator.  Regardless, when breaking down the percentages into finer detail than the GPRA indicators require, some of them are impressive.

· at least 81.3 percent of pre-service completers do obtain their license/endorsement/certification(s) and are working with EL students within one year; 

· at least 69.9 percent of in-service teachers who enter a program to increase their knowledge and skills (regardless of whether the program is intended to result in additional certification), do complete the program;

· at least 76.8 percent of in-service teachers who enter a program with the specific intent of obtaining additional certification/endorsement actually do so;

· at least 81.6 percent of in-service teachers do complete their programs even though the end-goal is to increase knowledge rather than further certification; and

· at least 63.9 percent of in-service teachers who complete a program are working in classrooms with EL students.

�	Information from the Carnegie Foundation website, retrieved from � HYPERLINK "http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CD4QjBAwAg&url=http%3A%2F%2Fclassifications.carnegiefoundation.org%2Fdescriptions%2F&ei=Qz97T_XiOKX10gHluoyWBg&usg=AFQjCNGtlqt5Fg7zCvt-WXoVg1xVNiv4-w&sig2=V6Wg4K0mHVqXtjdek8E7EQ" ��		� HYPERLINK "http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CD4QjBAwAg&url=http%3A%2F%2Fclassifications.carnegiefou%09ndation.org%2Fdescriptions%2F&ei=Qz97T_XiOKX10gHluoyWBg&usg=AFQjCNGtlqt5Fg7zCvt-WXoVg1xVNiv4-%09w&sig2=V6Wg4K0mHVqXtjdek8E7EQ" �http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CD4QjBAwAg&url=http%3A%2F%2Fclassifications.carnegiefou	ndation.org%2Fdescriptions%2F&ei=Qz97T_XiOKX10gHluoyWBg&usg=AFQjCNGtlqt5Fg7zCvt-WXoVg1xVNiv4-	w&sig2=V6Wg4K0mHVqXtjdek8E7EQ� 

� 	The GPRA indicators refer to individuals who were "state and/or locally certified, licensed, or endorsed in LEP instruction dur-	ing the reporting year."  For ease of communication, we will use the terms "certificate" or "certified" indicate "certified, li-	censed, or endorsed in LEP instruction during the reporting year – i.e., in place of the GPRA definition.

� For further information, see "A Conversation with Thomas R. Guskey," in C.J. Casteel & K.G. Ballantyne (Eds.).  (2010).  Professional          	Development in Action:  Improving Teaching for English Learners, pp 91-96.  Washington, DC:  National Clearinghouse for English Language 	Acquisition.  Retrieved from � HYPERLINK "http://www.ncela.gwu.edu/files/uploads/3/PD_in_Action.pdf" �http://www.ncela.gwu.edu/files/uploads/3/PD_in_Action.pdf�   

NCELA is operated under contract ED-04-CO-0094/0002 from the US Department of Education to The George Washington University.  Our mission is to provide technical assistance information to state education agencies, local education agencies, and others regarding the education of English language learners.

2011 Reporting on 2007 Projects