# Presentation: 375943

## The Long and Winding Road

**The Relationship Between Leadership Practice and Student Performance**
**Jonathan Supovitz**
**Philip Sirinides**
**University of Pennsylvania**
**Consortium for Policy Research in Education**
**June, 2008**
**IES Annual Conference**

**This research is supported by the Institute of Education Sciences, grant # R305C050041.  The opinions expressed are those of the authors and do not represent the views of the U.S. Department of Education.**

## Study Overview

**The literature on how school leadership improves student outcomes stretches back at least 40 years. **
**Initially, studies of school leadership focused on the role and influence of school principals.**
**More recently, researchers have started to explore the role and influence of other school actors as well.**
**In this study we seek to examine the relative influence of different school actors on instructional practice and student outcomes.**

## Literature Review on School Leadership and Student Outcomes

**Several comprehensive literature reviews of the effects of principal leadership on student outcomes.**

**Hallinger & Heck (1997) examined 43 studies conducted between 1980 and 1995. Found little evidence of direct effects, with most evidence pointing to indirect effects. Concluded that principals have a measurable but indirect effect on school effectiveness and student achievement. **
**Leithwood, Seashore Louis, Anderson and Wahlstrom (2004) conducted a wide-ranging analysis of educational leadership and concluded that leadership “is second only to teaching among school-related factors in its impact on student learning”. **

**Notes:**
Hallinger & Heck (1997) examined 43 studies conducted between 1980 and 1995 that examined evidence of the relationship between principal leadership and student achievement. 
Direct effects of leadership practice on student outcomes
Mediated effects studies, in which principal leadership was mediated by other people, events, or organizational factorsR
Reciprocal effect studies, in which the relationships between leadership efforts and school and environmental factors were interactive. 
Little evidence of direct effects, with most evidence pointing to indirect effects, 
Concluded that principals have a measurable but indirect effect on school effectiveness and student achievement.

## Literature identifies several important principal leadership characteristics 

**● ****Instructional mission for school (Goldring & Pasternak, 1994; Hallinger, Bickman & Davis, 1996).**
**● ****Encouraging trust and collaboration (Fink & Resnick, 1991; Blase & Blase, 1989)**
**● ****Hands-on support of instruction (Hallinger & Murphy, 1987; Supovitz & Poglinco, 2001)**

**Notes:**
Much of this work places different levels of emphasis on one or more of three different aspects of instructional leadership: creating a learning ethos (mission, vision) for the school and managing or developing the instructional program for the school; creating a supportive environment for instructional improvement, and providing direct, hands-on support for teachers’ instructional practice. 

## Growing Literature on Range of Leadership Activity in Schools 

**Individuals at all levels of schools influence reform efforts (York-Barr & Duke, 2004) **
**Distributed leadership perspective (Gronn, 2000; Spillane 2006)**
**Interest in and attention to teacher leadership (Mangin & Stoelinga, 2008)**

**Notes:**
A second, increasing researched strain in the educational literature is the range of leadership activity in schools. There is an acknowledgement that individuals at all levels of the school play a role in reform. This is represented in the literature on distributed leadership and the growing body of research on teacher leadership and informal leadership.  

## Attributes of teacher influence on peers identified in the literature 

**● ****Instructional ****conversation amongst peers (Silva, Gimbert & Nolan, 2000; LeBlanc & Shelton, 1997) **
**● ****Active interaction amongst faculty around teaching and learning (Wasley, 1991; Smylie & Denny, 1990)**
**● ****Strong instructional advice networks (Frank et al. 2004; Supovitz, 2008)**

**Notes:**
This research points to several dimensions of peer influence. First, there is some evidence that progressing schools have lots of instructional conversation, active interaction between teachers around teaching and learning, and strong networks amongst teachers. 

So we used these indicators on principal leadership and peer influence to guide our study. 

## Conceptual model

**Interaction around T&L**

**Instructional Conversation**

**Conceptual model**

**Principal**** Leadership**

**Peer ****Influence**

**Student Learning**

**Mission ****& Goals**

**Community & Trust**

**Focus on Instruction**

**Δ**** Instruction**

**Notes:**
Based on the literature, we developed a conceptual model to guide our study that was based on the precepts in the literature:

In our model we conceived that mission & goals, community & trust, and focus on instruction were manifestations of principal leadership, and 

Instructional conversation, interaction around teaching and learning, and advice networks were manifestations of peer influence. 

We hypothesized that principal leadership and peer influence both have an influence on changing teachers’ instructional practice, which, in turn, is a predictor of changes in student performance, ie student learning.

The hypotheses embedded in this model can be characterized in the following two research questions:

## Research Questions

**What is the relationship between teacher reported measures of leadership practice and peer influence with student achievement when including the indirect effects of teacher reported change in instructional practice?**
**What are the relative effects of teacher reports of principal leadership and teacher reports of peer influence on changes in instructional practice?**

**Notes:**
The hypotheses embedded in this model can be characterized in the following two research questions:

## Data

**● ****Survey data ****collected in 2007 via an IES funded evaluation of a principal leadership initiative.**
**● ****Survey of ****1,839 school faculty members in 38 elementary/middle schools (response rate = 81%).**
**● ****Focused only on 1,079 responding teachers of English and mathematics (and general elementary).**
**● ****End of year state test results in English and mathematics for 14,027 students in 2006 and 2007.**
**● ****Matching process resulted in 7,793 students and 519 teachers (English)/502 teachers (mathematics).  **
**● ****76% overlap in teacher samples.**

**Notes:**
Reason for 76% overlap: Mostly elementary teachers who teach both subjects, but we asked them about subject specific things.

Reasons for data reduction:

1) Only 1 year of test data
2) No link between teacher and students

## Measures & Reliabilities

**Faculty Trust – 6 items (ELA = .93, Math = .93) **
**Instructional Goals – 5 items (ELA=.91; Math = .90) **
**Instructional Expertise – 5 items (ELA = .91; Math = .91) **
**Peer Influence Scales**
**Instructional Conversation – 5 items (ELA = .86; ****Math = .84) **
**Interaction around Teaching and Learning – 4 items (ELA = .78; Math = .77) **
**Advice Networks – 2 items (ELA = .82, Math = .84)**
**Change In Teacher Practice Scale – 7 items (ELA = .94; Math = .95)**

**Measures & Reliabilities**

**Notes:**
The measures came from a teacher survey administered in 2007.

There were three principal leadership scales and three peer influence scales.

A few things notable here are:

Items specific to the subject content.

Advice networks scale is a collection of two social network variables.  - out-degree, or the extent that teacher sought instructional assistance from other teachers, and tie span, or the extent that teacher sought assistance outside of her grade level or subject area. 
 Change in teacher practice scale was teacher self reported change collected as one year of data.

## Two Phase Analysis

**● ****Phase 1: Teachers linked to students**
**	****	****	****Parallel HLM analyses in English & Math**
**● ****Phase 2: Structural equation models examining relationships between indicators of principal leadership, peer influence, and changes in teacher instructional practice.**

**Notes:**
Our analysis was conducted in two phases.

The model for phase 1 is a 3-level random intercepts model in which change in student achievement (which we will call LEARNING) is regressed on self-reported teacher change in instructional practice.

By allowing the effects of instruction on student learning to vary randomly by teacher we are able to make unbiased estimates of teacher effects.

Models were run separately for ELA and mathematics

Level 1 variables included student ethnicity, gender, LEP, and poverty (lunch assistance). 
Anything other than teacher practice at level 2?
At level 3 we aggregated the level 1 variables to control for school contextual effects. 

In phase 2 we used an SEM framework to examine the structural relationships between and amongst principal and peer influences on teacher practice. 
This was done in a single level structural equations model.

Now Phil is going to talk about the results.

## Measure of Student Learning

**2006 and 2007 end-of-year state assessment**
**Different tests in different years**
**● ****Transition to new state assessment by year, grade**
**● ****Tests not vertically equated**
**● ****Re-standardized within year and grade**
**Change in student rank**
**● ****Difference scores as the student outcome**
**● ****Measure of students relative change in ranking**

**Notes:**
Student achievement data in ELA and mathematics was measured using an end-of-year state assessment.
The 06-07 school years fell within a period of transition for phasing in a new state assessment.  This resulted in different tests in different years depending on grade.
Our study was designed to examine predictors of student learning as defined by change in standardized test scores over one year.
Because the two assessments were not horizontally equated, we re-standardized achievement data within year and grade so that the two tests were on the same metric
The result of this is that we are now examining changes in student relative rankings. 
We use difference scores as the student outcome 
Difference scores assume the amount of change is on an interval scale and answers the question did you move up or down the ladder relative to other students in the 

## Results: Relationship between change in practice and student learning (Random Effects Null Model)

**	****ELA****	****Math**
**Student****	****91%****	****88%**
**Teacher****	**** 9%****	****10%**
**School****	**** 0%****	**** 1%**

**Notes:**
As Jon said the first phase of this study examines the relationship between teacher practice and student acheivement.  This phase uses a HLM framework in which students are nested within teachers who are nested within schools.  We first used an unconditional random effects model to see if there exists all three levels contain significant variation in the outcome of student learning.  

Here we use abbreviated notation to show the unconditional model.

All three levels were found to be significant for math, although only a small amount of the variation was found at the school level.  IN ELA only the first two levels, student and teacher were found to have significant variation.

Overall, Ten percent of the variation of change in student achievement occurs at the class level, whereas virtually no change can be explained by students’ school.

These results tell us two things.
First, it is important to model teacher and school effects in our sample of 38 schools and over 500 teachers.

And the results also reinforce the general trend in the literature of an inability to detect direct effects between principles and students.  We see here that student learning is invariant at the school level for ELA and very small for math.  This constriction of variation severly limits the statistical power of predictors variables at level 3.

## Results: Relationship between change in practice and student learning (Fixed Effects Model)

**Note: Model includes covariates for minority status, gender, LEP, and Free-Reduced Lunch by group centering at levels 1 and 2. **

**Change in Instruction** **	**** .06*****	**** .01****	****(.02)****	****(.02)**

**ELA****	****Math**

**Student Learning**

**Δ**** Instruction**

**Notes:**
The Phase 1 model includes the predictor of teacher change in practice and covariates.

Covariates were included using group centering so as to explain variation in learning at all levels.

Effects are presented in standardized units for teacher practice only.

Change in teacher practice explains change in student achievement for ELA (standardized effect size = 0.06), but not in mathematics

We take this an encouraging evidence which tells us that those teachers who report making changes to their practice can be shown to raise the ELA scores of their students on the end of the year state assessment.

Let us now process ten to examine previously discussed principle leadership  and teacher predictors on change in instructional practice

(my back of the envelope calculation indicated that for an .06 increase in this context we are talking about raising percentile stnadings by 4-5 percentile points)

## Results: Model of Change in Instruction in ELA

**Principal**** Leadership**

**Mission ****& Goals**

**Community & Trust**

**Focus on Instruction**

**.87***

**.62***

**.98***

**.06***

**Δ**** Instruction**

**Peer ****Influence**

**Advice Networks**

**Interaction around T&L**

**Instructional Conversation**

**Notes:**
Using the conceptual model which Jon just went through, we estimate the structiral model which is here laid out

Wherein two second order factors are used to preduict change in instruction.  

(in pahse one we constructed a scale for change in instruction and here we treat it as latent)

All of the variables are latent and the 34 items linked to first order factors are not shown

We first present the effects of part of the model highlighting principal leadership.  

All of he structural coefficients presented will be for the standardized solution. This is to facilitate comparison of the magnitude of effects within the model as well as with results for math.

All 3 paths to first order factors are large and significant.

Note that the path coefficient to ‘focus on instruction’ is close to 1 indicating that it could well stand in place of the second order factor in its ability to predict change in instruction.

The effect of principal leadership on change in instruction is significant and small.

## Results: Model of Change in Instruction in ELA

**Principal**** Leadership**

**Mission ****& Goals**

**Community & Trust**

**Focus on Instruction**

**.87***

**.62***

**.98***

**.06***

**.83***

**.61***

**.27***

**.28***

**.41***

_**Model Fit **_
**CFI = .95**
**RMSEA = .05**

**Δ**** Instruction**

**Peer ****Influence**

**Advice Networks**

**Interaction around T&L**

**Instructional Conversation**

**Results: Model of Change in ****Instruction in ELA**

**Notes:**
Now presented are all of the structural coeffcients and the non-caual component linking second order factors.

We see that the effect of peer influence on change in instruction is significant and almost five times as large as that of principal leadership.

The correlation between principal leadership and peer influence is significant and moderate.  But we are uncertain of how to intepret this because of the conflation between the real likelihood that schools which have higher of one will have higher of the other, and predisposition of survey respondents to rate this similarly.

A final word about this model, we wee this model as having acceptable fit but we’ll talk later about ways in which we are working to achieve better fit.

## Results: Model of Change in Instruction in Math

**Principal**** Leadership**

**Δ**** Instruction**

**Mission ****& Goals**

**Community & Trust**

**Focus on Instruction**

**.82***

**.61***

**.97***

**.07***

**.79***

**.63***

**.33**

**.28***

**.39***

_**Model Fit **_
**CFI = .94**
**RMSEA = .05**

**Peer ****Influence**

**Advice Networks**

**Interaction around T&L**

**Instructional Conversation**

**Results: Model of Change in ****Instruction in Math**

**Notes:**
For math we use a different sample, different set of subject specific variables for the outcome and also many different items of principals and teachers which are subject area specific.

There are several things you should notice in these results.

First we find essentially the same structure for principal leadership effects as we saw in ELA.

Second there is a very similar structure for peer influence.

We are again satisfued with fit but are considering ways of improving fit.

## Discussion

**Educational leadership influences practice which changes performance.**
**Results are consistent in both English and math models.**
**Principal instructional expertise is the most salient perceived principal leadership characteristic related to changing teacher instructional practice.**
**Instructional collaboration amongst peers is the most influential teacher characteristic related to changing teacher instructional practice.**
**Peer influence is more influential than principal leadership on changing teacher practice in both English and mathematics.**

**Notes:**
The results are consistent with the literature that suggests that school principals have a small but significant effect on student achievement, mediated by teacher instruction. 

## Next steps

**Combine both phases of the study into one multi-level model so we can examine both direct and indirect effects simultaneously.**
**Look at effects of principal leadership on peer influence related to changing instruction.**
**Add contextual effects of school characteristics into the models.**
**Analysis of in-sample and out-of-sample teachers to assess attrition bias.**

## The Long and Winding Road

**The Relationship Between Leadership Practice and Student Performance**
**Jonathan Supovitz**
**Philip Sirinides**
**University of Pennsylvania**
**Consortium for Policy Research in Education**
**June, 2008**
**IES Annual Conference**

**This research is supported by the Institute of Education Sciences, grant # R305C050041.  The opinions expressed are those of the authors and do not represent the views of the U.S. Department of Education.**