# Presentation: 531306

## What Makes For a Good Teacher and Who Can Tell?

- Douglas N. Harris                               Tim R. Sass
- Dept. of Ed. Policy Studies                 Dept. of Economics
-     Univ. of Wisconsin	                         Florida State Univ.

- IES Research Conference, June 2008

## Background

- Growing Interest in Linking Teacher Performance and Compensation
- “Value Added” scores based on performance of a teacher’s students have limitations
- Only available for teachers in tested subject (reading and math)
- Subject to measurement error and inter-temporal instability
- Movement to Grant Principals Greater Autonomy in Personnel Decisions
- Charter schools
- School reform in New York City

## Background

- Still Know Relatively Little About What Teacher Characteristics are Associated With Ability to Improve Student Performance
- Most recent studies using panel data on individual student achievement find that teacher experience (in the early years) matters, but little else in the way of readily observable characteristics seems to affect student achievement
- Knowing what characteristics determine teacher quality has implications for teacher preparation, licensure, hiring practices and professional development

## Research Questions

- Do principals’ evaluations of teachers contain information beyond readily-observable “objective” teacher characteristics?
- What factors determine a principal’s overall evaluation of her teachers?
- What specific teacher characteristics are associated with a teacher’s ability to raise student achievement (ie. “value added”)?
- Can principals do as well (or better) at predicting future teacher value added than past performance measures?

## Literature Review

- Early Literature
- Murnane (1975)
- Conditioning on student characteristics and prior achievement, a one-standard-deviation increase in principal’s rating is associated with a 0.125 standard deviation increase in student third-grade math scores
- Peterson (1987, 2000)
- Finds weak correlation between principal ratings of teachers and parent or student satisfaction
- Milanowski (2004), Kimball, et al. (2004), White (2004)
- Comparisons of principals’ four-point teacher evaluation scales with single-year estimates of teacher value added
- Generally find weakly positive correlation between principal evaluations and estimated teacher quality (correlation = 0.2 - 0.4)

## Literature Review

- Recent Literature
- Jacob and Lefgren (2005)
- Consider both principal’s overall rating and evaluations of specific teacher characteristics
- Find principals good at identifying best and worse teachers, but less able to distinguish teachers in the middle
- Principal evaluations better at predicting future student achievement than teacher experience or educational attainment, though generally not as good as value-added estimates
- Principal evaluations better at predicting parental requests for teachers than teacher value-added scores

## Data

- Anonymous mid-sized school district in Florida
- Schools serve a diverse set of student populations
- Free/Reduced-Price Lunch:  5-90%
- Percent Minority:  10-90%
- Longitudinal student-level achievement data
- Grades 1-10
- 1999/00 through 2004/05
- Math and reading scores on the Stanford Achievement Test

## Data

- Principal Interviews
- In-person interviews conducted in Summer 2006
- Single-blind evaluation of up to 10 teachers (5 per subject)
- Random sample of teachers at school who taught at least three classes in the tested grades and subjects during 1999/00-2004/05
- 9-Point Rating Scale
- Not effective (1-3):  performance is substantially below minimum standards
- Adequate: (4-6):  meets standards but can improve in several areas.
- Exceptional (7-9):  teacher is among the best I have seen
- Asked to evaluate teachers on:
- Overall performance
- Ability to test scores
- Specific characteristics

## Sample Studentand Teacher Characteristics

- 		_Math_	_Reading_
- Total Students	76,308	70,916
- Total Teachers	     943	960
- Principals Interviewed	       30	30
- Teachers Rated by Principal	234	231

## Sample Teacher Grade Levelsand Principal Ratings

- 							_**Math**_**                 **_**Reading**_
**Teachers**
- Taught Primarily Elementary School			0.718		0.706
- Taught Primarily Middle School			0.152		0.153
- Taught Primarily High School			0.131		0.140
**Mean Principal Rating of Teacher**
- Overall					7.103		7.108
- Ability to Raise Test Scores			7.232		7.164
- “Caring”					7.397		7.468
- “Enthusiastic”					7.269		7.385
- “Motivated”					7.436		7.494
- “Strong Teaching Skills”				7.560		7.597
- “Knows Subject”				7.868		7.896
- “Communication Skills”				7.594		7.710
- “Intelligence”					7.897		7.922
- “Positive Relationship with Parents”			7.511		7.596
- “Positive Relationship with Students”			7.670		7.730

## Pairwise Correlation of Principal’s Ratings ofTeachers With Teacher Characteristic Factors (Math)

|  | Overall Rating | Ability to Raise Test Scores | Inter-personal Skills | Motivation/ Enthusiasm | Works Well With Others | Knowledge/ Teaching Skills |
| --- | --- | --- | --- | --- | --- | --- |
| Overall Rating  | 1 |  |  |  |  |  |
| Ability to Raise Test Scores | 0.745** | 1 |  |  |  |  |
| Interpersonal Skills  | 0.703** | 0.550** | 1 |  |  |  |
| Motivation/ Enthusiasm | 0.738** | 0.596** | 0.734** | 1 |  |  |
| Works Well With Others | 0.762** | 0.598** | 0.756** | 0.732** | 1 |  |
| Knowledge/ Teaching Skills | 0.881** | 0.752** | 0.612** | 0.682** | 0.644** | 1 |

- ** Significant at the 0.05 level

## Pairwise Correlation of Principal’s Ratings ofTeachers With Teacher Characteristic Factors (Reading)

|  | Overall Rating | Ability to Raise Test Scores | Inter-personal Skills | Motivation/ Enthusiasm | Works Well With Others | Knowledge/ Teaching Skills |
| --- | --- | --- | --- | --- | --- | --- |
| Overall Rating  | 1 |  |  |  |  |  |
| Ability to Raise Test Scores | 0.736** | 1 |  |  |  |  |
| Interpersonal Skills  | 0.719** | 0.626** | 1 |  |  |  |
| Motivation/ Enthusiasm | 0.699** | 0.569** | 0.714** | 1 |  |  |
| Works Well With Others | 0.725** | 0.589** | 0.762** | 0.677** | 1 |  |
| Knowledge/ Teaching Skills | 0.861** | 0.697** | 0.644** | 0.682** | 0.650** | 1 |

- ** Significant at the 0.05 level

## Factor Loadings of Normalized Principal Ratings (Math)

| Teacher Characteristic Rated by Principal | Interpersonal Skills | Motivation/ Enthusiasm | Works Well With Others | Knowledge/ Teaching Skills   |
| --- | --- | --- | --- | --- |
| Intelligent   | -0.0481 |  0.0839 |  0.0606 |  0.7067  |
| Works Well With Grade Team/Dept.  | -0.0046 | -0.0887 |  0.9711 |  0.0399  |
| Works Well With Me (Principal)  |  0.1743 |  0.0835 |  0.7415 | -0.0814  |
| Positive Relationship With Parents  |  0.7231 |  0.0781 |  0.0768 |  0.0742  |
| Positive Relationship With Students  |  0.9408 |  0.0103 | -0.0131 |  0.0636  |
| Caring |  0.5591 |  0.1372 |  0.2422 | -0.0185  |
| Enthusiastic |  0.1086 |  0.9721 | -0.0707 | -0.0035  |
| Motivated |  0.0398 |  0.5224 |  0.2802 |  0.1624  |
| Strong Teaching Skills |  0.1512 |  0.0258 | -0.0462 |  0.8471  |
| Knows Subject | -0.0088 | -0.0551 | -0.0036 |  0.9831  |
| Communication Skills | 0.1040 |  0.1705 |  0.2734 |  0.3191  |

## Factor Loadings of Normalized Principal Ratings (Reading)

| Teacher Characteristic Rated by Principal | Interpersonal Skills | Motivation/ Enthusiasm | Works Well With Others | Knowledge/ Teaching Skills   |
| --- | --- | --- | --- | --- |
| Intelligent   |  0.0030 |  0.0055 |  0.0332 |  0.7182 |
| Works Well With Grade Team/Dept.  |  0.0243 |  -0.0598 |  0.8522  |  0.0838 |
| Works Well With Me (Principal)  |  0.1390 |  0.0461 |  0.8329 | -0.0597 |
| Positive Relationship With Parents  |  0.7554 |  0.0528 |  0.0649 |  0.0757 |
| Positive Relationship With Students  |  0.9171 |  0.0271 |  0.0193 |  0.0313 |
| Caring |  0.6046 |  0.0983 |  0.2541 | -0.0367 |
| Enthusiastic |  0.0775 |  0.9879 | -0.0464 | -0.0191 |
| Motivated |  0.0247 |  0.5366 |  0.2017 |  0.2445 |
| Strong Teaching Skills |  0.2237 |  0.0174 | -0.0802 |  0.8140 |
| Knows Subject | -0.0834 | -0.0156 |  0.0385 |  0.9819 |
| Communication Skills |  0.1650 |  0.2167 |  0.1715 |  0.3290 |

## Estimation Procedure

- Estimate Student Achievement Model
- Includes student, teacher and school fixed effects
- Estimated teacher effects yield within-school measure of teacher effectiveness
- Regress Estimated Teacher Fixed Effects on “Objective” Teacher Characteristics and Normalized Principal Ratings
- Objective characteristics include teacher experience, possession of an advanced degree and certification status
- Weighted least squares estimates with the square root of the numbers of students per teacher as weights

## Value-Added Model

| Ait | = | achievement level of student i in year t |
| --- | --- | --- |
| Xit | = | time-varying student characteristics |
| P-ijmt | = | time-varying characteristics of peers in classroom j in school m |
| i | = | student fixed-effect |
| k | = | teacher fixed effect |
| m | = | school fixed effect |

## Value-Added Results Used for Estimation of Teacher Effects

-  (Grades 2 – 10, 1999/2000 – 2004/05)
-                                                                                                          Math                        Reading_ __   _
- ________________________________________________________
- Number of Schools Attended				-2.131**		-0.977
- 						(2.02)		(0.84)
- Attended Different School in Prior Year			 1.910***		 0.834
- 						(2.99)		(1.15)
- Class Size						-0.111**		-0.038
- 						(2.21)		(0.68)
- Proportion of Classroom Peers 				-7.788***		-2.481
-           Who are White					(4.37)		(1.30)
- ________________________________________________________
- Covariance of Achievement Gain and:
- 	Student FE					0.186		0.183
- 	Teacher FE					0.032		0.100
- 	Model (Including School Indicators)		0.191		0.183
- 	Error					0.591		0.534
- No. of Observations			                                    76,308                            70,916
- ________________________________________________________
- Note:  Model Includes Student, Teacher, School and Grade-By-Year fixed effects. t-ratios adjusted for clustering at the classroom level.

## Weighted Least Squares Estimates of the Relationship Between Principal Ratings and Estimated Teacher Effectiveness 

- 				         Math		       	       Reading
- 				   [1]	    [2]		   [1]	   [2] _________________________________________________________
- Overall Rating			 2.685***		 	 1.661*
- 				(2.82)			(1.76)
- Ability to Raise Test Scores			 2.570***			 0.975
- 					(4.52)			(1.48)
- _________________________________________________________
- R-squared				0.078	 0.149		0.061	0.057
- No. of Observations			   234	   207		   231	   201
- _________________________________________________________
- Note: Model includes a set of six experience category indicators and indicators for possession of advanced degrees and full
-           certification.

## WLS Estimates of the Relationship Between Principal Ratings and Estimated Teacher Effectiveness -- By Grade Level  	

- WLS Estimates of the Relationship Between Principal Ratings and Estimated Teacher Effectiveness -- By Grade Level
- 							    Math				                            Reading
- 						[1]	               [2]		           	  [1]               [2]
- ________________________________________________________
- Overall Rating  				3.328***					 2.483**
- 	Elementary 				(2.84)					(2.15)
- Overall Rating				 1.424					 0.029
- 	Middle/High				(0.87)					(0.02)
- Ability to Raise Test Scores						  4.071**		 	 2.971**
- 	 Elementary						  	 (3.31)				(2.25)
- Ability to Raise Test Scores					             1.456				-0.554
- 	 Middle/High						        (0.88)				(0.32)
- _______________________________________________________
- R-squared					0.082	      0.126			0.067	 0.071
- No. of Observations			             	    	   234	         199			   231	    201
- _______________________________________________________
- Note: Model includes a set of six experience category indicators and indicators for possession of advanced degrees and
-           full certification.

## WLS Estimates of the Relationship Between Teacher Subjective Characteristics and a Principal’s Overall Rating of Teachers

- 					             Math		Reading
- _______________________________________________________
- Interpersonal Skill			             0.091*    	  	 0.192***
- 					            (1.86)		(3.26)
- Knowledge/Teaching Skills		             0.592***		 0.592***
- 					         (14.53)		(11.49)
- Motivation/Enthusiasm		            0.069   	 	-0.002
- 					           (1.45)		(0.04)
- Works Well With Others		            0.237***                 	 0.193***
- 				           	           (4.83)		(3.30)
- _______________________________________________________
- R-squared				           0.853		0.786
- No. of Observations			             207		   206
- _______________________________________________________

## WLS Estimates of the RelationshipBetween Teacher Subjective Characteristicsand Estimated Teacher Effectiveness

- 					                         Math	                 Reading
- _______________________________________________________
- Interpersonal Skill				-0.250  	   	 1.656
- 						(0.15)		(0.93)
- Knowledge/Teaching Skills			 1.579   	   	 3.044*
- 						(1.10)		(1.96)
- Motivation/Enthusiasm			-1.030      		-1.532
- 						(0.61)		(0.96)
- Works Well With Others			 2.523   	   	-1.745
- 						(1.46)		(0.99)
- _______________________________________________________
- R-squared					0.078		 0.056
- No. of Observations				   207	         	    206
- _______________________________________________________

## WLS Estimates of the RelationshipBetween Teacher Subjective Characteristicsand Estimated Teacher Effectiveness (Math)

- 					    [1]	      [2]	        [3]	          [4]
- _______________________________________________________
- Interpersonal Skill 			 1.866*
- 					(1.94)
- Knowledge/Teaching Skills 			   2.295**
- 						  (2.38)
- Motivation/Enthusiasm 		 	 	     1.661*
- 					 		    (1.77)
- Works Well With Others 		 			       2.572***
- 					 			      (2.63)
- _______________________________________________________
- R-squared				0.058	  0.066	    0.055	      0.072
- No. of Observations			  207	    207	      207	        207
- _______________________________________________________

## WLS Estimates of the RelationshipBetween Teacher Subjective Characteristicsand Estimated Teacher Effectiveness (Reading)

- 					    [1]	        [2]	           [3]	             [4]
- _______________________________________________________
- Interpersonal Skill 			 1.167
- 					(1.15)
- Knowledge/Teaching Skills 			    1.881*
- 						   (1.84)
- Motivation/Enthusiasm 		 		       0.559
- 					 		      (0.56)
- Works Well With Others 		 			          0.521
- 					 			         (0.50)
- _______________________________________________________
- R-squared				0.035  	   0.045	       0.030	         0.030
- No. of Observations			   206	      206	          206	            206
- _______________________________________________________

## 						    Math		      Reading

					            [1]                [2]	[1]	   [2]
______________________________________________________________________

Overall Rating			          2.107***      	              1.446***
					         (3.86)                              (2.69)	

Teacher Fixed Effect (from 99/00-03/04)		            0.230*** 		 0.106
						           (3.85)		(1.44)
______________________________________________________________________

R-squared				        0.236            0.237           0.115	0.114

No. of Observations			         5361             5361            4399	4399
______________________________________________________________________

Note:  All models include controls for individual student mobility, class size, peer characteristics, student fixed effects (from
          1999/00-2003/04), school indicators and a constant term.

- 					            [1]                [2]	[1]	   [2]
- ______________________________________________________________________
- Overall Rating			          2.107***      	              1.446***
- 					         (3.86)                              (2.69)
- Teacher Fixed Effect (from 99/00-03/04)		            0.230*** 		 0.106
- 						           (3.85)		(1.44)
- ______________________________________________________________________
- R-squared				        0.236            0.237           0.115	0.114
- No. of Observations			         5361             5361            4399	4399
- ______________________________________________________________________
- Note:  All models include controls for individual student mobility, class size, peer characteristics, student fixed effects (from
-           1999/00-2003/04), school indicators and a constant term.

- Estimates of the Determinants of Student Achievement Gains(Grades 2 – 10, 2004/05)

## Summary

- Principal ratings of teachers predict teacher value-added even after controlling for teacher experience, educational attainment and certification status
- Principal ratings most strongly influenced by knowledge/teaching skill of teachers and the teacher’s ability to work with others
- A teacher’s ability to raise test scores is also most strongly influenced by knowledge/teaching skill and ability to work with others
- Relationships imprecise when all factors included simultaneously
- Principal’s overall evaluation does as well as past value-added at predicting current teacher effectiveness

## Implications

- Possibly greater role for principals in evaluating teachers and in retention decisions
- “pay for performance” scheme which includes principal ratings likely better than current pay scale based on teacher experience and education alone
- “value added” scores could be part of the mix as well
- Possibly greater emphasis on teaching skills/knowledge in teacher preparation and professional development