Source: http://vamboozled.com/2014/06/
Timestamp: 2019-04-23 16:40:00+00:00

Document:
June | 2014 | VAMboozled!
Rethinking Value-Added Models in Education: Critical Perspectives on Tests and Assessment-Based Accountability by Audrey Amrein-Beardsley.
Here is a book that every teacher must read in the era of standardized test-based accountability. In school after school, in state after state, teachers are being judged (in small or large part) on value-added models (VAMs) based on student performance on standardized tests. Amrein-Beardsley is the foremost authority on VAMs in the country. She created the blog Vamboozled! as a forum for information and criticism of VAM based accountability measures. In this book, Amrein-Beardsley takes a scholarly approach to the dismantling of the research and rhetoric behind VAMs.
Her conclusions: VAMs are unreliable, invalid, nontransparent, unfair, full of measurement errors, and being used inappropriately to make key decisions about teacher retention, termination and pay. Other than that they are just peachy.
Beardsley’s book is not just a critique of a failed accountability measure, however. In the final chapter she offers alternatives, solutions and conclusions. Her key recommendations are listed here.
Credibility and trust are key to any evaluation scheme. VAMs lack this basic requirement.
Educators’ professional judgment should not be removed from any evaluation system.
Any evaluation system should rely on multiple measures that are aligned with locally defined criteria for demonstrating effectiveness.
Any evaluation system must include a plan to evaluate and refine the system.
If you’re performance is being judged, even in small part by value-added measures, this book is a must read.
The other two books listed??
50 Myths & Lies That Threaten America’s Public Schools: The Real Crisis in Education by David C. Berliner, Gene V. Glass and Associates.
A Chronicle of Echoes: Who’s Who in the Implosion of American Public Education by Mercedes K.Schneider.
I just came across the first external review of my new book, Rethinking Value-Added Models in Education on Amazon, and I thought I would share this review with you all in that I am happy to announce that it captures exactly what I meant to do with this book, particularly for those of you out there who are practitioners working within (and under) the realities the come along with the adoption of VAMs for increased accountability.
Mission accomplished, I hope! I hope as well that, as this reviewer wrote, this book will continue to “provide the research” you all might need to continue “to [wisely] challenge your candidates and policy makers” in these regards.
For more information about the book, see it on Amazon or see a prior post about it here.
Moshe Adler on Chetty et al.
Well, one of this study’s biggest critics, and most important critics as an expert in economics himself, has written a response to all of this. More specifically, Moshe Adler – economist affiliated with Columbia University and Empire State College, State University of New York and author of the 2010 book, Economics for the Rest of Us: Debunking the Science That Makes Life Dismal – first (in April 2014) wrote a critique of Chetty et al.’s studies for the National Education Policy Center (NEPC) during which he also compared this study from its first release in 2011 (cited by Obama) to its current 2014 version (cited by the judge in Vergara v. California) that can be found here, after which Chetty et al. wrote a response (in May 2014) that can be found at the bottom of the page here, after which Adler released a final response (in June 2014) that can also be found at the very bottom of the page here.
For those of you interested in reading all three pieces, all three are really quite critical to understanding Chetty et al.’s methods, as well as their methods’ shortcomings; their assumptions, as well as their feasibility in context; their results, as unfortunately overstated as they are; and the like. For those of you who want just the highlights, though (as also referenced in the press release just released here), here are the nine key points taken from now the two studies. These are the nine concerns made and established by Moshe in the first round, to which Chetty et al. responded, but seemingly still stand as solid criticisms.
Value-added scores in this report and in general are unstable from year to year and from test to test, but the report ignores this instability.
The report inflates the effect of teacher value-added by assuming that a child’s test scores can be improved endlessly [over time].
The procedure that the report develops for calculating teacher value-added varies greatly between subjects within school levels (math or English in elementary/high school) and between schools within subjects (elementary or middle school math/English), indicating that the calculated teacher value-added may be random.
The commonly used method for determining how well a model predicts results is through correlations and illustrated through scatterplot graphs. The report does not [however] present either the calculations or graphs and instead invents its own novel graph to show that the measurements of value-added produce good predictions of future teacher performance. But this is misleading. Notwithstanding the graph, it is possible that the quality of predictions in the report was poor.
An earlier version of the report found that an increase in teacher value-added has no effect on income at age 30, but this result is not mentioned in this revised version. Instead, the authors state that they did not have a sufficiently large sample to investigate the relationship between teacher value-added and income at any age after 28, but this claim is untrue. They had 220,000 observations (p. 15), which is a more than sufficiently large sample for their analysis.
The method used to calculate the 1.34% increase [that a 1 standard deviation increase in teacher VA raises wages at age 28 by 1.34% (p. 37)] is misleading, since observations with no reported income were included in the analysis, while high earners were excluded. If done properly, it is possible that the effect of teacher value-added is to decrease, not increase, income at age 28 (or 30).
The increase in annual income at age 28 due to having a higher quality teacher “improved” dramatically from the first version of the report ($182 per year, report of December, 2011) to the next ($286 per year, report of September, 2013). Because the data sets are not identical, a slight discrepancy between estimates is to be expected. But since the discrepancy is so large, it suggests that the correlation between teacher value-added and income later in life is random.
In order to achieve its estimate of a $39,000 income gain per student, the report makes the assumption that the 1.34% increase in income at age 28 will be repeated year after year. Because no increase in income was detected at age 30, [however] and because 29.6% of the observations consisted of non-filers, this assumption is unjustified.
The effect of teacher value-added on test scores fades out rapidly. The report deals with this problem by citing two studies that it claims buttress the validity of its own results. This claim is [also] both wrong and misleading.
Following the Vergara v. California decision now two weeks ago (see three recent VAMboozled posts about this court decision here, here, and here), one of my mentors and still friends/colleagues – Gene Glass, Regents’ Professor Emeritus at ASU – wrote the following on his twitter and Facebook account about Raj Chetty’s and Tom Kane’s testimonies, testimonies that played key roles in the judge’s (unfortunate) decision ruling teacher tenure as unconstitutional.
Testimony in Vergara by Harvard profs. Does anybody–other than Judge Treu–really believe these guys?! Amazing! — These dudes must be in love with their production functions. How anyone but a know-nothing judge could buy this stuff is amazing to me. A bad teacher costs a class of 28 kids $1.4 Million in life-time earnings. Really? What do a couple of lousy economists cost society?
Glass then followed up with a comment: “Aristotle in Nichomachean Ethics: “We must not expect more precision than the subject-matter admits.” (Chp. 3)…after which a Facebook friend replied…”Aristotle was 2.13 times more effective than Plato at teaching Aristotle’s ethics.” Funny!
Anyhow, somebody else at The Becoming Radical seemed to come across this post and wrote another blog post around it titled, “The Very Disappointing Teacher Impact Numbers from Chetty.” To read more about what this author has titled the now “very famous” but still “mostly hypothetical” Chetty et al. study and what their numbers ACTUALLY mean, even if true and accurate in the real world, click here.
Well, “WT” said nothing, besides a bunch of nothingness surrounding his anonymous withdrawal from the conversation string. Perhaps this was Chetty as “WT?” Perhaps not, but I’d bet some serious cash whoever “WT” was was pretty darn close to the Chetty et al. study and didn’t want to show it.
Sooo…now getting to the best part of all of this and how this has, in an interesting and similar turn of events, evidenced itself elsewhere. This video just came out on our local news station in Phoenix about a very similar situation in the state of Arizona and the state’s Superintendent of Public Instruction – John Huppenthal – engaging in some equally shady blogging tactics. Do give it a watch if you want yet another check on reality.
I received this “public letter” via email and felt compelled to share it with you all. It is written by Ed Johnson, “Advocate for Quality in Public Education” and devotee of W. Edwards Deming. It is about the quite honestly shameful testimony of Harvard Economics Professor Raj Chetty (the source of prior posts here and here), his testimony’s impact on the Vergara v. California decision last week (see two recent posts about this court decision here and here), and the deleterious implications of the ways that Chetty and those whom he advises (including Obama’s key leaders) think about educational measurement — implications that unfortunately span well beyond the teachers at the center of what is now so many similar VAM-based measurement systems.
Thanks mainly to Raj Chetty’s testimony, corporate school reformers won, or bought, the Vergara v. California case. Now, in consequence, there should be grave concern. California Superior Court’s ruling and Chetty’s cheerful and gleeful testimony offer insight into the kind of intelligence-without-wisdom that can only lead to hastening the demise of democratic ideals along with the rise of plutocratic ideals and attendant morally and ethically corrupt and corrupting ways of social life legitimized in public law. It is worth spending nearly two hours to watch and listen to Chetty’s testimony, so one may disallow oneself ever being able to claim, as some in Germany once claimed, “I didn’t know.” Be sure to listen to Chetty explain President Obama’s and the White House’s involvement.
Chetty based his testimony on having posed and answered for himself the question: “Are teachers’ impacts on students’ test scores (“value-added”) a good measure of their quality?” He testified that his research findings brought him to answer the question in the affirmative. As if nothing much beyond test scores matter, Chetty concluded that teachers’ impact on students’ test scores are a good measure of the quality of students’ teachers. Now just think about that for a minute and what it means and the potential precedent it holds for the future.
Chetty applied economic reasoning to “big data” sets of students’ test scores and relatable other kinds of data to, in effect, contrive little more than a method for distributing teachers over the mythical bell curve. Bottom-line result? “Déjà vu all over again” – that is, teachers ranked. The five percent or so of teachers forced into the head of the bell curve are deemed, arguably, highly valuable teachers. Similarly, the five percent or so of teachers forced into the tail of the bell curve are deemed, arguably, non-valuable teachers. Then Chetty intimated that the teachers ranked by his method establish the justification to fire the non-valuable teachers among them. And the quicker that is done, the better for the students. But, of course, “the quicker, the better” can only be done if teachers are disallowed tenure – that is, if teachers are allowed no rights to due process.
Now, the rub is, why would anyone today believe Chetty’s magical “value-added” genie of a measure will still be in its bottle in the future? Why would anyone believe corporatists will never apply Vergara v. California to employees? Why would anyone believe our government will never back Vergara v. California to be applied beyond public school teachers to include ever more citizens? Why would anyone believe “teachers” will never be generalized to “people” at the behest by plutocrats’ desires to get rid of non-valuable people the quicker, the better? Why would anyone believe eugenics will never be embraced, again?
“Let X be the contribution of some individual, (YX) the effect of the system on his performance. Then suppose that we have some number for his apparent performance, such as eight mistakes during the year, or sales of $8,000,000.
“Then, X + (YX) = 8.
“We need X. Unfortunately, there are two unknowns and only one equation. Johnny in the sixth grade knows that no one can solve this equation for X. Yet people that use the merit system think that they are solving it for X. They ignore the other term (YX), which is predominant.
“There is another factor to take into account, the Pygmalion effect. Rated high at the start, anyone stays high. Rated low at the start, he stays low.
“Ranking creates competition between people, salesmen, teams, divisions. It demoralizes employees.
–W. Edwards Deming. The New Economics for Industry, Government, Education (pp. 25-26). Kindle Edition.
Raj Chetty’s value-added measure (VAM) contrivance simply invites staying stuck in the status quo of regressive and mechanistic ways of thinking that try to reduce life’s complexity to simplistic algorithms and scripts. Instead, the country could be progressing, provided it had leaders that held a stronger adherence to democratic ideals than to plutocratic ideals, and that held a more just, moral, and ethical social systemic world view (see Ackoff, 11/1993).
Frankly, where are the words with which to express deeply felt sorrow and revulsion with Raj Chetty’s VAM, California Superior Court’s ruling, and Obama’s involvement? How ironic that the country’s first self-proclaimed “African American” President serves a chief facilitator of attacks on public education.
“I am part cognitive scientist, part statistician, and I work for ACT, Inc. About a year ago I was asked whether value-added models (VAMs) provide good estimates of teacher effectiveness. I found lots of papers that examined if different statistical models gave similar answers to each other and lots showing certain groups tended to have lower scores, but all of these papers seemed to bypass the question I was interested in: do VAMs accurately estimate teacher effectiveness? I began looking at this and Audrey invited me to write a guest blog post describing what I found.
The difficulty is that in the real world we don’t really know how effective a teacher, that is in “objective” terms, so that we do not have much with which to compare our VAM estimates. Hence, a popular alternative/way for statisticians to decide if an estimation method is good or bad is to simulate data using particular values, and then see if the method produces estimates that are similar to the values expected.
The trick is how to create the simulation data to do this. It is necessary to have some model for how the data arise. Hence, I will use a simple model to demonstrate the problem. It is usually best to start with simple models, make sure the statistical procedures work, and then progress to more complex models. On that note, I have an under review paper that goes into more details with other models with more simulation studies. Email me if you want a copy.
Anyhow, the model I used to create the data for this simulation starts with three variables. The first encapsulates everything about the students that is unique to them, including ability, effort, grit, and even the home environment (called AB in the code below). The second encapsulates all the neighborhood and environmental factors that can influence everything from financial spending in schools to which teachers are hired (called NE). This value is unique to each teacher (so it is simpler than real data where teachers are nested in schools). The third is teacher effectiveness (called TE). I created it from the neighborhood variable plus some random variation. These three variables would be unmeasured in a real data set (of course, some elements of them may be measured, but not all elements of them), but in a computer simulation their values are known.
In addition, there are two sets of test scores. The first are scores from before the student has encountered the teacher (called PRE). They are created by adding the student ability variable, the neighborhood variable, and some random variation. The second set of test scores are from after the student has encountered the teacher (called POST). They are created by adding the first set of test scores, the student ability variable, one-fifth of the teacher effectiveness variable (less than the other effects since the impact of a single teacher is usually estimated at about 10% or less of student achievement), and some random variation.
Again, however, this model is simpler than real educational data. Accordingly, there are no complications like missing values, teachers nested within schools, etc. Also, I will use a very simple VAM, just using the first set of scores to predict the second set of scores, but allowing for random variation by teachers and by students. Given the importance placed on the results of VAMs in many different countries and in many industries (not just in education), this method should work…Right?
Theories about causation actually suggest that there will be a problem. I won’t go into the details about this (again, email me for my paper if you want more information), but using the first set of scores in the VAM allows information to flow between the teacher effectiveness variable and the second set of test scores through the other unmeasured variables. This messes up trying to measure the effect of teachers on the final scores.
But let’s see if a simulation supports that this is problematic as well. The simulation used the freeware R and the code is below so that you can repeat it, if you want, or change the values. Download R onto your computer (follow the instructions on http://cran.us.r-project.org/), open it, copy the code, and paste it into R. The # sign means R ignores the remainder of the line so I have used that to make comments. If you think another set of numbers are more realistic, put those in. Or construct the data in some other way. One nice thing about simulations is you can keep trying all sorts of things.
For the values used, the correlation is -.13. A negative correlation means the teachers who are more effective tend to have lower estimated teacher effectiveness scores. That’s not good. That’s bad! Don’t get hung up with this specific number, though, as it moves around depending on how the data are constructed. The correlation can go up (e.g., change any one, but not two, of the first four values to -1) and down (e.g., increase the size of AB2POST to 2).
Given this, there is a conclusion, as well as a recommendation. The conclusion is that value-added estimates can be very inaccurate for what seem to be highly commonsensical and plausible models for how the data could arise, and where they are bad is predicted from theories of causation. The recommendation is that those promoting and those critical of VAMs should write down plausible models for how they think the data in which they are interested arose and see if the statistical procedures used perform well.
*The personal opinions expressed do not necessarily represent the views and opinions of ACT, Inc.
Following up on my most recent post regarding the “Vergara Case Lost in Los Angeles,” it seems “they” aren’t stopping in Los Angeles.
A VAMboozled! follower posted this recent op-ed on this topic on Aljazeera, in which the authors write about how “Students Matter, financier of the case (Vergara vs. State of California) with the funding of tech magnate David Welch, is preparing to bring similar suits to New York, Connecticut, Maryland, Oregon, New Mexico, Idaho and Kansas. This state-by-state strategy follows a historical precedent by similarly motivated groups, such as turn-of-the-20th-century women suffragists and contemporary gay marriage activists, seeking to challenge regulations at the state level to build a groundswell for national upheaval.” To read more, click here.
But also in breaking news, it seems that the Houston Independent School District (HISD) – the district in which the American and Houston Federation of Teachers are suing the district for their wrongful use of, and over-emphasis on VAMs (see a prior post about this here) – is to be defended by the same law firm who just won the Vergara trial.
As for the national and local unions’ response? As per a joint press release: “Hiring a powerful law firm and accepting a grant from one of the country’s most conservative and well-endowed foundations shows the Houston school system will spend as much money as it can to defend what we believe are unjust and unfair actions, rather than act in the best interests of the district’s students and teachers.” To read more, click here.
A few months ago, I wrote a few posts about the “Vergara v. California” case happening in Los Angeles, California, as well as posts about the testimonies of Thomas Kane (Economics Professor from Harvard University), David Berliner (Regents Professor Emeritus from Arizona State), Linda Darling-Hammond (Professor from Stanford), and Jesse Rothstein (Economics Associate Professor from the University of California – Berkeley).
The case involved nine public school students (backed by some serious corporate reformer funds) who were collected by the “Students Matter” to challenge five California state statutes that supported the state’s “ironclad [teacher] tenure system.” The prosecution’s argument was that students’ rights to a good education were being violated by teachers’ job protections…protections that were making it too difficult to fire “grossly ineffective” teachers. The protections at issue included the set of statutes that granted permanent employment status to teachers after 18 months on the job, that required a lengthy procedure to dismiss teachers, and that set up a seniority system in which the teachers most recently hired were the first to be fired when layoffs occurred, or in recent years needed to occur given economic crises and swings.
The suggested replacement to the “old” way of doing this, of course, was to use value-added scores to make “better” decisions, as based on “the data,” about whom to fire and whom to keep around as per whether they were positively impacting student achievement, and growth in student achievement. This is precisely why this case mattered so much for this audience here, particularly as we continue to think about VAMs and how they are being positioned, many times falsely, for that which they theoretically can do versus what they really can do in practice.
The case will certainly be appealed to a higher court, however, so while a loss there is still hope, as per Welner.
Again, to read this post, please click here. It is definitely worth the balanced read and point of view.
The Education Department is pulling Washington state’s No Child Left Behind waiver because the state has not met the department’s timeline for tying teacher evaluations to student performance metrics.
Washington is the first state to lose its waiver. The loss will give local districts less flexibility in using federal funds. For instance, they may now be required to spend millions on private tutoring services for at-risk students. The waiver revocation could also result in nearly every school across the state being labeled as failing under NCLB.
Washington had pledged in its waiver application to make student growth a significant factor in teacher and principal evaluations by the 2014-15 school year. But the state Legislature refused to pass a bill mandating that student performance on statewide assessments be included in teacher evaluations. The department placed the state on “high-risk” status in August. Arizona, Kansas and Oregon are also at risk of losing their waivers.

References: V. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v.