# Learned helplessness about "teaching to the test"

I keep wondering why is there so much learned helplessness about "teaching to the test".

It is often used as an example of Goodharting (e.g. [here](https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.html)), and the implied conclusion seems to be... that it is wrong to try testing students systematically, because it is known that it inevitably causes teaching to the test, which ruins education?

Every time I read something like that, I think: Why can't we simply *fix the test*, so that "teaching to the test" either becomes impossible, or becomes the right thing to do? To me this seems like the obvious reaction, so I am surprised that I don't see it more often.

*

Let me give you a fictional example: Suppose that a school teaches children the multiplication table up to 10×10. The department of education creates a test, containing two questions: "how much is 5×5?" and "how much is 7×8"?

Yes, if you are going to judge schools based on how well students answer *these two questions*, of course many teachers are going to follow the incentives, and instead of multiplication, they will spend all the time at classroom making the students memorize "5×5=25" and "7×8=56", even if doing so means that there will be no time left for other multiplication examples. So the next generation of students will have no idea how much is 6×6, despite officially having multiplication in the curriculum.

A scary story, isn't it? Does it mean that we should never test children on multiplication?

No, that would be completely stupid! (Looking around anxiously, hoping that someone agrees with me...)

The problem with the proposed test is that out of one hundred *possible* multiplication problems, it predictably tests two *predetermined* ones.

Well, how about instead of that, each year generate two multiplication problems *randomly*? That way, teachers won't know which specific multiplication problems they need to teach, so the best educational strategy will be to teach *all* of them.

Okay, one problem with this is so obvious that even I can predict it. If you literally choose the problems randomly, some years you are going to get "1×1" and "1×2" or something like that on the test, and people won't stop talking about how you ruined mathematical education forever, and how *unfair* it is that one generation of students got such easy problems, compared to the previous and the following years.

But if you do the simple fix and remove multiplication by one from the test, sooner or later the teachers will notice, and soon we will have a generation of students who have never learned how to multiply by one. Is there a way out of this dilemma?

Uhm, maybe we could use some heuristic to classify the multiplication problems into difficulty categories, such as:

*   trivial: multiplication by one or zero; otherwise
*   easy: multiplication by two or by ten, or if the result is at most ten; otherwise
*   medium: at least one of the numbers is smaller than six; otherwise
*   hard: everything else.

We could discuss the specific boundaries of the categories, but that's not the point. The test will contain six randomly generated problems: one trivial, one easy, two medium, two hard, in random order.

I think we are done here.

*

The same kind of thinking could be used for many objections that come up against testing. Let me try a few:

"Some schools teach basic math, other schools teach advanced math."

Let's make a "basic math" test and an "advanced math" test. The students who only had basic math will do the first one; the students who had advanced math will do both.

Why both, instead of only the advanced one? That's for the the students whose abilities were somewhere between both groups, decided to take the advanced lessons and then failed at the advanced test. I want to know how they stand compared to the kids who took basic math, for two reasons:

*   to lower the risk for those kids: now they don't have to choose between doing well on the basic test (which might seem "good, even if nothing special" to a third party), or doing very badly on the advanced test (which might seem worse), because they will still have the good results on the basic test;
*   to reduce the temptation for kids who suck at math to take the advanced lessons anyway, if they predict the opposite reaction of the third party: that even bad advanced test results will look better than good (or bad) basic test results.

Generally, you can reduce various kinds of "strategizing" by always making the advanced students also take the test for the basic students. It will allow you to easily compare all students on the same metric (the basic test). It might even stop various conspiracy theories about how the advanced classes are secretly the easier ones, etc.

"But still, this motivates teachers to only teach the stuff that is in the textbooks, and never even mention anything that would be considered too advanced, even if there happens to be a good opportunity for that. Also, kids who do math competitions won't be able to get extra points; they will seem just like someone who merely... studies okay."

Solution: add a few bonus questions that are definitely outside the curriculum for given age. Make it clear that those are bonus questions and that the students are not really expected to solve them. Maybe put them on an extra paper that the student has to explicitly ask for. That way, someone can score e.g. 110% on the test.

...I could go on, but I guess the point is clear. Whenever the objection is "but the test doesn't X", the proposed solution is to *make* the test X.

*

This is something that feels completely obvious to me, but I never hear it mentioned when people discuss "teaching to the test", so... I am looking forward to your replies.