Poor Oversight Leads To A State Wasting $365k On A Broken Model That Archaeologists Are Told To Use
By: Archaic Inquiries
Tags: Archaeology, Government Waste, Machine Learning, Transportation, Data Science

Poor Oversight Leads To A State Wasting $365k On A Broken Model That Archaeologists Are Told To Use Archaic Inquiries Follow Dec 23, 2020 · 8 min read

The last week has seen a flurry of emails exchanged with the SHPO and DOT of a state on the East Coast that were a mix of sad, bewildering, and frustration inducing. The scandal is still unfolding in real time, so I won’t out the specific state until I hear their response (though even the district archaeologists know what state it is at this point).

Below you will find why the $365k-model is useless, and then a description of why — to my reading — this happened. The second part is definitely the most important, so feel free to skip ahead.

The Seeds Of The Bitter Harvest

A few years back, a DOT got visited by some Good Idea Fairies, who instructed them to create a statewide predictive model for prehistoric archaeological sites. Naturally, they ended up partnering with one of the large engineering firms who has to pay for archaeological surveys all the time.

A cynical person would wonder why some company that has to pay for surveys on a regular basis would be commissioned by their regulator to provide a model on where they don’t have to survey, but I prefer to read the situation more charitably.

For the measly sum of $365,900.70 of taxpayer money, this firm agreed to provide the shiniest of all shiny models, complete with a GIS overlay that SHPO/DOT could make available to archaeologists.

This model would later (and as of writing, still is) be required to be utilized on most DOT projects and all large projects that require cultural resource surveys.

The Delivery

After working on this model from middle of 2013 to around the new year for 2015, the contracted consultant had managed to deliver a model and 7 volumes of documentation to the DOT.

Unfortunately, no one seems to have read the documentation until now. The model likely spits out nonsense numbers. A very simplified discussion of why is below.

The consulting company used all the unsurveyed land of this state as the negative data set.

Note how known sites = 1 and unobserved background land = 0

I hope this makes you say, “Wait. Isn’t that the very land we want the model to make predictions about?” Congratulations. You could have saved this state from a 1/3 of a million dollar mistake.

The environmental background is not a negative data set

Using the null data at all in the models (a mixture of regressions and Random Forest) is generally not appropriate anyways. And it certainly shouldn’t be used as the negative data. But even if the null portion of the data was in the independent variables instead of the dependent one, you can still wreak havoc with your models.

The way they checked their work did not utilize best practices for handling data.

Why didn’t the consultant’s data scientist realize they had goofed up at Step 1 before toiling away for a year and a half? It has to do with how they checked their work.

Normally the gold standard is to hold back a randomly selected portion of your model. Then, once you’ve trained your model, you can see if it actually performs when given real-world data.

That was not done here.

Now, there is a reason why. It’s because the data scientist did this type of validation within each model. Unfortunately, that’s not enough. We won’t get into the technical reasons of why, but you want nested layers of validation — you want to check your work both inside the model and outside.

Now, having used up the entire store of data by running the models, how did the consultant check their work? Basically, they used some arcane statistical methods. And they probably should have worked. But they didn’t.

They treated the known sites as a product of random sampling.

We all know that even within a project area, not all land is sampled at the same rate. Just to use shovel testing (because STPs are easy to count), let’s assume you have 100 acres of land. You consider 50 acres to be “high” probability and 50 acres to be “low” probability, and you test them at different intervals — 15 meters and 30 meters are common.

This means that 80% of your testing is done on the high probability land — because you can fit ~16 shovel tests on an acre at 15 meter intervals vs ~4 at 30 meters. So you need to find 80% of your sites on the high probability portion just to establish that both areas have equal probability of producing sites.

So. You have a bunch of known sites. But they weren’t discovered from a random sampling of the landscape. They were found by looking where we thought we’d find them.

Intuitively, most archaeologists know this. Why does it matter here? It matters because the distribution of the independent variables for the known (positive) data set was used for these arcane statistical tests. And that distribution was biased in ways the data scientist didn’t know and had no way of accounting for.

See, this is kinda the reason why we hold back a last tranche of data.

But wait! We haven’t hit the scandalous part yet!

I could delve deeper into the modeling and why it cut some corners, but some guy making a mathematical mistake isn’t what’s the big deal. No, it’s much worse than that.

The question isn’t why the model was made incorrectly — it’s why that wasn’t noticed until the last gasps of 2020, when some random person (me) happened to be reading the documentation as part of a background review of models in use at different agencies.

Where was this guy’s boss?

You’d think that for $365,000, you’d not throw one guy into a room alone for a couple years and then let him hand over the finished product without looking it over.

There clearly was no “second set of eyes” on this project. All the way back in early 2014, in the third volume of documentation delivered to the DOT, the fatal mistake had already been made (using null data as negative data).

Did this project have no oversight? You can’t even write a fiction story that’s 100+ pages without an editor. Why did a highly technical GIS-and-machine learning model not merit the gaze of someone else, either colleague or superior, before charging a DOT hundreds of thousands of dollars?

Where was the DOT’s due diligence?

Upon being delivered the documentation, who at the DOT was reading it? My guess is either nobody or nobody who understood it.

This is pure speculation on my part, but I’d be willing to guess that the documentation was read, but not thoroughly, and not by someone who felt they knew enough about the technical side that they felt they could comment.

Given the basic nature of the biggest model-breaking error, anyone with an undergraduate statistics course under their belt should have been able to pick up on the problem. But as a friend pointed out, very few programs steer archaeologists into math classes, and so very few archaeologists are comfortable criticizing smart people about complex methodology.

Why wasn’t this tested in the last half decade, since it’s used to guide fieldwork plans?

The SHPO forwarded me a short white paper from 2017–2018 where they’d attempted to at summarize, if not evaluate, the model and real-world data.

The thrust of their data suggested that they didn’t understand how to evaluate the model, and to their credit, the SHPO author and the intern paper it was an extension of, stated as such.

I did manage to pull enough data out of that one paper to look at a sample of 133 reports for 2017 across three regions of that state to give us some idea, though, of the model’s predictive power. Note that the numbers below are my own calculations, since I have not seen nor been told of any quantitative attempt to evaluate the SHPO/DOT model.

The sample showed that each probability tier held the following sites: Low: 11.76%, Med: 5.88%, High: 82.35%. This would lead to a naive estimate that one should expect the Low areas of these regions to contain around 11% of the sites. Most archaeologists stop here.

But adjusting for the fact that the Low area is 67% of the land in these regions and was only about 39% of the land, we end up with a weighted average of Low: 34.16%, Med: 8.02%, High: 57.82%.

Now, I didn’t adjust for anything else — like more sampling per acre of the High area — since I didn’t have that information. But note how the expected number of sites in the Low areas went from 11% to 34%, and all I did was account for the under/over representation of each probability tier in their sample.

The less intense sampling per acre for the Low area vs the High area will definitely push the expected sites in unsurveyed Low area higher. Note that as you approach 67% of sites expected in the Low area, you are doing only as well as a coin flip (since 67% of the total land in those regions is Low).

And how about the Medium area, huh? That’s 23% of the land by area, and only expected to contain 8% of sites based on this sample? This is a great time to point out that probability tiers shouldn’t be named things like “Medium” because it could actually end up with the lowest probability.

The TL;DR Takeaway

This wasted $365k of taxpayer money. It shows a cavalier approach to quality control at both a major company and the DOT it sold this model to. It also shows that a broken model is currently used for large, significant projects. I hope that it has never played a role in route selection for eminent domain projects.

And honestly, if you want a predictive model, get something for about 3–5% of the cost of this one. Keep it simple, so you can update it and test it yourself. Complex math does not perform better than simpler math, even without the raft of quality control issues with this model.

I’m currently awaiting the SHPO and DOT responses, as I informed them only late last week. I should note that I do not work in this state, nor have any plans to do so in the near future. I have not been paid or otherwise compensated to lay out this critique to the DOT and SHPO. My motivation is mainly that I am professionally embarrassed by the lack of numeracy/oversight that led to this model being used to guide policy.

I am available to make suggestions to and look through more of your data — within limits — for those who work within these organizations (they know who they are after my initial email was disseminated, I believe).