Mostly Helpful Econometrics
By: Mikey Shulman
Tags: Naturallanguageprocessing, Econometrics, Machine Learning, Artificial Intelligence, Economics

Mostly Helpful Econometrics

When asked how he would spend an hour to save the world, Albert Einstein famously said that he would spend 55 minutes defining the problem and 5 minutes solving it. Einstein’s suggestion has profound lessons that apply to all fields, in particular to machine learning.

Historically, machine learning and econometrics¹ have been two sides of the same coin. Both fields use similar techniques (e.g., logistic regression) to solve similar prediction problems, albeit with a completely different vernacular and jargon. The key difference between machine learning and econometrics is the approach to framing and understanding a problem.

Econometricians like to quantify causes and effects, and therefore spend time examining coefficients and designing hypotheses and experiments to test them.

Machine learning practitioners approach problems by trying to make the best inference and capturing as much of the variance in the data as possible.

Recently, practitioners of these two schools of thought have drifted away from one another. Machine learning researchers favor exploiting the abundance of easily obtained data to train larger, more powerful, more complex, and less interpretable models, and econometricians favor cleverly crafted experiments and more interpretable models. Accordingly, we see big tech companies hiring economists to design and implement A/B tests to sell ads, rather than design the ML models to suggest the most relevant ads.

Modern machine learning, and modern natural language processing in particular, has reached an interesting plateau: while it is clear that larger datasets, larger models, and more compute power improve performance on a codified set of benchmark problems, it is increasingly difficult to quantify how well models learn and generalize.

For example, while it is (at least in theory) straightforward to obtain a large corpus of text, train an ever-increasingly large transformer-based language model, fine tune it on SQUAD 2.0, and measure our performance (e.g. f1) on the test set, we have little grasp of how well our model will do on different machine comprehension tasks, how much our fine-tuning might have degraded our language model’s performance on other tasks, how well it will perform on the same task out of data set, what knowledge our language model might have catastrophically forgotten, etc.

In other words, while the community has made such astonishing progress over the last 5 years, and while solving a well-defined problem is straightforward, we have reached the point where it is now unclear how benchmark progress correlates with natural language understanding. For example, all state-of-the-art machine comprehension models showed dramatic decreases in performance when a few non-grammatical words were added to the paragraph.

An Alternate Path Forward

Many believe that the route forward is to continue in a similar tact, for example by inventing new pre-training tasks, creating complex networks with memory modules, and designing different learning methods and curricula. Others believe that borrowing more from linguistics is a more fruitful path to better quantify our machine intelligence and generalization in natural language. However, machine learning’s old friend, econometrics, may have the tools for making the most progress given the current state of the art.

In particular, there are three ways in which modern NLP can better progress by emulating economists:

First, economists are generally clever in their data collection efforts, since data can be expensive to obtain. The ML community has typically favored quantity over quality with data (e.g. C4, and common crawl), and while this bias has enabled incredible progress, it has, for example, made it difficult to distinguish between improvements in model architecture, data, and training procedures. On the other hand, the right data can often make all the difference, and ML researchers who have embraced this quality-over-quantity approach have often been successful. For example, asking human annotators to make small edits to summaries to improve their quality gave researchers pairs of summaries that are as close as possible textually but as different as possible in quality.

Second, economists are experts in crafting natural experiments to distinguish between hypotheses. There are far too many examples to cite, but introducing cell phones to a market to study how markets respond to efficient information spreading is an exemplary instance of economists’ acumen for experiment design. While the authors of SQUaD did an excellent job crafting a problem and dataset that highlights a large gap between human and machine performance, it took two years of brute force to discover that questions with no answers would uncover a fundamental difference in comprehension ability between humans and machines. This simple hypothesis and its accompanying test cases propelled the plateauing field of machine comprehension for years.

Lastly, economists are excellent problem framers who are able to identify the eigenquestions of an issue, and reframe a problem to explain discrepant data. For example, economists are able to understand the implications of taxes (e.g. personal vs corporate) by recasting the effects in terms of tax incidence, measuring to whom the burden of the tax ultimately falls instead of the entity on whom the tax is levied. Master problem framing can be seen in all areas of economics, for example, in reframing the endowment effect in terms of salience and context instead of loss aversion. Here, too, ML researchers who adopt an approach more in line with economists’ strengths have honed their craft and advanced their field. Consider the GLUE set of benchmark tasks, which were designed to demonstrate and quantify natural language understanding, but became obsolete when increasingly powerful transformer-based models surpassed human performance. Subsequently, careful problem framing by the authors of the SuperGLUE set of benchmark tasks better aligned machine learning problems with the intricacies and difficulties of natural language and allowed researchers to further explore more powerful models.

However, the NLP community is playing catchup in acquiring the skills economists find so valuable. Might questions with no answers have appeared in SQUaD 1.0, and might the community have arrived at SuperGLUE tasks before GLUE? But alas, these breakthroughs were found inefficiently, after years of brute forcing with increasingly large models revealed inadequacies with their predecessors.

The machine learning community, therefore, should look toward its economist friends for help and inspiration in making progress, and economists should consider entering the field of machine learning because of the abundance of problems amenable to their skillset. And while Einstein’s time management suggestion might be extreme, the machine learning community would be wise to heed some of his advice.

Thank you to Patrick O’Neill for his invaluable input.

Footnotes