Big Data : New Tricks for Econometrics

5 Nowadays computers are in the middle of most economic transactions. 6 These “computer-mediated transactions” generate huge amounts of 7 data, and new tools can be used to manipulate and analyze this data. 8 This essay offers a brief introduction to some of these tools and meth9 ods. 10 Computers are now involved in many economic transactions and can cap11 ture data associated with these transactions, which can then be manipulated 12 and analyzed. Conventional statistical and econometric techniques such as 13 regression often work well but there are issues unique to big data sets that 14 may require different tools. 15 First, the sheer size of the data involved may require more powerful data 16 manipulation tools. Second, we may have more potential predictors than 17 appropriate for estimation, so we need to do some kind of variable selection. 18 Third, large data sets may allow for more flexible relationships than simple 19 ∗Hal Varian is Chief Economist, Google Inc., Mountain View, California, and Emeritus Professor of Economics, University of California, Berkeley, California. Thanks to Jeffrey Oldham, Tom Zhang, Rob On, Pierre Grinspan, Jerry Friedman, Art Owen, Steve Scott, Bo Cowgill, Brock Noland, Daniel Stonehill, Robert Snedegar, Gary King, the editors of this journal for comments on earlier versions of this paper. 1 linear models. Machine learning techniques such as decision trees, support 20 vector machines, neural nets, deep learning and so on may allow for more 21 effective ways to model complex relationships. 22 In this essay I will describe a few of these tools for manipulating and an23 alyzing big data. I believe that these methods have a lot to offer and should 24 be more widely known and used by economists. In fact, my standard advice 25 to graduate students these days is “go to the computer science department 26 and take a class in machine learning.” There have been very fruitful collabo27 rations between computer scientists and statisticians in the last decade or so, 28 and I expect collaborations between computer scientists and econometricians 29 will also be productive in the future. 30 1 Tools to manipulate big data 31 Economists have historically dealt with data that fits in a spreadsheet, but 32 that is changing as new more detailed data becomes available; see Einav 33 and Levin [2013] for several examples and discussion. If you have more 34 than a million or so rows in a spreadsheet, you probably want to store it in a 35 relational database, such as MySQL. Relational databases offer a flexible way 36 to store, manipulate and retrieve data using a Structured Query Language 37 (SQL) which is easy to learn and very useful for dealing with medium-sized 38 data sets. 39 However, if you have several gigabytes of data or several million observa40 tions, standard relational databases become unwieldy. Databases to manage 41 data of this size are generically known as “NoSQL” databases. The term is 42 used rather loosely, but is sometimes interpreted as meaning “not only SQL.” 43 NoSQL databases are more primitive than SQL databases in terms of data 44 manipulation capabilities but can handle larger amounts of data. 45 Due to the rise of computer mediated transactions, many companies have 46 found it necessary to develop systems to process billions of transactions per 47