It’s virtually impossible to study the entire internet; there are over a billion websites, with many times as many webpages. Since there’s no list of them all to choose from, performing a random sample of the internet is off the table. Even if it were possible, most people only see a tiny fraction of those websites regularly, so a random sample may not even capture the internet that most people experience.We ended up using the websites of the Russell 1000, the top U.S. businesses by market capitalization, which we hoped would be representative of trends in mainstream, corporate web design. We also studied two other sets of sites, one with Alexa’s 500 most trafficked sites, and another with sites nominated for Webby Awards.Because we were interested in the visual elements of these websites, as data, we used images of their web pages from the Internet Archive, which regularly preserves websites. And since we wanted to gather quantitative data comparing millions of website pairs, we needed to automate the analysis process.To do that, we had to settle on a definition of “similarity” that we could measure automatically. We investigated both specific attributes like color and layout, as well as attributes learned automatically from data using artificial intelligence.For the color and layout attributes, we measured how many pixel-by-pixel edits we would have to make to transform the color scheme or page structure of one website into another. For the AI-generated attributes, we trained a machine learning model to classify images based on which website they came from and measure the attributes the model learned. Our previous work indicates that this does a reasonably good job at measuring stylistic similarity, but it’s very difficult for humans to understand what attributes the model focused on.