The Great Silence of 2020 Web Design: The ‘Annihilation’ of Algorithmic Comprehensibility
By: Eleanor Amaranth Lockhart
Tags: Web Development, Data Science, Algorithms, Aliens, Scraping

One of my earliest exposures to data science research, or really any research at all, when I was growing up was SETI@Home, which when I was 11 promised a tantalizing possibility: what if you could potentially be the person to (have your computer) discover a signal from alien life. SETI@Home, many people’s first exposure to the kind of distributed processing that today makes up, among other things, cryptocurrency mining, is a pattern recognition project and in 1999 when it launched and I enthusiastically installed it on my 233 megahertz Windows 98 PC, it was my first exposure to the kind of pattern searching in data that I now do as a data researcher. Spoiler alert: my HP Pavilion never found an alien signal. Disappointing, but not remotely surprising. (This is an article about data science, but it’s also about alien first contact narratives — they’re not as unrelated as you think, and I ask that you bear with my layers of metaphors here. The Python code will come eventually!)

For anyone who ever wanted to meet aliens, Fermi’s paradox is a grim problem: shouldn’t we have, by now? Humans have been searching the stars with SETI for nearly half a decade, and have turned up no credible evidence of alien life. As suggested in Sky & Telescope by SETI astronomer Seth Shostak back in 2006, one reason SETI’s quest for extraterrestial intelligence may seem, and ultimately prove, futile, is because of the progress of technology. That is to say: if hypothetical alien intelligence has a history that mimics ours, they will have a lifespan with a very short amount of time where they are detectable by radio telescope scans like the kind traditionally conducted by SETI.

For millions of years before the radio, human beings had no appreciable radio signal impact upon the universe. At the time SETI was conceived, we had a lot — leading to the famous scene in the 1997 Jodi Foster SETI drama Contact where one of our collective history’s least admirable members became the representative of all humanity by having (allegedly, in the story) featured in one of the earliest high power radio signals. What the ultimately optimistic Contact counts on and highlights in the movie’s opening credits, which feature a stellar journey at light speed of a signal sent back in response to the Fuhrer from the Vega system, is that we will continue to transmitting — so even if Adolf Hitler’s opening to the 1933 Olympics were the first transmission an alien civilization ever saw of us, it wouldn’t be the last, because they would begin to receive more and more information in the form of high powered television and radio signals, possibly allowing them to understand us.

Dr. Ellie Arroway, a fellow Dr. Ellie, scans for alien life. Image manually retrieved from Empire Online’s review of CONTACT; copyright Warner Brothers (1997).

But according to Shostak, the amount of time when our civilization is visible in this way may actually end within a few decades of when Contact is supposedly set (1997) — and as Shostak’s article came out in 2006, we may already be past that era. We publish reams and reams of data — as I know well, being a scholarly researcher and data science — some of which we would probably like aliens to see, and a lot of which we very much would not. But the chances are miniscule aliens will see our civilization today, in its greatness (the music of Evanescence), its horribleness (the speeches of so many present-day authoritarian leaders), or even its mediocreness (Avengers: Endgame), is miniscule, because we move all that data over fiber optic cables, or at most, terrestrial-directed satellites whose data links don’t bounce off into space.

I am not an alien, but I am a data researcher working on creating a portfolio of web-data based projects with actionable findings to answer questions in which I am interested — questions which may relate to that great, mediocre, or horrible mass media of our time, or to the subject of my doctoral dissertation, mass online harassment related to “nerd” culture. And in that sense, today’s web is increasingly hostile and, yes, alien — not necessarily or only in terms of its denizens and what they put there, but in terms of its architecture and structure. This is a complex issue with positive and negative aspects, and I want to dig into these problems a bit, using the concept of alien contact as a metaphor, while discussing some of the challenges I’ve faced in data science research, how I overcame them when I was a university-based academic researcher, and how I’m overcoming them as an independent researcher now. In essence, however, I am concerned that in terms of the distribution of information on the web, we are entering an end to the “broadcast era” of easily available data for research, moving to a number of new communication standards that carry great advantages for performance, privacy, and other concerns, but which stand to make a huge amount of human knowledge that probably should be public effectively invisible to researchers, just as our civilization may now be to any alien SETI programs that are out there.

Web X.0 and Data Harvesting

When SETI@Home was getting started and I was an enthusiastic preteen messing around with whatever programming tools I could find on my aforementioned 233 Megahertz Intel Celeron desktop with a CRT (capable of up to 1024x768 display resolution!) I, of course, had a Geocities. And an Angelfire, and a FortuneCity, and at least half a dozen other sites on free web hosting providers long since lost to the churn of history (but probably archived somewhere on the Internet Archive, which I’ll talk about in a little while more extensively). HTML 3.0 and then 4.0 were the languages in which I created my first public facing web content, even though I had used BASIC for Mac and Microsoft QBASIC to write rudimentary games before that, as seems to be the programming “origin story” for most of my coder generation. Most of these free website tools offered a WYSIWYG (What You See Is What You Get) design interface, but it was limited by the lack of what we would now call responsive features in browsers, so those of us who were enthusiastic about our Star Wars fan pages learned HTML. This was Web 1.0, and at its more “mature,” Web 1.0 offered a quite organized, dignified, and structured insight into content that once would have been hosted on non-web-based mainframes and servers.

Academic and government institutions shared scientific findings — one of my favorites at the time was of course NASA’s Jet Propulsion Laboratory, which shared and continues to share under newer web paradigms images of deep space collected by Hubble and other observatories. Accessing JPL to get a new space-y desktop background in those days was pretty simple: you’d navigate to the website, probably from a bookmark (we still have those, but does anyone really use them? Be honest…) and then select the kind of image you were looking for (pictures of planets in our solar system, extrasolar imaging, etc.), and eventually you’d either get to a page that embedded the image you were looking for with a standard <img> tag, or you’d get a description with a link to an FTP directory that you could access in your browser (technically leaving the “web” altogether) to download various resolutions and formats of the picture.

Although the HTML4 standard was technically released in 1997, just a year after I started using the Internet and about a year before I learned to write in HTML, most amateur web designers and many professional web designers would continue for years — some to this day, for nostalgia or ideological reasons — to more or less use HTML to describe visually what the user was going to see, within the heavy set of constraints that Netscape Navigator and Internet Explorer offered at the time. While undoubtedly many government and academic sites adopted the growing standard of using <div> tags to make readable markup code, my review of the Internet Archive’s copy of JPL’s front page from September of 2000 finds that very little useful information is contained in what we now call the DOM (Document Object Model) of the page. The front page has a blurb about an article related to the discovery of a new “dog bone” asteroid, and if this were the modern JPL website, <div> tags would set the title, text, link, author, and other content and meta-content of this article aside in such a way that I could write an automated script to, say, check JPL daily and save new articles to a local database. In 2000, all I would have been able to do was save the page — it’s unlikely that even studying the exact structure of the page would have allowed me to make a reliable “scraper” that persisted across multiple iterations of the page itself.

Ah, Web 1.0 design. So crisp and clean, and yet so inscrutable to algorithms!

(I know “scraping” is something people have ethical concerns about — I do too, and I’ll discuss them subsequently, but in my opinion they’re essential to the practice of many kinds of data research, and JPL, as a government website meant to disseminate information, is a great example of a page that there should be minimal ethical concerns about scraping.) The point I’m making here is that even websites that were designed to disseminate data “back in the day” were still mostly only accessible to manual traversers of the web, which wouldn’t have been great for someone trying to do the sort of research I do now back then. One of the things that incentivized web developers to adopt more computer-comprehensible HTML structure was search engine optimization (SEO): the same technology that boosted your page rank by making it friendly to Google’s web crawlers also made it useful for more niche automated web programs, like those used by data scientists to scrape pages.

If “Web 1.0,” insofar as that’s a thing we can delineate, was the web before its “broadcast era” where researchers could find and collate data, like alien intelligences watching transmissions from a vast difference, “Web 2.0” was/is that era itself, for better and for worse. Because of my research interests, I’m focused on the better, but I intend to acknowledge the worse too. As dynamically generated server-side webpages became more ubiquitous, it became easier to predict where exactly in the webpage’s code — the DOM — particular information would fall. The proliferation of content management systems (CMSs) mean that you can look at a blog written in WordPress and find tell-tale signs that code can identify algorithmically reflecting all of the blog post’s content and metadata. By the 2010s, managing entire sites and even businesses in WordPress became a fairly normal and popular thing, and this was just one example. Because computers were generating the structure of the page, it was very easy to reverse engineer, and it was this environment when beloved scraping tools like Beautiful Soup and Scrapy were developed for. RESTful APIs, which existed in a sort of undefined raw form in Web 1.0 in the sense that you could often reverse engineer CGI queries that used URLs to communicate with the server, were formalized in the ’00s and meant that data queries to a server sent by a human couldn’t be easily distinguished from queries sent by a bot (or an alien!)

This era is not gone, and it probably never will entirely be gone. But something I found to be rather confusing as I was doing research for this article (after hitting a wall with my data collection) was numerous articles about the idea of Web 3.0 being allegedly a “more open” web. As a data scientist who needs data I simply cannot collect manually, I cannot agree with this, and it’s not the direction I see things going. Right now, there are pages and sites on the Internet from which I can get useful information, and pages and sites from which I cannot. An example of a website that still works according to the machine-generated, machine-readable paradigm for the moment is mass media review site Metacritic, which covers movies and television but largely deals with video games. Right now I am working on a number of projects in which I try to answer research questions about both the public opinion about and financial success of media properties that include minority characters (especially LGBT people and women) in leading roles, and accessing the raw opinion of the online public is an important factor in this. Because Metacritic is the dominant location in gaming for both collating formal critic reviews and for users to post their own opinions (often in famously toxic ways), I have access to this kind of raw opinion when it comes to games — but not for movies, where Rotten Tomatoes, which I’ll discuss in a moment, fills much the same role. (People post reviews of movies to Metacritic, and its indexing of professional reviews is just as good as Rotten Tomatoes’, but it doesn’t reflect the raw public sentiment about movies the way it does about games.) To gain access to critics’ reviews of The Last of Us Part II, a controversial game that produced a vast diversity of opinions, I wrote a Python script to obtain the URLs of all those reviews (which I was later able to obtain the text and metadata of very simply with the Newspaper3k library, which specifically targets news and news-like sites to find the contents and authors/titles of articles).

This is a really simple solution, and I was able to add some database code and change up the specific elements I was looking for and extract over 25,000 (very angry!) user reviews of the same title from the user reviews section, and place them in a local PostgreSQL database. (All of this was done consistently with the robots.txt for the site, for folks concerned with ethics.)

You can’t do this for RottenTomatoes. You can’t do it reliably with gaming review site Polygon either. RottenTomatoes is especially egregious with its user reviews, and its “Web 3.0” method of doing things is why. (Polygon is crawlable to an extent, it just doesn’t support deep dives into the archives, even for humans, which is sort of a separate issue.) RottenTomatoes loads 8–12 user reviews of a given film at a time, and you can only access the next set by clicking, at which point an API request using JavaScript is sent to the server (I’m working on reverse-engineering this right now to figure out exactly what kind of request it is) and you have to keep clicking, manually. The testing library Selenium can simulate this sort of thing by trying to fool the server into thinking it’s a human (this is the sort of thing that Captchas are designed to prevent, of course, but as far as I know RottenTomatoes doesn’t use those, nor does it actually forbid scraping — it just makes it really hard), but Selenium is not really for scraping. The point of the library is to do test-based development on JavaScript sites, and its ability to help with scraping is sort of a bonus. It’s not reliable, it’s not what it’s designed for, and changes to the site that break scraping scripts are going to be much harder to reliably fix when Selenium has to be put in the mix. The data isn’t inaccessible, but it’s exponentially hard to get to — and this is becoming a trend.

I’m also studying web development, and something I react in my JavaScript text made me feel very depressed and inspired this post: it was simply a basic description of the origin and purpose of GraphQL, which is designed in part to supplant RESTful APIs. GraphQL is a way of directly sending queries to the server’s backend, developed by Facebook to allow for the very complex kinds of queries the Facebook client has to make, many of which would be difficult to execute with a RESTful process. (The “Graph” in the name may seem confusing until you realize it refers to Facebook’s Social Graph.) Medium, incidentally, uses GraphQL.

The thing that has me broken up about GraphQL is that it takes us back in time, almost. Back when I had my 233Mhz Celeron, there was the web and there were applications, or, you know, “programs” as we used to call them. These days video games and specialized software are close to the only “programs” we run locally — even things like the Atom Text Editor are essentially JavaScript webpages running inside a light virtualization layer. The weird thing is that running programs locally didn’t use to make them necessarily understandable to the user — they were usually compiled code in a low-level language like C, so unless you could read binary Matrix style or were one of those super-haxxorz who could figure out memory locations that were being referenced like people used to do to cheat at games, you pretty much knew what the program’s UI told you. The web was somewhat unique in that you could always “view source” and could often decipher API requests being sent from traditional types of web pages and even some “web applications.”

Today, I believe the dominance of the web application is taking us back to that era when data is obscured and we only see what developers want us to see. This is quite justifiable in a lot of cases — obviously, if your doctor’s office or your university maintains a site that contains information about your health or your grades, you probably don’t want that to be easily reverse engineered. Infosec is good! But as design principles for public facing web sites — like RottenTomatoes — become entangled with “responsive” web design that depends heavily on queries that are difficult or even impossible to emulate, massive amounts of public information, such as the sentiments people have about a movie, vanish from a research point of view. Us aliens, us researchers, can’t access it — and honestly, how useful is it to the average viewer either? 12 random user reviews of a movie are unlikely to reliably tell you if you want to see it or not, not without some way to sort and filter it.

The Researcher’s First Contact: Ethics and Destructive Probes versus the Quest for Knowledge

I mentioned I was going to talk about the ethics of scraping. As I’ve noted, the two websites I refer to scraping above, RottenTomatoes and Metacritic, don’t forbid scraping, and Metacritic had no problem with me downloading 25,000 reviews of a game — twice, even, once to raw text files and once into an SQL table. It’s clear RottenTomatoes is scraping-hostile, but they don’t say you can’t do it. And this brings me to the question of what should be scraped, and to two other first contact stories about actual aliens, both of which take a much darker look at the prospect of encountering sentient life. When we consider the potential and existing harms of web scraping, but also the consequences of silencing the exchange of web data for research, I think these provide worthwhile food for thought.

In Peter Watts’ novel Blindsight (hosted in a very Web 1.0 HTML format), the “broadcast era” explanation for the failure of SETI intersects with another popular explanation: the deadly or destructive probes scenario, known probably most famously today as the inspiration for the Reapers in Bioware’s Mass Effect video game series, which were ultimately inspired by Fred Saberhagen’s Berserker cycle. In Blindsight, an alien intelligence that the narrator refers to as Rorschach, a sentient starship, arrives in our solar system and begins plotting our doom, because it views sentient, conscious life as a threat. The story takes place in our future, and Siri, the narrator, infers that Rorschach began traveling to destroy us as a result of our broadcast era — it detects signals indicating sentience and consciousness, and it seeks them out to destroy them. Blindsight, a sort of science fiction horror novel and an inversion of many of the positive first-contact tropes found in Contact or Interstellar, posits that making information visible and intelligible can be an existential threat.

Saberhagen’s Berserkers, like Rorschach, are a terrifying idea of an artificial intelligence that does NOT want to be “like us.”

And it’s absolutely correct, if we look at how human communication has been reshaped since the founding of Facebook in 2004. (Not to lay all of this at Facebook’s feet, to be clear; I suspect this trend would have occurred regardless, and Facebook isn’t even currently the driving force in many of the dangerous elements of open communication.) Although there was a lot of strange paranoia that, to be honest, wasn’t really founded in anything real when I was a kid about how you shouldn’t share your real name online because you might be kidnapped by people-kidnappers, the reality has caught up to the paranoid fantasy. Social media has exposed so much of our lives, and even people who think they are careful, think they don’t reveal themselves, slip up.

We know marketing companies have tons of data on us (and I’m not against this, to be clear! Marketers want to sell you things or persuade you — not harm you!) Unfortunately, sites like the Internet Archive — which I use, and whose archives of JPL I link in this article itself — also make it essentially impossible to erase a digital footprint. The Internet Archive relies on web scraping, incidentally, and it’s highly likely that its crawler is thwarted by the kinds of responsive design that also thwart my research crawlers — which means that if you’re building a personal website with data you don’t want to persist forever, maybe you should be building with complex React components and GraphQL and all the rest! But of course, we don’t do personal websites anymore, except as essentially digital business cards that we really have zero problem being archived forever. All the stuff we might wish would vanish, that could be used to destroy us like Rorschach wants to destroy humanity, lives in social media — in things like Internet Archive archiving deleted tweets, for instance. A true definition of a double-edged sword, archiving tweets allows us to hold political leaders accountable, but also, as I document extensively in my doctoral dissertation, foments harassment against marginalized groups and people. If someone’s personal information or secrets are exposed on the social web, erasing that data from amoral archives that, like the Rorschach spacecraft, function without true awareness or consciousness of what they are doing, is nigh-impossible.

So I understand why web crawling, “web bots,” and so on, have the reputation they do — of alien invaders, here to destroy. Because, sometimes they are. And I am not an “information wants to be free” person — if anything, I frequently feel we could take a step backwards, back toward pseudonymity, back toward privacy, even as I recognize that that’s likely impossible. But research is research, and the twin examples of Metacritic and RottenTomatoes provide an example of information, and the answers to questions, being rendered utterly inaccessible — “annihilated,” if you will — by modern web design paradigms. One research question I want to complete as part of my data science research portfolio is to compare audience reactions to the video game The Last of Us Part II with the film Atomic Blonde. This is both because I’m a fan of both of these, and because they are similarly rare entities — media about a lesbian or bisexual woman who engages in a lot of fighting and combat, something typically reserved for heterosexual men in media. I want to compare how people react to them. But I can’t, because I can only get information about The Last of Us Part II’s user reviews from a meaningful source. I need that RottenTomatoes data, and I’ve got hard work ahead to access it, if I even will be able to at all. No one’s privacy is being protected by this — people post these reviews publicly, they do show up in public search results with whatever user name they entered, and in any case I have no need to identify specific users, so in the case of my Metacritic user review scraping script, I simply scraped only the review text, because I’m interested in natural language processing and sentiment analysis, and I only need the text.

In the 2018 first-contact thriller Annihilation, an alien intelligence finds itself in the American South seemingly by accident, and Natalie Portman’s character is sent in to investigate after her husband, played by Oscar Isaac, disappears on an earlier expedition. Like Blindsight, Annihilation has horror at first contact with alien life, rather than the wonder of Contact.

The characters in Annihilation have good reason to fear the unknown — and yet that unknown simply wants to gain knowledge about them.

The meteor that impacted a gulf coast lighthouse is apparently an existential threat to life on Earth, growing an increasingly wide zone where its DNA blends with the DNA of terrestrial life. It’s a menace. It’s also trying to understand — which is why it sends back what appears to be a duplicated version of Portman’s husband, and the ending implies that Portman herself has been replaced with a clone of some kind, despite her successful destruction of the lighthouse in the film’s climax. What I see as the real tragedy in Annihilation is not that characters die, but that the hostility between the humans and the life/intelligence carried by the meteor might well result from an inability for the meteor to research humans and their culture. “Broadcast era” aside, it crashed into Earth (apparently) by mistake, it’s just here — it probably wouldn’t have been listening for transmissions until it got here in any case. And yet it’s here now, and the only way it can understand is through apparent hostility. Thus, it creates simulacra of humans to enter our culture and understand.

Web scraping puts us somewhere between the aliens in Contact and a less homicidal version of the “aliens” from Annihilation. We are forced to use deception — browser user agents, “headless browsers” such as Selenium, and so on — to gain access to a vast volume of data. Without this data, we will not understand, and we will make mistakes like the beings in Contact, who sent back images of Adolf Hitler without understanding that humans would find this quite threatening. They did not understand the context. Sites that reflect the common ideas and desires and emotions of humans need to be accessible, and certainly shouldn’t be made inaccessible to scraping simply because of design trends (rather than an intentional decision to shield data that should be shielded). The protection of privacy from Rorschach-like threats must be distinguished from the hiding of data from legitimate data scientists. The “annihilation” of public information from analytical use is a tide I fear deeply as a researcher.