Manufacturing Authenticity In the Age of Social Media
By: Asaph Kupferman
Tags: Neural Nets, Yelp, Naturallanguageprocessing

Generating fake Yelp reviews with natural language analysis and deep learning.

By: Asaph Kupferman and Jesse Xu

The past 10 years has seen a dramatic rise in the prevalence and influence that digital platforms have exerted on our lives. For the most part, this has been hugely beneficial; we can stay socially connected with our Middle School friends via Facebook, browse cat videos posted by someone in South Korea, or even look at thousands of people’s written impressions of a product before electing whether or not to purchase.

However, the nature of these platforms allow for extreme anonymity; someone can easily be creating a false digital footprint on any of these platforms. Profiles can be created with random pictures found on the internet, and reviews be completely fabricated. The inherent separation between content creator and consumer created by the internet allows for potentially nefarious actors to easily distort reality. It’s fairly well-documented that Instagram influencers (or even businesses trying to build a following on the platform) will “buy” followers for their page who do not actually exist outside of a “bot farm” somewhere across the globe. Similarly, it’s known that Amazon sellers will try to engage in similar practices to boost their product’s reviews/ratings. This is sometimes done by incentivizing customers (via direct cash payments) to leave glowering reviews, but positive reviews are also often generated by bots. During the 2016 United States Presidential Election, Russia famously ran a massive Social Media campaign through Facebook that reached nearly 126 million people. This continued throughout the 2020 United States election.

We were inspired after seeing the Yelp Dataset, a massive accumulation of millions of reviews and businesses. In the current age of social media “fakeness,” we wondered: could we also reliably produce “fake” restaurant reviews that might sway a potential customer?

Even this simple search of “Best Bunch in Philadelphia” inundates me with thousands of convincing, heartfelt reviews (both extremely positive/negative) motivated by experiences individuals have had at some of these Philadelphia Brunch establishments.

Given the impressive advances in Natural Language Processing in recent years (see: GPT-3), and the overwhelming amount of existing resources invested in manufactured authenticity on the internet, we thought it’d be interesting to explore “faking” Yelp reviews.

The Dataset

Luckily, Yelp did a good job organizing their dataset and provided ample documentation for understanding its structure. The dataset is comprised of 6 JSON files, corresponding to different pieces of information. For our analysis, we were only concerned with “business.json” and “review.json.” These files contained information on the various businesses included in the dataset (such as name, address, business category, various attributes, etc.) and a list of the reviews stored on Yelp with their relevant attributes (user, business, etc.), respectively. Due to the sheer size of the datasets, we uploaded the files to an Amazon S3 Folder (Amazon Web Services Simple Storage) and then used Apache Spark on an Amazon EMR cluster (Elastic MapReduce; a framework for distributing large computations across multiple Amazon Elastic ComputeCloud instances for faster return times, essentially virtual machines) to extract the .json files to Spark DataFrames. This was necessary due to the sheer sizes of these files, ranging in the tens of millions of records (with the column for the review records including full paragraphs!); attempting any operations using Google Colab resulted in quickly running out of RAM and a disconnected runtime. Thus, we had to run everything in Spark.

The dataset consisted of 209,393 businesses listed in business.json. However, we needed to filter for only restaurants, which we did by only keeping businesses that included one of a master list of tags we compiled. After filtering, we were left with 26,789 restaurants, and the dataset looked like this:

Image of DataFrame containing pre-filtered dataset of restaurants.

Moving on to reviews.json, importing the dataset to spark using a pre-defined schema yielded the following table:

Image of DataFrame containing pre-filtered dataset of restaurant reviews.

This filtered dataset is massive, containing 8,021,122 reviews of Yelp businesses (each one being multiple sentences, so at least 25 million sentences and 200+ million words!). One nice thing about the dataset is that each review has a tag corresponding to the relevant businesses. Therefore, we can use a simple Inner Join to combine the two datasets and drop any reviews not related to a business we’re interested in. Now we can begin our Exploratory Data Analysis (EDA).

Exploring The Data

The Businesses Dataset

First, let’s explore the business category. Where are these businesses located? The Yelp Dataset mentions that it contains information for businesses in “10 Metropolitan Areas,” but doesn’t actually list what these metropolitan areas are! Briefly glancing at the image of the businesses dataset above shows that each record contains the “city” and “state” associated with each business. Right off the bat, Montreal and Las Vegas seem to align with expectations. However, I’ve never heard of the “Scottsdale, AZ” metropolitan area. A simple Google Maps search shows Scottsdale:

Source: Google Maps. Map of Scottsdale, AZ.

So Scottsdale, AZ is actually a city contained within the Phoenix, AZ metropolitan area! This makes a lot more sense. However, how do we systematically find the 10 prevalent groups that each business belongs to?

In this case, there are many ways to skin our cat. The most straight-forward machine-learning approach would be to apply a clustering algorithm to the dataset (based off the latitude/longitude coordinates, or the Zip Codes) and see what comes up. However, before doing that, I thought it’d be worth investigating these restaurants by geographic location in a bit more detail:

Console result of finding all unique city/state values.

Interesting, let’s see what the breakdown of restaurants is by state:

Number of restaurants listed, by state

It looks like the restaurants in the dataset are primarily situated in Ontario, Arizona, Nevada, Quebec, Ohio, North Carolina, Pennsylvania, Alberta, Wisconsin, Illinois, South Carolina, with the other records appearing to be incorrectly listed. However, why do we have 11 states with a substantial number of listed businesses, if we only have 10 metropolitan areas represented in the dataset?

In order to tease out the large metropolitan areas, I first looked up the biggest cities in each of these states. For example, Arizona: Phoenix, Ontario: Toronto, Ottawa, Illinois: Chicago, Aurora, Urbana Champaign, Pennsylvania: Philadelphia, Pittsburgh, etc. Then, for each state, I found all of the unique cities listed, and sorted them by number of reviews:

Number of businesses listed, by city, in PA

I repeated this process systematically for each state represented, usually yielding one likely candidate for each state. However, something interesting happened when investigating South Carolina:

Number of businesses listed, by city, in SC

There are only 9 cities listed in South Carolina! However, looking into these cities yielded an interesting result:

Map of Fort Mill, SC. Source: Google Maps.

These cities (which is really a misnomer; many are simply “towns”) are considered part of the Charlotte Metropolitan area! Using this approach, we have found our 10 metropolitan areas: Toronto, Phoenix, Las Vegas, Cleveland, Charlotte, Pittsburgh, Calgary, Madison, Quebec, Urbana-Champaign. We can simply sort the restaurants by state (including SC businesses as in NC) and continue:

Number of restaurants listed on Yelp, by metropolitan area.

What are the most prevalent types of restaurants in each city? Here are the top 5 for each, listed:

List form of 5 most popular restaurant types, by metropolitan area.

Does it surprise anyone that 5/7 American cities have “Pizza” or “Fast Food” as the most common type of restaurant? Here are those same results, visualized, but for the top 15 types: