Deriving Patterns of Fraud from the Enron Dataset | by Mahnoor Javed | Towards Data Science

The Enron fraud is a big, messy and totally fascinating story about corporate malfeasance of nearly every imaginable type.
In this article, we will use Python to analyze the dataset, and find out patterns and clues through data exploration, as well as build a regression model that could predict the bonus of a person at Enron based on the salaries they receive.
But first, we need to know a bit about the biggest corporate fraud in American history!
Enron Corporation was an American energy, commodities, and services company based in Houston, Texas. It was founded by Kenneth Lay as a merger between Houston Natural Gas and InterNorth in 1985.
Although it started as an energy company, it was involved in so many complex things that people didn’t understand what exactly the company did, and as Bethnay McLean asked “How exactly does Enron make its money?”.
Enron was fantastic! Very attractive to investors and the fastest growing company making money at a rate no one has ever seen before. They were the Wall Street darlings, a company that could never lose. It was well-connected politically, was making money at an incredible rate, and seemed to be the leading innovative company and very profitable investment. Fortune named Enron “America’s Most Innovative Company” for six consecutive years.
Jeffrey Skilling was hired by Kenneth Lay in 1990 as chairman and chief executive officer of Enron Finance Corp. In 1992, Skilling devised a new accounting technique called the Mark to Market, under which the value of the asset could be adjusted on the balance sheet from its historical cost up to the fair market value, and the difference be captured as the gain or revenue. So under this technique, projected revenue was captured as revenue of today and Enron reaped billions. The company’s share prices went up from $10/share to around $80/share from 1999 – 2000. Enron was also guilty of setting up side companies and moving the debts to the new companies' financial records.
Enron was thus portrayed as a very profitable investment when all numbers on the financial statements were illusionary and fake!
Enron filed for bankruptcy on December 2, 2001. Arthur Anderson, one of the “Big Five” accounting firms, which audited the financial statements was dissolved as a result of the Enron scandal.
The Enron email and financial datasets are big, messy treasure troves of information, which become much more useful once you know your way around them a bit. Enron’s complete data may be downloaded from this link here, and the refined pickle files may be downloaded from the following Github repository along with the complete code used in this article.
github.com
First of all, we will explore the dataset we have. Enron data is stored as a pickle file, which is a handy way to store and load Python objects directly.
The aggregated Enron email + financial dataset is stored in a dictionary, where each key in the dictionary is a person’s name and the value is a dictionary containing all the features of that person.
Let us find out the number of people in the dataset. We will first load the pickle file to retrieve the pickled data and store it in the variable enron_data (which is a dictionary). We will use Python’s len() command to calculate the length of the dictionary loaded from the pickle file, and print the value.
import pickleenron_data = pickle.load(open("D:\ML Project\Enron Case\enron_data.pkl", "rb"))print("No of People in the Enron Dataset", len(enron_data))
As can be seen from the variable explorer, enron_data is a dictionary with 146 keys. It means there are 146 data points or persons we will be dealing with in our dataset.
Moreover, it can be seen that enron_data is a nested dictionary. We have dictionaries within the dictionary and we have a total of 21 features for each person in the Enron dataset.
Let us access the features of the nested dictionary. We will first get the first dictionary item of enron_data and then use it to print its features:
first_item = list(enron_data.keys())[0]print(enron_data[first_item].keys())
The 21 features are:
‘salary’, ‘to_messages’, ‘deferral_payments’, ‘total_payments’, ‘loan_advances’, ‘bonus’, ‘email_address’, ‘restricted_stock_deferred’, ‘deferred_income’, ‘total_stock_value’, ‘expenses’, ‘from_poi_to_this_person’, ‘exercised_stock_options’, ‘from_messages’, ‘other’, ‘from_this_person_to_poi’, ‘poi’, ‘long_term_incentive’, ‘shared_receipt_with_poi’, ‘restricted_stock’: , ‘director_fees’.
Let us explore the ‘poi’ feature. POI stands for “Person of Interest” and includes all the people who are indicted, charged with crime, and people who settled without admitting guilt and testified in exchange for immunity from prosecution in the Enron Scandal.
The POI list had been prepared by Katie from Udacity’s course “Intro to Machine Learning”. The files can be found in the GitHub repository.
The following is the code to calculate the number of People of Interest in the dictionary. Out of the 146 people in the enron_data dictionary, the number of POI is 18.
people = 0poiCount = 0for person in enron_data:    people = people + 1    if enron_data[person]['poi'] == 1:        poiCount += 1print("No of POI:", poiCount)
Out of the 146 people in the enron_data dictionary, the number of POI is 18.
poiFile = open("D:\ML Project\Enron Case\poi_names.txt")poiCount2 = 0lines = poiFile.readlines() for line in lines:    if line[0] == '(':        poiCount2 += 1print("Total POIs:",poiCount2)
There are a total of 35 POIs in the text file.
Let us dive into the data available for the CEO at Enron: Jeffrey K Skilling.
jeffrey_k_skilling = enron_data['SKILLING JEFFREY K']print(jeffrey_k_skilling)
On printing we get the following information:
Now, let us access the features of the different people in the dictionary. Following is the code to access and print Skilling’s and Lay’s salary, bonus, and total stocks value:
print("Skilling's Salary:", enron_data["SKILLING JEFFREY K"]["salary"])print("Skilling's Bonus", enron_data["SKILLING JEFFREY K"]["bonus"])print("Skilling's Total Stocks Value", enron_data["SKILLING JEFFREY K"]["total_stock_value"])print("Lay's Salary:", enron_data["LAY KENNETH L"]["salary"])print("Lay's Bonus", enron_data["LAY KENNETH L"]["bonus"])print("Lay's Total Stocks Value", enron_data["LAY KENNETH L"]["total_stock_value"])
Let us find out how many people at Enron had quantified salary:
people = 0have_quantified_salary = 0for person in enron_data:    people = people + 1    if enron_data[person]['salary'] != 'NaN':            have_quantified_salary += 1print(have_quantified_salary)
95 people from the 146 persons in enron_data have a quantified salary. The rest have NaN values in their salary features. This may pose problems while applying regression so we will remove them.
Now let us see the range of the salaries of the Enron employees in our dataset. We will use plt.hist from the Matplotlib library in Python and create a histogram to visualize the salaries. We will also be using featureFormat function which will convert the dictionary list into a numpy array:
feature1 = ["salary"]enron_data.pop('TOTAL', 0)salary = featureFormat(enron_data, feature1)counts, bins = np.histogram(salary)plt.hist(bins[:-1], bins, weights=counts)
As can be seen, most of the employees had their salaries within the $0.1M — $0.3M range. We have a few outliers here as well. As can be seen, these are the data points for receiving salaries greater than 1 million USD.
Let us find out the outliers and who is receiving such salaries:
for x in salary:    if x>1000000 :        print(x)
The salaries beyond $1M are the following:
Let us see who earned these salaries:
for person in enron_data:    if enron_data[person]["salary"] == 1072321:        print(person)    if enron_data[person]["salary"] == 1111258:        print(person)    if enron_data[person]["salary"] == 1060932:        print(person)
Let us do the same with the ‘bonus’ features:
feature2 = ["bonus"]enron_data.pop('TOTAL', 0)bonus = featureFormat(enron_data, feature2)counts, bins = np.histogram(bonus)plt.hist(bins[:-1], bins, weights=counts)for x in bonus:    if x>5000000 :        print("Outlier:", x)
Hence, Lavorato John J, Lay Kenneth L, Belden Timothy N and Skilling Jeffrey K are the 4 people who earned the highest bonuses during their employment at Enron. What’s interesting is that although Lay and Skilling have been known for their fraud, we get additional 2 persons. Are these POIs as well? Let’s check:
if enron_data["LAVORATO JOHN J"]["poi"] == 1:    print("John J Lavorato is a Person of Interest (POI)")if enron_data["LAY KENNETH L"]["poi"] == 1:    print("Kenneth L Lay is a Person of Interest (POI)")if enron_data["BELDEN TIMOTHY N"]["poi"] == 1:    print("Timothy N Belden is a Person of Interest (POI)")if enron_data["SKILLING JEFFREY K"]["poi"] == 1:    print("Jeffrey K Skilling is a Person of Interest (POI)")
Now let us use some machine learning algorithms that would predict the bonus knowing the salary of the person.
For this, we will be using sklearn’s linear regression.
But first, let us plot the scatterplot to visualize how the salary and bonus of a person are related. We will use the features “bonus” and “salary” and define them in our own features_list.
enron_data.pop('TOTAL', 0)features_list = ["bonus", "salary"]data = featureFormat(enron_data, features_list, remove_any_zeroes=True)target, features = targetFeatureSplit( data )
We will then use the sklearn.model_selection’s train_test_split to split the data into training and testing data.
from sklearn.model_selection import train_test_splitfeature_train, feature_test, target_train, target_test = train_test_split(features, target, test_size=0.5, random_state=42)train_color = "b"test_color = "r"
Lastly, we will plot the scatterplot using matplotlib:
import matplotlib.pyplot as pltfor feature, target in zip(feature_test, target_test):    plt.scatter( feature, target, color=test_color ) for feature, target in zip(feature_train, target_train):    plt.scatter( feature, target, color=train_color )plt.xlabel(features_list[1])plt.ylabel(features_list[0])plt.scatter(feature_test[0], target_test[0], color=test_color, label="test")plt.scatter(feature_test[0], target_test[0], color=train_color, label="train")plt.legend()
As can be seen, the data has been divided equally between testing and training data and are colored in the scatterplot for visualization purposes.
Now we will use Linear Regression from sklearn in order to build the regression model for our data:
from sklearn.linear_model import LinearRegressionreg = LinearRegression()reg.fit(feature_train, target_train)coef = reg.coef_intercept = reg.intercept_print('Slope: {}, Intercept: {}'.format(coef, intercept))training_score = reg.score(feature_train, target_train)test_score = reg.score(feature_test, target_test)print('Score when same data is used to train and test: {}'.format(training_score))print('Score when separate test data is used: {}'.format(test_score))
And plot the regression line.
When we have predicted the regression line, and know the slope as well as the new intercept, we can now predict the bonuses of people at Enron based on the salary they receive.
So these were some of the tasks we did with the Enron dataset. The dataset is enormous, and there are hundreds of other cool stuff you could do with it. For the sake of this article, we keep to the basic ones. You may explore further on your own! 😄