Using Graph Theory to Efficiently Solve Data Science Problems
By: Max Hilsdorf
Tags: Graph, Mathematics, Data Science, Programming, Python

Why should I learn this right now and not later?

There are plenty of things to learn in the field of data science. Typically, as a Data Scientist, you tend to learn as you go. If your current task requires new methods, you do some research until you can solve the problem at hand. However, applications of graph theory often hide where we do not expect them if we have not learned the fundamentals of this discipline. In this post, I will show you how, for certain applications, the very basics of graph theory can

speed up your code dramatically

your code dramatically reduce functions with plenty of looping, indexing, etc. to sweet one-liners

make possible certain tasks that were previously practically unsolvable

All concepts learned will be very basic and accompanied by images or code. Plus, I will not introduce more than 5 new terms.

Basic Concepts and Terminology

A graph consists of a set of nodes {a1, a2, …, and} as well as a set of edges, which connect some or all of the nodes with each other (Fig. 1). For example, let there be a set of airports {ap1, ap2, ap3}. If there are flights between all airports but ap2, which only offers flights to ap3, the edges are {(ap1,ap3), (ap3,ap1), (ap2,ap3), (ap3, ap2)}. Typically, the term “graph” refers to those graphs where all edges also exist in their reversed form (ap1,ap3)->(ap3,ap1). Else, it is called a directional graph, which is not part of this guide.

Figure 1: Example of a graph with three nodes and two (unique) edges. All images by author unless otherwise noted.

Every graph can be represented in numerical form by its adjacency matrix. In this case, it is:

As you can see, this symmetrical matrix holds a 1 where there exists an edge connecting two nodes and a 0 where there is not. A more technical description would be the number of paths of length 1 between all pairs of nodes.

That is all the theory you need to get going. Let us solve some real-world problems. You can find all of the code in this github repo.

Data Preparation

Dataset

As a real-world application, I decided on a flight dataset. Publicly available, this dataset includes all flights landing or starting at 2500 participating airports. Download the dataset with the fingerprint “md5:0df632f65e1d7b7dd6f89294e81861e0” to replicate my analysis.

Cleaning

For this analysis, I only kept the starting (“origin”) and landing (“destination”) locations as well as the timestamp (“day”). Also, for demonstration purposes, I took a random sample of 1000 flights. For an exact replication, please see the linked repository.

Analysis

To exemplify the utility of the graph theory approach, we will pose a few basic questions one could have for a data scientist regarding the flight dataset. Then, we will compare the simplicity and efficiency of a typical pandas approach to the graph approach.

Pre-Analysis: Create Adjacency Matrix

The graph method is based on manipulating the adjacency matrix. Thus, we will have to create it first. View the time spent on computing the adjacency matrix as an investment. It will be worth it.

First, we need to get a list of all unique airports in the dataset. This is very easily done using pandas and numpy (do not forget to import those).

airports = np.unique(np.append(df[“origin”], df[“destination”]))

Since python lists are ordered, every airport is now associated with a fixed numerical value. This value will represent its position in the adjacency matrix. Also, we can use these to map every string in the dataset to a numerical value, which allows for efficient computation of the adjacency matrix.

mapping_dict = {k:i for i,k in enumerate(airports)}

df_mapped = df.applymap(lambda x: mapping_dict[x])

Now, the adjacency matrix is just one step away. We are going to create an nxn null-matrix, n being the number of unique airports. Then we loop through each data frame row just once and see whether a 0 value needs to be changed to 1.

# Create null-matrix

A = np.zeros((len(airports), len(airports)), dtype = int) # Enter 1 into the null-matrix where there is an edge

for date, flight in df_mapped.iterrows():

i, j = flight["origin"], flight["destination"]

if A[i,j] == 0:

A[i,j], A[j,i] = 1,1

Alright, we have done it! Here is our adjacency matrix:

Question 1: Can you fly from A to B directly?

This question seems brutally trivial. It is! Yet, even for this kind of question, the graph method outperforms the standard approach. A common approach to solving this would be to loop through each row and return True if there is such a direct path. A function of this sort could look like this:

def exists_direct_path(df, node1, node2):



# Loop through each row

for i, row in df.iterrows():



# Check whether (node1, node2) or (node2, node1) is in the dataset

if {row["origin"], row["destination"]} == {node1,node2}:

return True



return False

Using the adjacency matrix, simply indexing it will do the trick:

A[node1, node2] == 1

But wait? Is this not a false comparison? Does the “exists_direct_path” function not do the same thing we did in the pre-analysis? Indeed, the graph method only has the edge here because we have pre-computed what the common method does now. However, that is the point! We have looped through every row once and can now check every path simply by indexing. Using “exists_direct_path”, every single call of the function will perform the same loop. That is clearly inefficient.

In fact, doing 200 of such comparisons took my laptop 18 seconds. The graph method took 0.0006 seconds. We can already see:

There are problems which are perfectly solvable using standard python and pandas methods, but terribly inefficiently compared to a graph-based method.

Question 2: How many airports is A directly linked to?

Let us attend to a slightly more sophisticated, but still simple problem. Imagine your boss wants to start a subsidiary in another city, but it is of crucial importance that the city be optimally connected to the rest of the world.

Solving this with conventional methods would mean to loop through all rows and keep track of all the possible destinations found in something like a list. This is what it could look like:

def degree(df, node): # degree is the technical term



# Setup empty list

flights = []



# Loop through every row

for i, row in df.iterrows():



# If the node is either an origin or a destination, there must be a direct path

if row["origin"] == node:

flights.append(row["destination"])

elif row["destination"] == node:

flights.append(row["origin"])



# Remove duplicates

flights = list(set(flights))



return len(flights)

Using the adjacency matrix, this is, again, much simpler:

A[node].sum()

Just like before, the runtime efficiencies are astonishing. Getting the degree of 200 nodes took my laptop 19.5 seconds, while the graph-based method took 0.02 seconds.

Question 3: How many times will I have to change when traveling from A to B?

Let us come to the last and rather sophisticated problem. Imagine a customer is planning his vacation. He wants to fly off at his local airport and has five possible destinations in mind. However, he wants to change only two times or less on his journey. In this case, I will actually start with the graph-based method. You will soon see why.

We can make use of another handy property of the adjacency matrix. Remember that earlier I defined the adjacency matrix as indicating the number of paths of length 1between every pair of nodes. It turns out that if you raise the adjacency matrix to the power of 2, you get the number of paths of length 2 and so on. In other words: If a flight between the airports ap1 and ap2 requires a minimum of three changes, you will find that: (A**3)[ap1, ap2] > 0. Hence, to find the shortest path between two airports, we need to multiply A with itself until we find a value > 0.

This trick allows us to find the shortest path efficiently by raising A to an increasing power until a path is found. Once it is found, the last number A was raised to is the minimum path length. We also need to tell the computer when to stop looking, else it will search forever until it finds a path. In this case, I set this max length to 10. If no match is found for A¹⁰, the program will just return 10.

def shortest_path(A, x, y, iterations = 10, mapping_dict = mapping_dict):



# Create copy of Matrix to work on

M = A.copy()



# Get numerical representations of airports

i, j = mapping_dict[x], mapping_dict[y]



# Define the maximum power to which A is to be raised

iterations = 10 # A = A*A until a path is found or the max iterations are reached

for k in range(iterations):

if M[i,j] == 0:

M = np.matmul(M,A)

else:

return (x,y,k)

Getting back to the problem at hand, your customer would like to fly off at “LFPO” and has the destinations “YWHA”, “LTAC”, “LIRF”, “EVRA”, and “KRFI” in mind.

home_airport = "LFPO"

vacation_destinations = ["YWHA", "LTAC", "LIRF", "EVRA", "KRFI"]

max_changes = 2

Using the “shortest_path” function, we can easily tell him which destinations can be reached with a maximum of 2 changes.

n_changes = [shortest_path(A, home_airport, destination) for destination in vacation_destinations] acceptable = [destination for location, destination, changes in n_changes if changes <= 2] acceptable

Output:

['LIRF', 'EVRA']

Now, let us implement this with a different approach that is not built on graph theory. It is not trivial to figure out how to do this. In fact, I believe the only way to solve is this is to try every combination of chaining multiple flights onto each other. Once all combinations are found, the shortest path is the combination with the shortest length of all combinations with the correct start and endpoints.

If you have ever worked with algorithms with exponential complexity, you probably already have a grasp of how bad this method is. It may work fine with a total of 5 or 10 airports. But take 1000 airports and your grandchildren will have retired before your analysis finishes. In contrast to this exponential complexity, the graph-based method has a complexity of O(n³), as all it does is multiply matrices.

It may be that some very smart people came up with algorithms that I am not aware of and that are more efficient than this brute force approach. Still, it becomes clear that:

For certain data science problems regarding the existence of certain relationships between multiple objects, knowing basic graph theory will allow you to write efficient code and solve “basic” problems which are practically unsolvable with other methods.

Conclusion

We have explored the use of graph theory approaches to answering certain (seemingly) simple data science questions. It has become clear that many problems which we would be able to solve using loops etc. are much more easily solved using graph-based approaches. Once the adjacency matrix is created, most questions regarding the existence of relationships between objects of interest can be answered using single expressions like indexing or summing. If your code is to be applied to large amounts of data or if the same problem is likely to occur again, computing the adjacency matrix once is more than worth it.

Further, some problems are practically unsolvable using common approaches, because the algorithmic complexity is incompatible with even medium-sized datasets. If such a situation occurs, you might want to go one step back and see whether the graph-based approach holds an easy solution, just like it did in this guide.

For further reading, I will link you to this great article by Vaidehi Joshi. It covers the theoretical foundation in more detail, offers more practical examples of graph theory applications, and links to some more in-depth literature.

Thank you very much for reading!