Implementing GraphX/Graph-frames in Apache Spark
By: Neupane Parlad
Tags: Apache Spark, Graphx, Data Science, Programming, Python

Example

Find all the edges connected to Andrew

Find all incoming connection to Andrew

Determine the number of links coming out from Andrew using the outDegrees

Triangle Count Problem

This count column identifies how many triangle relationships the vertex is participating in.

The output shows that Andrew and Sierra have the maximum triangle counts, since they are involved in three kinds of relationships apply a PageRank Algorithm. It was originally started by Google’s founders to identify the most important pages on the Internet. The PageRank algorithm measures the importance of each vertex in a graph. Assume a scenario where one Twitter user has 10 important followers, and each of those followers has multiple followers in turn. This is to say that the PageRank algorithm considers each important follower a legitimate endorsement of the Twitter user and thereby gives a higher ranking to the user.

>>> pageRank = graph.pageRank(resetProbability=0.20, maxIter=10)

Here, we are calling the PageRank algorithm using the pageRank methods. It takes two attributes

resetProbablity: This value is a random value reset probability (alpha). maxIter: This is the number of times you want pageRank to run.

You can see from the original person’s schema that a new column has been added called PageRank and new column weight has been added to the original relationship schema. These both columns contributed to the PageRank score

What is the PageRank Score?

Let’s look at the weight: weight is the determining factor to calculate page rank

As you can see from the table William’s relationship with Andrew gets the maximum weight since it is unique. No one other than Andrew is a friend to William. By applying the Breadth-First Algorithm let’s find the shortest way to connect to the people. We are going to apply for BFS.

You might have often noticed LinkedIn telling you how far you are from any new user. For example, you will notice that a user whom you would like to connect to is a second connection or a third connection. This tells you that you are two vertices away from the vertex from where you are looking. This is one way of identifying how far a vertex is to another vertex.

Another example is finding the shortest path between two cities or two airports. Breadth-First Search is one of the shortest path-finding algorithms and it helps us identify the shortest path between two vertices.

Find the connection between Bob and William Since only Andrew is directly connected to William. So, for Bob to be able to connect to William.

What is the Shortest Path? How to Find it?

GraphFrames provides an API, called bfs(Breadth-first search), which takes a minimum of two parameters. Those are fromExpr: Expression to identify the from vertex. toExpr: Expression to identify the vertex.

graph.bfs(fromExpr=””Name=’Bob’””,toExpr=””Name=’William’””).show()

This shows the shortest path between William and Bob. It can be more practical if we try to find the real distance. You can replace people’s names with city names and find the shortest path.