Build and Analyze Knowledge Graphs with Diffbot | by Khuyen Tran | Towards Data Science

What do you often do when you want to learn more about a person? You might decide to read about a person using websites such as Wikipedia.
However, the text can be lengthy, and it might take you a while to get the pieces of information that you need.
Since we are better at absorbing information in form of images than text, wouldn’t it be nice if you can get a graph like below when searching for Steve Jobs?
That is when Diffbot comes in handy. In this article, you will learn how to build and analyze knowledge graphs using Diffbot.
A knowledge graph represents a collection of entities, which are objects, events, abstract concepts, and illustrates the relationship between them. Knowledge graphs are used by Google to enhance its search engine’s results.
Diffbot is the world’s largest knowledge graph that allows you to access a trillion connected facts across the web.
To start with Diffbot, sign up for a 14-day free trial. This free trial will allow you to search Diffbot’s Knowledge Graph.
After logging in, click the Search option to search the Diffbot Knowledge Graph.
Have you ever wondered:
which school Apple’s employees went to,
what their majors were,
and what the most popular roles at Apple are?
We can answer these questions using Diffbot’s Search. Start with searching for the Diffbot ID of Apple Inc. by:
selecting “Organization”
filtering by “name” that “contains” “apple”
clicking “Search”
You will see 61,820 results that are related to apple. Since we are only interested in Apple Inc., click “Apple Inc.”
You will be directed to the website that shows specific information about Apple Inc. Copy the ID in the website’s URL. The ID found in the URL below is “EHb0_0NEcMwyY8b083taTTw”.
To search for all employees in Apple Inc, click the Query tab then pass:
type:Person employments.employer.id:”EHb0_0NEcMwyY8b083taTTw”
... to the search box.
After clicking “Search”, Diffbot will return over 200k results.
Click the graph icon in the top right corner to view the knowledge graph of the results.
And you should see the knowledge graph like below! This graph shows the relationship between Apple Inc. and other entities. The relationships with these entities can be “city”, “employer”, “metro area”, “institution”, etc
Note that this graph only shows the top 25 results. To get the knowledge graph of the next 25 most important results, click the “More” button at the top of the graph.
To get a better view of a certain node, you can move it to another location.
To view the information of each entity, click that entity. You should see its information like below:
To get the results in JSON format, click the JSON tab in the top right corner.
What are the skills of Apple’s employees? We can find that out by looking at the skills section in the JSON file.
One employee can have many skills and different employees have different skills. Is there a way that we can summarize these skills? That could be easily be done by adding facet:skills.name to the previous query:
type:Person employments.employer.id:"EHb0_0NEcMwyY8b083taTTw" facet:skills.name
facet will display summary data about the values of the specified field
skills.name tells Diffbot to access the name field within the skills field in the JSON file.
Top 50 skills of employees at Apple:
What were the majors of Apple’s employees in college? We can find that out by adding facet:educations.major.name to the query:
type:Person employments.employer.id:"EHb0_0NEcMwyY8b083taTTw" facet:educations.major.name
The top 25 majors of employees at Apple are:
From the graph above, we can see that the top majors are related to computer science, engineer, and business administration. The results are not surprising since Apple is a technology company that specializes in consumer electronics, computer software, and online services.
What are the top schools that Apple’s employees went to? That could be found by adding facet:educations.institution.name to the query:
type:Person employments.employer.id:"EHb0_0NEcMwyY8b083taTTw" facet:educations.institution.name
The top 5 schools that Apple’s employees went to are:
Standford University
University of California Berkley
SJSU
Harvard University
University of Phoenix
We can see that a lot of these universities are the country’s top universities and are located in California. This doesn’t mean that you need to go to these top universities to get accepted in Apple. It does mean that if you go to these universities, you might have a higher chance of being accepted to Apple.
Have you ever wondered:
what artificial intelligence companies’ revenues are,
which countries these companies are located at,
and how many employees are working in these companies?
Let’s answer these questions using Diffbot. Start with selecting “Organization”, then filter the search by industries. Click the dropdown box and scroll down until you see the “Artificial Intelligence Companies” option.
The knowledge graph of the first 25 companies showing up in the search results:
What are the top 50 descriptors of these artificial intelligence companies? Let’s find that out by adding facet: descriptors to the previous query:
type:Organization industries:"Artificial Intelligence Companies" facet: descriptors
What are the revenues of these companies?
type:Organization industries:"Artificial Intelligence Companies" facet: revenue.value
The majority of these companies have revenues ranging from $100k-1 million.
To get companies whose revenues are in a certain range, you can add the following to your query:
type:Organization industries:"Artificial Intelligence Companies" revenue.value>=100000 revenue.value<500000
Another interesting thing to find out is which countries have the highest number of AI companies.
type:Organization industries:"Artificial Intelligence Companies" facet: location.country.name
Locations of the top 25 countries:
Names of the top 5 countries with the highest number of AI companies:
What are the average numbers of employees in these companies?
type:Organization industries:"Artificial Intelligence Companies" facet: nbEmployees
The majority of companies have a number of employees ranging from 5 to 100 employees. There are only 4 companies with more than 100k employees. What are they?
Let’s find that out by clicking the number in the table below the graph:
There are only 4 companies in the world with more than 100k employees and they are all Japanese companies! This could be because many Japanese workers are loyal to their companies.
These results are cool. But what if you want to export the data from Diffbot for further analysis? That can be done using Python or command-line.
To use the Knowledge Graph API, make an HTTP GET request to the following endpoint:
https://kg.diffbot.com/kg/dql_endpoint?token=...&type=query&query=...
We can also create a function that returns the endpoint. This function takes:
Diffbot’s API token
the URL of the JSON file we want to download
The token can be found in the top right corner of your Dashboard.
The URL of the JSON file can be found by clicking the JSON tab in the top right corner.
Use the URL of the page you are directed to as the second parameter of the function get_diffbot_json .
And you will get a downloadable URL:
https://kg.diffbot.com/kg/dql_endpoint?token=your-token&type=query&query=type%3APerson+id%3A%22EyauXV2QZOW2fEP8Cb_Hoxg%22&from=0&size=25&output_format=
To download the data in the URL returned by the get_diffbot_json , use the requests library:
And the JSON file will be saved in your local machine!
You can also download the JSON file on your terminal using wget :
wget https://kg.diffbot.com/kg/dql_endpoint?token=your-tokenc&type=query&query=type%3APerson+id%3A%22EyauXV2QZOW2fEP8Cb_Hoxg%22&from=0&size=25&output_format=
Congratulations! You have just learned how to build a knowledge graph and summarize essential facts using Diffbot. I hope this article will give you the motivation to explore other topics that you are interested in.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: