Extract data from Crunchbase API V4.0 using Python
By: Priyanshu Madan
Tags: API, Python, Crunchbase, Data Science, Job Hunting

Photo by Carlos Muza on Unsplash

Given the current pandemic situation, there aren’t a lot of companies actively hiring. Alongside applying to career sites, I wanted to have a data-driven approach to my problem. Later in this series, I’ll use NLP and a Jobs API to find jobs that best match my profile. But for now, this post focusses on identifying high growth companies from CrunchBase, which I can then target for jobs.

Recently, Crunchbase released its API v4.0 and there isn’t any comprehensive documentation available online to take advantage of this powerful platform using python. So I thought to take a stab at it.

What is Crunchbase?

If you are reading this post I am going to assume that you already know what CrunchBase is. But if you don't, in simple terms it is a platform that helps users get all the information about companies all over the world. It includes information such as Revenue, Investors, Number of employees, contact information, and more. You can visit Cruncbase Data to get a complete list of data points.

CrunchBase API allows the developers to leverage this same data that powers CrunchBase.com and allow them to make their own application or website.

For my use-case, I am going to extract information for all the companies in Los Angeles

Let's get started

Step 1: Get CrunchBase API key and request URL

Again I am going to assume you have the API key handy, but if you don't, you can visit Crunchbase Data and register to get access to the API.

Next, we need the request URL. You can visit SwaggerHub on Crunchbase Data to get a complete list of Crunchbase endpoints along with their own examples that you can try yourself.

SwaggerHub

The API URL will be of the following format:

https://api.crunchbase.com/api/v4/entities/organizations/crunchbase?user_key=INSERT_YOUR_API_KEY_HERE

Since I am searching for organizations in LA, I will be using “POST /search/organizations” URL.

Now let's get to the actual coding part

Step 2: Request data using python

Import all necessary Packages

import requests

import json

import pandas as pd

from pandas.io.json import json_normalize

We will use the request module to send an API request to Crunchbase.

Define your API user key

userkey = {"user_key":"INSERT_YOUR_API_KEY_HERE"}

Define search query parameter

The search query parameter is the query that you will pass to the request API to get the required data. (You can see example queries on SwaggerHub)

Since I am finding companies in LA, my query will look something like this:

query = {

"field_ids": [

"identifier",

"location_identifiers",

"short_description",

"categories",

"num_employees_enum",

"revenue_range",

"operating_status",

"website",

"linkedin"

], "limit": 1000,

"query": [

{

"type": "predicate",

"field_id": "location_identifiers",

"operator_id": "includes",

"values": [

"4ce61f42-f6c4-e7ec-798d-44813b58856b" #UUID FOR LOS ANGELES

]

},

{

"type": "predicate",

"field_id": "facet_ids",

"operator_id": "includes",

"values": [

"company"

]

}

]

}

The first part of the query is “field_ids”. Here I mention all the entities that I need from CrunchBase. which in this case is [identifier, location_identifier, short_description, categories, num_employees_enum, revenue_range, operating_status, website, LinkedIn].

You can get a complete list of field_ids from SwaggerHub by clicking on the required API example and then switching to “schema” under description as shown below.

How to get the complete list of field_ids

“limit”: 1000 defines the number of results the query returns. In this case, it is 1000 which is the maximum limit for Crunchbase Pro.

“query”: {} defines the actual query part. In this case, I want to find companies in Los Angeles. So I have defined the “location_identifier” value as

“4ce61f42-f6c4-e7ec-798d-44813b58856b”

The above string is the UUID (universally unique identifier) of Los Angeles.

To get UUID of anything:

Go to SwaggerHub -> GET /autocomplete -> Click “Try it out” -> type in query in the querybox -> Execute -> copy the UUID in response body.

Get UUID for anything

The second part of the query is what we want Crunchbase to return, which in this case the company data. Hence we have set “facet_id” value as “company”.

Now that we have our query setup, we will now create 2 functions that will return the number of companies and extract data and save it as a pandas data frame. Here, POST request with API URL, userkey as a parameter, and passing query as json.

def company_count(query): r = requests.post("https://api.crunchbase.com/api/v4/searches/organizations", params = userkey , json = query)

result = json.loads(r.text)

total_companies = result["count"]

return total_companies def url_extraction(query): global raw r = requests.post("https://api.crunchbase.com/api/v4/searches/organizations", params = userkey , json = query)

result = json.loads(r.text)

normalized_raw = json_normalize(result['entities'])

raw = raw.append(normalized_raw,ignore_index=True)

Now since there are more than 1000 companies, I had to loop my query till I get all my results. The way I did this was by adding “after_id” key in the query part and the last UUID as the key. By doing this the loop will fetch new data after the last UUID that was fetched.

raw=pd.DataFrame() comp_count = company_count(query) data_acq = 0 # data_acq while data_acq < comp_count:

if data_acq != 0:

last_uuid = raw.uuid[len(raw.uuid)-1]

query["after_id"] = last_uuid

url_extraction(query)

data_acq = len(raw.uuid)

else:

if "after_id" in query:

query = query.pop("after_id")

url_extraction(query)

data_acq = len(raw.uuid)

else:

url_extraction(query)

data_acq = len(raw.uuid)

This is how the “raw” data frame looks after extraction.