Mining Data on LinkedIn. Working with job titles, colleagues and... | by Anh T. Dang | Towards Data Science

LinkedIn is a social networking site focused on professional and business relationships. In this article, I will introduce some ways for mining data on it.
Making LinkedIn API Requests by using LinkedIn OAuth credentials.
Downloading LinkedIn connections as a CSV File.
Doing some data analysis.
Before we get started, keep in mind that all tutorials that require you to install the python3-linkedin library are now not working.
p̶i̶p̶ ̶i̶n̶s̶t̶a̶l̶l̶ ̶p̶y̶t̶h̶o̶n̶3̶-̶l̶i̶n̶k̶e̶d̶i̶n̶
Don’t waste time on the following code.
p̶i̶p̶ ̶i̶n̶s̶t̶a̶l̶l̶ ̶p̶y̶t̶h̶o̶n̶3̶-̶l̶i̶n̶k̶e̶d̶i̶n̶f̶r̶o̶m̶ ̶l̶i̶n̶k̶e̶d̶i̶n̶ ̶i̶m̶p̶o̶r̶t̶ ̶l̶i̶n̶k̶e̶d̶i̶n̶̶̶A̶P̶I̶_̶K̶E̶Y̶ ̶=̶ ̶'̶w̶F̶N̶J̶e̶k̶V̶p̶D̶C̶J̶t̶R̶P̶F̶X̶8̶1̶2̶p̶Q̶s̶J̶e̶e̶-̶g̶t̶0̶z̶O̶4̶X̶5̶X̶m̶G̶6̶w̶c̶f̶S̶O̶S̶l̶L̶o̶c̶x̶o̶d̶A̶X̶N̶M̶b̶l̶0̶_̶h̶w̶3̶V̶l̶'̶̶A̶P̶I̶_̶S̶E̶C̶R̶E̶T̶ ̶=̶ ̶'̶d̶a̶J̶D̶a̶6̶_̶8̶U̶c̶n̶G̶M̶w̶1̶y̶u̶q̶9̶T̶j̶o̶O̶_̶P̶M̶K̶u̶k̶X̶M̶o̶8̶v̶E̶M̶o̶7̶Q̶v̶5̶J̶-̶G̶3̶S̶P̶g̶r̶A̶V̶0̶F̶q̶F̶C̶d̶0̶T̶N̶j̶Q̶y̶G̶'̶̶R̶E̶T̶U̶R̶N̶_̶U̶R̶L̶ ̶=̶ ̶'̶h̶t̶t̶p̶:̶/̶/̶l̶o̶c̶a̶l̶h̶o̶s̶t̶:̶8̶0̶0̶0̶'̶̶̶a̶u̶t̶h̶e̶n̶t̶i̶c̶a̶t̶i̶o̶n̶ ̶=̶ ̶l̶i̶n̶k̶e̶d̶i̶n̶.̶L̶i̶n̶k̶e̶d̶I̶n̶A̶u̶t̶h̶e̶n̶t̶i̶c̶a̶t̶i̶o̶n̶(̶A̶P̶I̶_̶K̶E̶Y̶,̶ ̶A̶P̶I̶_̶S̶E̶C̶R̶E̶T̶,̶ ̶R̶E̶T̶U̶R̶N̶_̶U̶R̶L̶,̶ ̶l̶i̶n̶k̶e̶d̶i̶n̶.̶P̶E̶R̶M̶I̶S̶S̶I̶O̶N̶S̶.̶e̶n̶u̶m̶s̶.̶v̶a̶l̶u̶e̶s̶(̶)̶)̶̶p̶r̶i̶n̶t̶ ̶a̶u̶t̶h̶e̶n̶t̶i̶c̶a̶t̶i̶o̶n̶.̶a̶u̶t̶h̶o̶r̶i̶z̶a̶t̶i̶o̶n̶_̶u̶r̶l̶ ̶ ̶#̶ ̶o̶p̶e̶n̶ ̶t̶h̶i̶s̶ ̶u̶r̶l̶ ̶o̶n̶ ̶y̶o̶u̶r̶ ̶b̶r̶o̶w̶s̶e̶r̶̶a̶p̶p̶l̶i̶c̶a̶t̶i̶o̶n̶ ̶=̶ ̶l̶i̶n̶k̶e̶d̶i̶n̶.̶L̶i̶n̶k̶e̶d̶I̶n̶A̶p̶p̶l̶i̶c̶a̶t̶i̶o̶n̶(̶a̶u̶t̶h̶e̶n̶t̶i̶c̶a̶t̶i̶o̶n̶)̶
First of all, we need to create a LinkedIn App, go and visit https://www.linkedin.com/developers/apps and follow guides. I created an application named MyLinkedIn.
We need to get company verification by sending the URL to a Page Admin. Here is an example of the URL in my case.
https://www.linkedin.com/developers/apps/verification/05ab9ce8-766e-46d5-a922-5778df7db07e
If there is no company to verify your application, please create a new company.
LinkedIn provides a service that lets people bring their LinkedIn profiles and networks with them to your site or application via their OAuth based API1.
To work with APIs of LinkedIn by using Python, you don’t to install any library, only importing libraries as follows.
import requestsimport stringimport random
The next step is to gain API access, take note of the client ID and secret.
Write a script to get the access token.
# Copy the client ID, secret, and redirect URI in the fields belowCLIENT_ID    = '...'CLIENT_SECRET = '...'REDIRECT_URI = 'http://localhost:8000'# Generate a random string to protect against cross-site request forgeryletters = string.ascii_lowercaseCSRF_TOKEN = ''.join(random.choice(letters) for i in range(24))# Request authentication URLauth_params = {'response_type': 'code',               'client_id': CLIENT_ID,               'redirect_uri': REDIRECT_URI,               'state': CSRF_TOKEN,               'scope': 'r_liteprofile,r_emailaddress,w_member_social'}html = requests.get("https://www.linkedin.com/oauth/v2/authorization",                    params = auth_params)# Print the link to the approval pageprint(html.url)# Click the link below to be taken to your redirect page.
In my case, the link as below, only clicking it.
You will get access by clicking the allow button as follows. LinkedIn redirects the user back to your website’s URL after granting access (giving proper permissions) to your application.
LinkedIn redirects the user back to your website’s URL after granting access (giving proper permissions) to your application.
This means that the value of the authorization_code is AQQ3C.......JR7mSw which is the code after ‘&code=...’, but don’t include ‘&state=...’. Pass this access token to the application.
AUTH_CODE ='AQQ3C.......JR7mSw'ACCESS_TOKEN_URL = 'https://www.linkedin.com/oauth/v2/accessToken'qd = {'grant_type': 'authorization_code',      'code': AUTH_CODE,      'redirect_uri': REDIRECT_URI,      'client_id': CLIENT_ID,      'client_secret': CLIENT_SECRET}response = requests.post(ACCESS_TOKEN_URL, data=qd, timeout=60)response = response.json()access_token = response['access_token']print ("Access Token:", access_token)print ("Expires in (seconds):", response['expires_in'])
It shows the result as follows.
Making a HTTP request to retrieve personal profile.
import jsonparams = {'oauth2_access_token': access_token}response = requests.get('https://api.linkedin.com/v2/me', params = params)print(json.dumps(response.json(), indent=1))
It will return my LinkedIn profile. There are many field selectors that enable the client fetch more information from the API2.
See3 https://developer.linkedin.com/docs/fields/positions for details on additional field selectors that can be passed in for retrieving additional profile information. (The LinkedIn API limits which data can be queried, e.g. you can no longer use the API to retrieve your list of connections on LinkedIn4).
params = {'oauth2_access_token': access_token,          'fields': ["localizedFirstName,localizedLastName,id"]}response = requests.get('https://api.linkedin.com/v2/me', params = params)print(json.dumps(response.json(), indent=1))
Here, we get localizedFirstName filed and localizedLastName filed. It’s returning the result as follows.
We can use field selector syntax to request additional details for APIs.
params = {'oauth2_access_token': access_token,          'fields': ['lastName:(preferredLocale:(country,language))']}response = requests.get('https://api.linkedin.com/v2/me', params = params)print(json.dumps(response.json(), indent=1))
It shows the result.
Go download your LinkedIn data here5: https://www.linkedin.com/psettings/member-data.
Once requested, LinkedIn will prepare an archive of your profile data, which you can then download. Here is my csv file (called Connections.csv).
Now that you have a basic understanding of how to access LinkedIn’s API, let’s dig into some more specific analysis.
First of all, we load data files.
import osimport csvCSV_FILE = os.path.join('./', 'resources', 'Connections.csv')csvReader = csv.DictReader(open(CSV_FILE), delimiter=',', quotechar='"')contacts = [row for row in csvReader]
It is pointing this to our ‘Connections.csv’ file. Now, we try to know how many people are working at Google.
workingAtGoogle = 0for contact in contacts:    for t in contact['Company'].split('/'):        if (t == 'Google'):            workingAtGoogle = workingAtGoogle+1print('There are %d people who are working at Google.' % (workingAtGoogle))
I have 129 connections who are working at Google.
Next, we are looking for how many CEOs are there in my connection?
whoareceos = 0for contact in contacts:    for t in contact['Position'].split('/'):        if (t == 'CEO'):            whoareceos = whoareceos+1            print('There are %d people who are CEOs.' %(whoareceos))
It shows that there are 253 people who are CEOs in my connection.
We also try to visualize the LinkedIn Network. Load data file using the pandas library.
import pandas as pddf = pd.read_csv(CSV_FILE)print(df.head())
It lists my connections as follows.
It includes the emails, current positions and companies of everyone in my network and when they became a part of my network.
Analyzing what time we connected.
import plotly.express as pxfig = px.scatter(df, x="First Name", y="Connected On")fig.show()
Because printing rows of a CSV file is clearly not the best way to visualize or understand my network. I use plotly.express library to visualize my entire network in one place.
fig = px.treemap(df, path=['Position', Company'], width=1200, height=1200)fig.show()
Easy, right? We learned how to gain API access to LinkedIn. We also do some API requests. And lastly, we do a simple data analysis in our LinkedIn connections.