How to Scrape LinkedIn
By: Max Bade
Tags: LinkedIn, Web Scraping, Pandas, Python, Software Development

It’s become rather difficult to scrape some of the larger tech websites, such as LinkedIn. Likely due to the amount of personal information at stake. But I’m here to mess up everything they worked so hard to prevent. Also, For the record, this is for educational purposes only.

Setup Steps

Libraries

Soup Objects

CSRF Tokens

Login Request and HTML

Potentially add more pages

Scraping Metadata for Each Column

Create and Append Columns to DataFrame

Libraries and Soup Object

This part is a little tricky. Frist import the libraries, then create the request session, then I just added some link variables to different linkedin pages, then I created the soup variables and this is where it’s a little bit fuzzy. You’ll have to use the csrf line to get past the token, that linkedin requires for each request. Then you’ll pass that into the login request. Keep in mind, this uses your linkedin! I don’t know the parameters or rules for what will ultimately get you blocked, but use at your own risk. I never had any issues though.

Tips:

If you get stuck, just refresh the notebook (jupyter gets hung up some times).

This project is using a url with the query “aspiring data scientist” in the url.

Use dummy account linkedin Creds if you’re scared.

import requests

from bs4 import BeautifulSoup #create a session

client = requests.Session() #create an email and password variable

your_email_here = "some_email"

your_password = "your_password"

HOMEPAGE_URL = ‘

LOGIN_URL = ‘

CONNECTIONS_URL = ‘

ASPIRING_DATA_SCIENTIEST = ‘ #create url page variablesHOMEPAGE_URL = ‘ https://www.linkedin.com' LOGIN_URL = ‘ https://www.linkedin.com/uas/login-submit' CONNECTIONS_URL = ‘ https://www.linkedin.com/mynetwork/invite-connect/connections/' ASPIRING_DATA_SCIENTIEST = ‘ https://www.linkedin.com/search/results/people/?keywords=Aspiring%20Data%20Scientist&origin=GLOBAL_SEARCH_HEADER' #get url, soup object and csrf token value

html = client.get(HOMEPAGE_URL).content

soup = BeautifulSoup(html, “html.parser”)

csrf = soup.find(‘input’, dict(name=’loginCsrfParam’))[‘value’] #create login parameters

login_information = {

‘session_key’: your_email,

‘session_password’: your_password,

‘loginCsrfParam’: csrf,

}

Results:

Create a List out of the Soup Object

Since Linkedin (I think — and don’t quote me on this) uses a javascript injected way of storing the data, beautifulsoup was less helpful. Thus, I scraped the metadata. It’s way easier, in my opinion to scrape data inside of a dataframe, rather than raw text. Thus, I put everything in a list, put that list in a dataframe and then created a string column using map .

#create dataframe with soup contents

import pandas as pd soup = list(soup)

soup df = pd.DataFrame()

df['test'] = soup #Create string column of soup contents

df['liststring'] = [','.join(map(str, l)) for l in df['test']] df

Results:

Now that’s a good lookin’ DataFrame!

Parse Out The Interesting Data

The project was a work in progress the entire time. Scraping text to parse out bits and pieces of actual data is not glamorous. There may be a quicker way to do a lot of this in regex — I exited this step as soon as I found working code.

#parse out all the text except the interesting stuff

df1 = pd.DataFrame(df.liststring.str.split('{').tolist()).stack().reset_index()

df1.columns = ['level_0','level_1','test'] df1 = df1[df1['test'].str.contains("textDirection")]

df1 = df1[df1['test'].str.contains('type":"com.linkedin.voyager.common.TextViewModel"},"type":"PROFILE')]

df1

Results:

Getting Closer!

Let’s Make Some Actual Columns Shall We?

This code snippet creates two columns in our dataframe: name and shared_connections with the account that logged in.

pd.set_option('display.max_rows', 500)

pd.set_option('display.max_columns', 500)

pd.set_option('display.width', 1000)

pd.set_option('display.max_colwidth', None) df1 = df1.replace('"textDirection":"USER_LOCALE","text":"', '', regex=True)

df1 = df1.replace(' shared connections","snippetText":', '', regex=True)

df1 = df1.replace('type":"com.linkedin.voyager.search.SearchHitV2","headless":false,"socialProofText":"', '', regex=True)

df1 = df1.replace('","', '', regex=True)

df1 = df1.replace(' shared connectionsnippetText":', '', regex=True)

df1 = df1.replace('type":"com.linkedin.voyager.common.TextViewModel"},"type":"PROFILE', '', regex=True)

df1 = df1.replace('$$1', '', regex=True)

df1 = df1.replace('$$2', '', regex=True)

df1 = df1.replace('$$3', '', regex=True)

df1 = df1.replace('$$4', '', regex=True)

df1 = df1.replace('$$5', '', regex=True)

df1 = df1.replace('$$6', '', regex=True)

df1 = df1.replace('$$7', '', regex=True)

df1 = df1.replace('$$8', '', regex=True)

df1 = df1.replace('$$9', '', regex=True) #drop columns

df1.drop(['level_0'

,'level_1']

,axis = 1, inplace = True) #Create shared connections column

df1['shared_connections'] = df1['test'].str[-1:] #remove last 3 characters

df1['name'] = df1['test'].str[:-3] #drop test col

df1 = df1.drop(['test'], axis = 1) #rename columns

df1 = df1[['name', 'shared_connections']] df1

Results:

Now You’re Talkin’

Another Column, On Me

This bit of code creates a new dataframe with one column called test because i’m that original; this column is the title of the user — for instance, my title says “Data Engineering Consultant”.

df2 = pd.DataFrame(df.liststring.str.split('{').tolist()).stack().reset_index()

df2.columns = ['level_0','level_1','test'] df2 = df2[df2['test'].str.contains('"textDirection":"USER_LOCALE","text"')]

df2 = df2[df2['test'].str.contains('"com.linkedin.voyager.common.TextViewModel"},"nameMatch":false,"subline":')] df2 = df2.replace('"textDirection":"USER_LOCALE","text":"', '', regex=True)

df2 = df2.replace('"nameMatch":false,"subline":', '', regex=True)

df2 = df2.replace('com.linkedin.voyager.common.TextViewModel', '', regex=True)

df2['test'] = df2['test'].str[:-11]

df2 = df2.replace('","', '', regex=True) #drop columns

df2.drop(['level_0'

,'level_1']

,axis = 1, inplace = True) df2

Results:

Tight bro

Eh Eh Merge — Eh Eh NO (It’s a Friends Bit)

Lets merge the last two dataframes real quick.

#add title column

title = list(df2.test)

df1['title'] = title df1

Results:

Not. Too. Shabby. Shhh.. Abby (if you get this, we should be friends)

Shots All Around!

I really do have too much fun with these posts. Let’s add one more column called test — you heard me, this one will be for the location — if the user offered up that info on their profile that is.

#filter out location

df3 = pd.DataFrame(df.liststring.str.split('{').tolist()).stack().reset_index()

df3.columns = ['level_0','level_1','test'] df3 = df3[df3['test'].str.contains('"textDirection":"USER_LOCALE","text"')]

df3 = df3[df3['test'].str.contains('com.linkedin.voyager.common.TextViewModel"},"trackingId"')] df3 = df3.replace('"textDirection":"USER_LOCALE","text":"', '', regex=True)

df3 = df3.replace('com.linkedin.voyager.common.TextViewModel"}', '', regex=True)

df3 = df3.replace('"trackingId"', '', regex=True)

df3 = df3.replace('$type":"', '', regex=True)

df3 = df3.replace('type":"com.linkedin.voyager.search.BlendedSearchCluster"},', '', regex=True)

df3 = df3.replace('type":",:"G6x05qiqQfO29accDNG79w=="}],"type":"SEARCH_HITS","', '', regex=True)

df3 = df3.replace('$type":",:', '', regex=True)

df3['test'] = df3['test'].str[:-41] df3

Results:

Austin Texas, I see you

And To Round Out This Code

Let’s merge the last and the original dataframes together and see what we get. We’re all very excited.

#create location column

location = list(df3.test)

df1['location'] = location df1

Results:

Done Son

To Sum Up

We wrote a script that logs us in to linkedin, scrapes the first page of of a specific search, parses out name, shared_connections, title and location and put all this data into a pandas dataframe.

If you found this helpful, feel free to subscribe or smash that clap button yeah!

Source: https://github.com/maxwellbade/linkedin_scraper