Using Browser Cookies and Voyager API to Scrape LinkedIn via Python
By: Pratik Dani
Tags: LinkedIn, Web Automation, Towards Data Science, API, Python

Scraping LinkedIn is becoming difficult and difficult everyday especially if you are using open Softwares like UiPath, Selenium, etc. LinkedIn soft blocks you and adds a reCAPTCHA while logging in which becomes difficult to bypass. LinkedIn’s voyager API is probably the best way to extract information from LinkedIn.

LinkedIn’s data can be used for many purposes starting from lead generation, searching for jobs, if you are a recruiter you can find apt candidates for your organization, if you want to find investors for your company, the bottom line is you can use LinkedIn to get the data to suit your need. The most important thing you must know what you are looking for.

Let’s Begin

We’ll be using LinkedIn’s voyager API to scrape a company’s about page.

See/Find Voyager API’s full documentation read here.

First of all, the most important part is to get the browser cookies that we would be using in the later stages of the process. You’ll be needing two main cookies from your LinkedIn’s logged-in session, before delving more into different types of Linkedin Cookies, let’s see how we can access our cookies, just follow these simple steps:

1. Click on the options in the google chrome (top right with 3 vertical dots).

2. After this click on more tools followed by Developer Tools (you can also reach here by using the keyboard combination — ctrl+shift+I).

Now once you’ve gained access to the developer tools, you’ll see the tabs named Elements, console, etc. Navigate to the Application tab the tabs, and copy the value of two cookies from here named li_at and JSESSIONID respectively.

Note down the values of these 2 cookies and store it somewhere we will it later on. I have blurred out the values to prevent the potential misuse of my account.

Now, that we are done with the initial part, let’s get to the interesting part — you guessed it — Coding.

Below is the full code, Let’s look at it step by step.

The variables headers is a dictionary which contains the user-agent, you can also use your own agent by running a simple Google search for ‘my user agent’ and you’ll get a result similar to this, just paste your result and move ahead.

Now, LinkedIn assigns every company page a unique id, you can find it easily. Once you have that replace that number with one in the code.

https://www.linkedin.com/voyager/api/entities/companies/{company id}

Now, we will create a session with the cookies that we extracted from the browser initially.

s.cookies['li_at'] = "your li_at cookie"

s.cookies['JSESSIONID'] = "your JSESSIONID"

Also, we need the csrf-token authentication that we need to do and set it in our headers. For that, we will need to modify the JSESSIONID key, your JSESSIONID will look something like this:

"ajax:7454514581611861831542146"

Will need to strip " from the JSESSIONID and then add it to headers:

s.headers["csrf-token"] = s.cookies["JSESSIONID"].strip('"')

The last part is pretty simple, sending a get request to the voyager API and converting the response to the dictionary.

After this, you’ll see all info about the company.

Footnotes

There are many tools out there (one that I have personally used is PhantomBuster) that do exactly these same things but at scale and what they essentially do is they put a wait time between two requests along with some other tricks.

This script is just for extracting company details, if you try to run it in a loop to get the details of hundreds of thousands of Linkedin Company pages at once, Linkedin will block you and may also permanently ban your account which may be very very difficult to recover.

This is just one workflow that we build with voyager API, there are a whole bunch of cool things that you can do with this, starting from getting the profile of a person, replying to conversations, sending connection requests or responding to them, doing job search with advance filters, liking or commenting on the posts. In short, you can do everything that you do on LinkedIn but with its API and without the UI. Comment below if you want another article about the other things as well.