Explore and Visualize your LinkedIn Network with Python and Sentiment Analysis | by Khuyen Tran | Towards Data Science

If you are a LinkedIn user, have you ever wondered about the segments of people in your network? If you are working to expand your connections in the data science world, do most of the people in your network work in a data science-related field?
How about your messages? Are they mostly positive and about the topics that are related to your interests?
As an active user on LinkedIn with more than 500 connections, I was curious about the statistics of people in my network as well as the messages I received over the last 2 years.
What is good about analyzing LinkedIn data? Well, if you are optimizing your LinkedIn profile for job opportunities, why not use available data as your tool?
In this article, I will combine visualization and natural language processing to analyze my network and messages. After this article, you should be able to analyze your own LinkedIn data and gain insights from it!
It didn’t take me too much effort to get the data because you can directly download it from LinkedIn. Here is how you can do it.
Specifically, I imported the messages and connections data.
I’m going to be embedding the visualizations from each library using Datapane, which is a Python framework and API for publishing and sharing Python reports. If you want to leave a comment for any of my plot or table below, feel free to click “View full report” in the left corner to leave your comment, and I will get back to you!
Here are the notebooks for this article so you can follow along with the article and visualize your own data!
Start with import and check the data
In the table above, I just show 10 of my latest connections as examples.Connected On indicates the date I connect to that person. I will convert that column into a date-time and visualize it with Plotly
There is a peak in my number of new connections per day, especially from January 2020 to July 2020, the period when I was the most active on LinkedIn.
Which organizations do the people in my network work at? I could use a bar graph like this:
But maybe treemap does a better job to visualize companies in this case?
Treemaps display hierarchical data as a set of nested rectangles. Each group is represented by a rectangle, which area is proportional to its value.
With a treemap, it is easier to compare the proportion of one company related to the others! It looks like a large percentage of the people from my network are from Bayer Crop Science. The second-largest percentage is from my university.
Wow! I wouldn’t expect to see so many data scientists. It is great that the top common positions in my network are my target groups for networking.
Some people might have titles start with ‘data scientist’ but also have more words in their titles. Find out all the positions with words start with ‘Data scientist’
>>> network.Position.str.startswith('Data Scientist').sum()268
It is time to analyze messages! What hidden facts can I discover from 3895 messages over the last 2 years?
There are some interesting columns in this data. Since we are interested in the content of messages, let’s dig deeper into it.
Besides English, some of my messages are Vietnamese and some are other languages (that I had no idea what they mean). Since I just want to analyze English messages, I will use LanguageDetector from spacy_langdetect to extract English messages from the rest.
For example, I will check whether the text ‘This is an English test’ is detected as an English text
{'language': 'en', 'score': 0.9999987378584019}This is an english text. {'language': 'en', 'score': 0.999998324978528}
99% that it is an English test. Let’s try again with a Vietnamese text
{'language': 'vi', 'score': 0.9999995883738908}Đây là Tiếng Việt {'language': 'vi', 'score': 0.9999981449655853}
And it is detected as a Vietnamese! Now let’s use this with the entire message
Awesome! Now we have all the messages in English.
Named entity is a real-world object, such as persons, locations, organizations, products, etc.. For example, is Barack Obama a person’s name or a location? Is New York a location or an organization?
There are many ways to detect named entity, but SpaCy if my favorite tool. Let’s test it out with this message:
"This is amazing Tran, it's wonderful you could experience that part of Mexico. Yes, I was born in Guadalajara, and lived in Morelia when I was a kid. I visited Guanajuato just a couple of times. What is even better, I have been living in Japan and Singapore for many years, and about 6 months in Hà Nội. I was amazed about the tons of similarities between Vietnam and Mexico "
[(Tran, 'ORG'), (Mexico, 'GPE'), (Guadalajara, 'GPE'), (Morelia, 'GPE'), (Guanajuato, 'PERSON'), (Japan, 'GPE'), (Singapore, 'GPE'), (many years, 'DATE'), (about 6 months, 'DATE'), (Hà Nội, 'ORG'), (tons, 'QUANTITY'), (Vietnam, 'GPE'), (Mexico, 'GPE')]
ORG is organization, GPE is geopolitical entity, PERSON is the name of the person, DATE is date entity, and QUANTITY is quantity entity.
displacy.serve(message1, style=’ent’)
We can even also visualize named entity in the sentence
Some of these are not classified correctly such as Tran (a person’s name) or Guanajuato (a location), but most of these are correct! Let’s find the named entity of for all the messages
Most of the named entities in my messages are person entity! This seems like we got a lot of person entity in the messages, but we also should be careful about the correctness of the model. Although most of them are classified correctly, some are not.
What are the most common words in my LinkedIn messages?
Start with some basic data processing
After cleaning our text, let’s visualize what we got
These are the large words are the words we expect to see from LinkedIn messages. We also know that my conversation is really into data. Pretty cool!
What is the sentiment of my LinkedIn messages? I would guess most of them are mostly positive but by what percentage? Since I do not have the training data set available, I am using the pre-trained dataset instead. SentimentIntensityAnalyzer from NLTK is a perfect tool for this.
{'neg': 0.0, 'neu': 0.323, 'pos': 0.677, 'compound': 0.6369}
SentimentIntensityAnalyzer analyses what percentage are neutral, negative, and positive. Compound is the final result of a combination of percentages. Thus, we could create the function to analyze sentiment based on compound
Use this function to predict the sentiment of sample text
>>> predict_sentiment('Negative', 62.9)>>> predict_sentiment('It is super useful')('Positive', 77.3)>>> predict_sentiment('It is just okay')('Positive', 38.8)
This seems to work pretty well! If it doesn’t work as well for your text, find other ways to extract sentiment here.
Now we can apply this to all of our messages
As expected, most of the messages are positive. I couldn’t recall a time when my messages are negative. What are the negative messages and what words made them negative?
Let’s look at some negative words as well as visualize them
>>> sample = list(sentiment_df[sentiment_df['Sentiment'] == 'Negative'].Message)[2]>>> sample'lots code gets shipped w technical debt'
It is negative because of word debt!
What are the indicators of positive words? We take a sample of a positive sentence to find out
The word ‘glad’ makes the sentence more positive!
Congratulations! You have just learned how to analyze your own LinkedIn data or perhaps similar text data with natural language processing. We don’t know what is hidden inside the data until we make an effort to discover it. Analyze your own data and you will be surprised by what you see and well as learning something helpful!
Leave me a comment in any of the plots above if you have any questions! You could find all the plots and tables in the article in this report!
Want to generate your own report on your LinkedIn data? The notebooks for this article could be found here. Be sure to add CSV files in the directory of the notebook and you are ready to analyze!
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: