Data Scientist Seeking Data Scientist
By: Chris Kim
Tags: Machine Learning, Data Science, Jobs

I need to find a data scientist. More specifically, I need to find all data scientist roles open in the US. I promise it’s not because I want to talk to all of them. I actually need the job listings for data-related roles for my next project, a web-scraping project. More importantly, I needed job listings that had salary information. I heard that data scientists make more money than data analysts, but I wanted to prove it.

Webscraping

I actually think I studied html in middle school or high school, but that was more than a decade ago. All I remember were the tags. Fortunately, that’s really all I needed for the project using the Beautiful Soup library. If you don’t know or haven’t used Beautiful Soup, it is a way to scrape information from a website made with html. It’s simple in its simplicity; only an understanding of the html hierarchy is needed.

Find all the job titles on the first page of Indeed.com

From top-down, the code block above looks through every div tag, checks if an id attribute and data-jk attribute exist. Then it will grab the text from the a tag. This returned the job title, even those that were sponsored. Beautiful Soup is interesting in that you usually know what you are looking for, but need to find the path there.

I need more jobs

So I thought a while loop until I scraped 1000+ salaries would work. Unfortunately, Indeed is smart with their resources. The website caps searches to 1000 listings before stating that the remaining listings are repetitive and won’t be shown. To workaround this, I opened up my searches: look for four roles — ‘data scientist’, ‘data analyst’, ‘business intelligence’, & ‘research scientist’ — and twenty six cities. After dropping duplicate entries and job listings without salaries, I ended up with 2724 listings. A caveat, I had to scrape over multiple days. I didn’t want to run my scraper (which would take 30+ minutes) the day before the project was due.

The median salary for data-related roles is right over $77,000. The majority of the salaries are within the $60k — $100k range. With a bunch of salaries on the higher end. At least the upward growth of data-related job salaries is promising.

Call up the models

Here is a gist of all the libraries I imported to use for my modeling.

So you don’t have to count, I used 20 functions

That’s a lot. And I tried all of them (including a couple more). This was the first project where I was able to try almost every classification model that I’ve learned in the past 10 weeks. In order to use the classification models, I had to convert the words to some numerical input. The three options I used were count, hash, and tf-idf vectorizers.

After vectorizing, I called up all the classification models that I’ve used in sklearn: Logistic Regression, KNN, BernoulliNB, etc. I thought that the most complicated vectorizers and classifiers would give me the best accuracy score. But in the end, the simplest of both produced an accuracy score of 84.5% for my test data.

Count the words and use Logistic Regression. Which just goes to show that sometimes the simplest and easiest tools to use are sometimes the best. Maybe that’s why they’re the most commonly used.