California Data Science Job Market Analysis
By: Navid Mashinchi
Tags: Data Science, Web Scraping, Data Visualization, Exploratory Data Analysis, Data Cleaning

California Data Science Job Market Analysis

Be a data-driven job seeker

Clip by author

Objective:

The goal was to examine the current data science job market in California. Which city has the most job posts in CA and offers more opportunities? As a current Data Science Master’s student, I wanted to find out which jobs a master’s degree holder would be eligible to apply for and what skills are most in-demand. Based on my findings from the data that I have collected, I have conducted an Exploratory Data Analysis (EDA) to investigate California's market.

Methods:

Photo by Franki Chamaki on Unsplash

Data Collection: The first step was to collect the data. I decided to web scrape the biggest job search website, “Indeed.” In the search bar “What” I typed in “Data Scientist” and for “Where” I typed in “CA”. I collected all the job posts information from the last 14 days as of December 27th, 2020. The reason why I looked at the last 14 days is because I wanted to have a look at the most recent job posts. In the above time frame, 340 data science jobs have been posted. The data consisted of 340 rows and 5 features (Title, Company, Location, Summary, Job Description).

Data Cleaning/Feature Engineering: I used this dataset in my Jupyter setup and cleaned the data mainly using Pandas. The goal was to look at entry-level and mid-level data science positions. Therefore I had to extract all the job postings that were meant for Senior positions. I iterated over all the job titles and looked for the following keywords:

‘Senior’

‘SENIOR’

’SR’

’Sr’

’Sr.’

’Director’

’DIRECTOR’

’Manager’

”MANAGER”

‘Data Analyst’

’DATA ANALYST’

”Lead”

’lead’

“LEAD

Any job titles that included the above keywords were extracted. I also added Data Analyst to the list since we are strictly looking for Data Science jobs. When extracting the rows, our dataset went from 340 rows to 197. In other words, 58% of the jobs that I scraped from Indeed fits our target.

To figure out what the top skills are, I had to create indicator columns (with value True/False) for the following tools and skills:

Tools: ‘python’, ‘r’, ‘pytorch’, ‘sql’, ‘pyspark’, ‘aws’,’spark’, ‘sas’, ‘nosql’,’salesforce’,’tableau’, ‘pandas’, ‘scikitlearn’, ‘sklearn’, ‘matlab’, ‘scala’, ‘keras’, ‘tensorflow’, ‘scipy’, ‘numpy’, ‘matplotlib’,’spss’, ‘linux’, ‘azure’, ‘cloud’, ‘mongodb’, ‘mysql’, ‘oracle’,’snowflake’, ‘kafka’, ‘javascript’, ‘jupyter’, ‘perl’, ‘bigquery’, ‘unix’, ‘react’, ‘scikit’, ‘powerbi’, ‘lambda’, ‘ssrs’, ‘django’,’seaborn’, ‘github’, ‘git’, ‘splunk’,’rapidminer’,’jquery’, ‘nodejs’, ‘d3’, ‘plotly’, ‘bokeh’, ‘xgboost’, ‘rstudio’, ‘shiny’, ‘dash’, ‘hadoop’,’angular’, ‘nltk’, ‘flask’, ‘node’, ‘firebase’,’php’, ‘rpython’, ‘unixlinux’, ‘postgressql’, ‘postgresql’, ‘postgres’, ‘ruby’,‘tensor’,’dplyr’,’ggplot2',’esquisse’,’bioconductor’,’shiny’,’lubridate’,’knitr’,’mlr’,’quanteda’,’rcrawler’,’caret’,’rmarkdown’,’plotly’,’stringr’,’swirl’.

Skills: ‘statistics’, ‘chatbot’, ‘cleaning’, ‘blockchain’, ‘causality’, ‘correlation’, ‘bandit’, ‘anomaly’, ‘kpi’, ‘dashboard’, ‘geospatial’, ‘ocr’, ‘econometrics’, ‘pca’, ‘gis’, ‘svm’, ‘svd’, ‘tuning’, ‘hyperparameter’, ‘hypothesis’, ‘salesforcecom’, ‘segmentation’, ‘biostatistics’, ‘unsupervised’, ‘supervised’, ‘exploratory’, ’recommender’, ‘recommendations’, ‘research’, ‘sequencing’, ‘probability’, ‘reinforcement’, ‘graph’, ‘bioinformatics’, ’knn’, ‘outlier’, ‘normalization’, ‘classification’, ‘optimizing’, ‘prediction’, ‘forecasting’, ’clustering’, ‘cluster’, ‘optimization’, ‘visualization’, ‘nlp’, ‘regression’, ‘logistic’, ‘boosting’, ‘recurrent’, ‘convolutional’, ‘bayesian’,’bayes’, ’random forest’, ‘natural language processing’, ‘machine learning’, ‘decision tree’, ‘deep learning’, ‘experimental design’, ‘time series’, ‘nearest neighbors’, ‘neural network’, ‘support vector machine’, ‘computer vision’, ‘machine vision’, ‘dimensionality reduction’, ’text analytics’, ‘power bi’, ‘a/b testing’, ‘ab testing’, ‘chat bot’, ‘data mining’.

Regarding the minimum degree requirement. I created a point system from 1–3.

1 → Bachelor’s Degree

2 → Master’s Degree

3 → PhD

I created a dictionary that would map each point to the different syntax of the degree name.

Image by author

Using that dictionary, I would split each job description and check if any of the dictionary's keys were inside the list that has been split. We would get a list of numbers for each row in the dataset, and I would select the minimum number. For example, consider the following section of a job description:

“ Master’s Degree or Ph.D. required.”

We split the above line → [“Master’s”, “Degree”, “or”, “Ph.D.”, “required.”]

Based on the point system, we would get the following list [2,3].

We grab the minimum number 2.

If none of the above keys are inside the list, we add the value 0. Finally, we would change the values of 0–3 to the following:

0 → “Not Specified”

1 → “Bachelor’s Degree”

2 → “Master’s Degree”

3 → “Doctoral Degree”

The next aspect of feature engineering was to add the population number of each location to our dataset to calculate the number of job posts based on the 100,000 population rate. I used the pandas function pd.read_html, which reads HTML tables into a list of data frame objects. I used the following website to get the population information:

Last but not least, I had to add the coordinates of each location to the dataset. For this, I used geopy.geocoders from the library called Nominatim, which allows you to get OpenStreetMap data based on your target location.

Data Visualization: Concerning the plots, I used seaborn, matplotlib and folium. The first folium plot, which is at the top of the article, displays each data point based on the minimum degree requirement. A popup text has also been added for each data point. Once you click on a data point, the job title, company name, location and minimum education requirement will be displayed. The second folium plot shows which jobs a Master’s student is eligible to apply for.

Results — EDA:

Between December 13th and December 27th, most of the data science job posts came from the following locations: