Kaggle ML and DS Survey Analysis
By: Udit Gupta
Tags: Data Science, JavaScript, Charts, D3js, Dashboard

Introduction

We present this blog post as a part of our final project work in CSE-564 Data Visualization and Visual Analytics @ Stony Brook University — 2020

Machine Learning and Data Science has seen tremendous growth in the past decade. Analysis of very large datasets is now possible due to the emergence of novel methods in the hardware and software domain. These new opportunities have gained wide attention by professionals working in software as well as related fields.

The immense growth in the use of ML and DS techniques to solve tasks that were simply too hard to perform manually has been a source of motivation for developers to build even more sophisticated solutions. The popularity is such that Data Science has become the most in-demand career of the 21st century.

Objective

Given the popularity of the subject, it is a challenge to understand the overall dynamics that are shaping this industry. Everything from development of new tools to segments covered by existing tools, we can find some interesting information regarding the upcoming trends in this industry by collecting survey information from people that are working in this space.

In this project work, we are analyzing the most recent survey performed by Kaggle.com that was available as a data science challenge on the platform. The survey has more than 18000 respondents from across the globe.

In the next sections we discuss our development setup and further details about the datasets that we are using to generate our results. We also provide a summary about the end-to-end pipeline that we plan to build.

Data

Primary Dataset : Kaggle ML and DS Survey 2019

The datasets consists of around 250 different columns entailing different features and their subdivisions. On analyzing the data thoroughly, we selected the 11 most relevant features for visualization.

Gender: This column consists of the gender of each participant that took part in the survey. Age Group: This column takes into account the age of the respondent. The age has been binned into slabs like 18–21, 21–24, 25–29 etc. Country: This column has the countries from where the people responded. Highest Education planned or attained: This degree which the individual has achieved or is or planning to pursue. Size of Company: This column consists of the scale the company has in terms of the number of employees present in the company. Designation/Job Title: This column has the designation of the respondent. Experience in ML: This column has the work experience of the respondent. This column has been binned. Compensation: This column has the annual income or yearly compensation of the respondent in USD. Skill level: This column defines the skill set of respondents from novice to advanced. Visualization Tools Used: The visualization libraries that the developer has used or has some level of familiarity. ML Libraries Used: The ML and DS libraries that the person has used or has some level of familiarity.

The above columns required good amount of data pre-processing such as missing value imputation, grouping fields into higher classes, standardizing values.

Example: We grouped the Education data into 4 fixed categories reflecting simple coursework and high school level education into a single field : Coursework.

Visual Elements and Design

We made certain design choices in the dashboard that reflect the data type and the most suitable visual element for them.

Our design choices in selecting the data fields and their visualization types involved below queries:

Does this data field provide significant information or supplemental insights about the overall theme. If the data field provides significant information, what is its data type and how to best visualize that data. Do we need to provide filtering and interaction for this data field on the dashboard. If the field provides supplemental information, what is the most suitable summary statistic to use.

Once we had a solid strategy about each data field in our data, we decided to come up with a simple layout of the dashboard on the placement of different visualizations.

To bring a mix of visual elements, we observed the country field to have a 2 level hierarchy and could be plotted in a Treemap with continents as larger groups.

The gender, education level, age distribution and job-titles were categorical in nature and were best reflected through doughnut chart, bar charts and a bubble diagram.

Finally, the textual data in the visualization libraries and ml libraries was reflected using word clouds.

Architecture

Data flow

Backend

The python based backend performs all the data manipulation and pre-processing while exposing a single API endpoint for fetching data.

http://backend:port/data

The API endpoint accepts POST requests with a filter list for applying filters to the requested data. The frontend receives only the summarized data for displaying on the views. This enables our frontend to stay lean and perform optimally with reduced network usage.

This also enables us to perform large scale data filtering and aggregation so that backend can handle majority of the work while still keeping the UI fast and smooth in updates.

Example Data generated by Backend:

{

"age_group": {"25-29": 0.22609930516812904,...},

"gender": {"Male": 0.8184815134148197, ...},

"education": {"Masters": 0.43358523101891766,...},

"job_title": {"Student": 0.25135669726631843,...},,

"current_salary": "$0-999",

"size_of_company": "0-49 employees",

"years_in_ml": "1-2 years",

"primary_tool_used": "advance",

"viz_frameworks": {"Matplotlib": 10516,...},

"ml_frameworks": {"Scikit-learn": 9390,...},

"total_responders": 19717,

"country": {"country": {"0": "India",...}

}

The data is normalized so as to represent partition of the unity in the visualization for categorical fields whereas the summary statistics have the mode values.

Frontend

The frontend comprises of multiple JavaScript files that handle the rendering and updates to the UI. The frontend system is designed to perform asynchronous backend calls to API and update the UI elements with the new data. The filter list is generated by the selection on views and is saved in a local object to be preserved across sessions.

Example Filter list sent by Frontend:

{

'age_group': ['25-29', '22-24', '30-34'],

'job_title': ['Research Scientist'],

'education': ['Doctoral', 'Masters'],

'country': ['India', 'France']

}

The selected values are filtered in the data for each data field and the new summarized view is returned to the front-end.

In terms of UI, the dashboard supports a fully responsive design that adapts to the screen size of the device. Below is a snippet of the dashboard on a tablet screen.

Responsive UI

Results and Insights

Exploring the data through our dashboard we found some results reinforcing our hypothesis as well as some interesting patterns in the data.

Dashboard with no filters applied

Some general observations in global data:

India and United States have the largest number of respondents in the survey covering close to 40% of the total.

The gender distribution is highly skewed showing that Data Science is highly dominated by male community.

Most of the people working in the field are pursuing or have secured a masters degree.

The average age-group is between 22–30 across the globe.

In terms of job titles, we see a large number of students which explains that the field is relatively new.

In terms of visualization and ml tools, we see Seaborn is highly popular and Tensorflow dominating the framework space.

Next, we decided to explore the developments in the Research space for DS and ML.

Selected Respondents who are working on Research roles

We immediately see noticeable differences in the education, age-group, country distribution and tools used by these people.

USA is dominating in the research for ML and Data Science space.

Almost 50% of the respondents are either pursing or completed a Doctoral degree.

The average age-group has increased by close to 3 years such that people with 25–40 are working in this space.

The visualization tools like Bokeh and Shiny are D3.js are more prevalent in this industry. Also, the ML libraries like Pytorch, Xgboost and Keras are more popular among researchers.

Lastly, They have about 3–5 years of experience working in this field.

Conclusion

As a part of final project work we explored the Kaggle’s ML and DS Survey from 2019 that involved more than 19k respondents. The D3.js dashboard helped us analyze this data with ease and allowing us to find hidden insights by allowing multiple filters across the visuals.

References