Data Science Market
By: Raphael Bustamante
Tags: Data Science, Data Scientist, Exploratory Data Analysis, Data Science Market, Data

Introduction

I am trying to get into the Data-Science field, by learning and taking courses on the field. But besides learning the tools and skills needed. I often ask myself if I clearly understand the market field that I am willing to get in.

What languages were most popular? What Databases are being used? How diverse is the market? How much time do they work?

Based on those questions I decided to take a look at the Stack Overflow Survey, the data can be found here (https://insights.stackoverflow.com/survey).

And some of the questions that I would like to answer are:

How diverse is the personal profile of the tech workers? What are the Tools/Techniques most being used in the Data Science World? How the Data Science Marker is in Brazil? (since is the Market that I would like to Join in)

How is the Survey Constructed?

Before jumping in answering questions I would like to understand how the survey is built:

Big Numbers

- What questions are made every year?

- How to compare the trends over the year?

Big Numbers

First of all, to understand better the context of the Stack Overflow Data Survey, we are going to understand the big numbers of the Survey, trying to understand how the survey changed over the years it is fundamental to start getting context on how the StackOverflow Survey it is advancing towards the years.

Respondents Over Time

What questions are being asked every year? Surveys of the past year it is similar to the survey made this year? Below is the comparison between the question asked through the years.

Intersection 2015 and 2016: set() Intersection 2016 and 2017: set() Intersection 2017 and 2018: {'IDE', 'FormalEducation', 'VersionControl', 'SelfTaughtTypes', 'TimeAfterBootcamp', 'LastNewJob', 'Salary', 'Respondent', 'Gender', 'CareerSatisfaction', 'Currency', 'Methodology', 'Country', 'JobSatisfaction', 'CheckInCode', 'CompanySize', 'EducationTypes'} Intersection 2018 and 2019: {'CurrencySymbol', 'LanguageWorkedWith', 'Age', 'OpenSource', 'Employment', 'Respondent', 'PlatformWorkedWith', 'Dependents', 'UndergradMajor', 'PlatformDesireNextYear', 'Student', 'Country', 'DatabaseDesireNextYear', 'DevType', 'DatabaseWorkedWith', 'LanguageDesireNextYear', 'Gender'} Intersection 2017, 2018 and 2018: {'Country', 'Respondent', 'Gender'}

Conclusion

After looking at the basic structure of the surveys through the year it is clear that they don’t have a pattern of the survey, which makes understanding trends over the year difficult. And since the years 2018 and 2019 seems to have more common columns, the same questions in the survey we are going to stick with those years.

1. How are the personal characteristics of the respondents?

The first thing about distinctive personal and professional information is to define what is to be personal and professional information. As this may be subjective.

Top 10 Countries with Most Respondents

There are 183 different nationalities in the 2018 survey and 179 in 2019;

The top 10 Countries in 2018 corresponds to 63.17% percent of the total respondents and in 2019 to 62%;

Top 10 Countrys with most respondents in 2018 and 2019

Gender

There are 59458 and 77919 male and 39397 and 10964 not male in the survey of 2018 and 2019 respectively;

There are 1031 Transexual people in the 2019 survey, an equivalent of 0.01% of the Total respondents;

Value Counts of Gender of the Respondents

Conclusion

As we previously suspected and we can corroborate it seems that there is no pattern in elaborating these surveys. Meaning that it is difficult to analyze and comparate it to the previous year.

Despite that non-continuous survey, it is important to notice that the last survey made a huge advance by asking personal questions that can indicate a positive tendency on the technological business. The social inclusion, by asking the respondent the Gender and if they are Transexual. For me, this step towards social inclusion can (hopefully) indicate that the industry is becoming more diverse and possibly starting to break stereotypes.

2. What are the Tools/Techniques most being used in Data Science

To understand what tools and techniques are being used we can check the survey to analyze how to use it. But first, we need to restrict the public to the respondents that are Data Scientists.

To understand the Databases, Language and Platform the Respondents most works we condensed two sets of information: What type they are currently working and which they want to work with next.

1. Databases

Database that Respondentes most work

2. Language

Language that Respondents most work

3. Platform

Platform that Respondents most work

Conclusion

As could check the Databases desired to work are mostly SQL with the associate the vendors:

1. PostgreSQL

2. MySQL

3. SQLite

And also a No-SQL Database MongoDB.

For most respondents, Python is the most important Language, followed by R and SQL.

And Linux is the most important platform to work with followed by Windows there is also an important fact to notice, the AWS platform, which can indicate that Cloud Computing is becoming an important platform to work.

3. How things are in Brazil in 2019?

Big Numbers

There are 2505 and 2263 or 0.07% and 1% respondents from Brazil in the survey of 2018 and 2019 respectively.

Years of Coding

By visual inspecting the variables Age1stCode and YearsCodePro, we can understand when most of the people start coding and how many years of experience they have.

Distribution of Age 1st Code of Data Scientists in Brazil

Distribution of Years of Pro Coding of Data Scientists in Brazil

Distribution of Work Week Hours of Data Scientists in Brazil

The mean hours worked per week is approximately 40 hours/week.

Best Practices

Besides coding and getting work done there are a few techniques and workflows that help to build better applications and codes, how often Data Scientists in Brazil review their and peers’ code, and how is the Test and Validation Routine?

Code Review Practices by Data Scientists in Brazil

Unit Tests Practices by Data Scientists in Brazil

Development Environment

I am also trying to understand the context of how the environment of the programs is set up, to become a better developer. In this context, there is a trending topic at the moment, Containerization of Environment and it’s applications, how people in Brazil often use it?

Containers Usage by Data Scientists in Brazil

Development Environment used by Data Scientist in Brazil

Conclusion

Understanding how many years of experience, hours of work it is important to know the market that I am willing to get a job. This can helps me to understand better the frustration of not getting a job in the field.

Starting to understand what are the best practices, environments that are mostly being used also helps to align with the market techniques and tools most being used.

In this context

Final Conclusions

Before jumping in to starting learning and looking for a job is best to try to understand how the target market and its players are organized so the study can be better oriented, the job search can be better targeted.

This initial study helped me to understand what tools (databases, languages, and platforms are being used. How the market it’s organized in terms of diversity (gender, country) and the characteristics of the market that I a willing to start a position.

This combined with the hands-on process of getting the data and processing it and start doing a real-life Data Science Project.

And how about you? What are your perceptions of the market? What tools and techniques to learn? How is the market that you are in organized?

The Jupyter Notebook that used in this Blog Post can be acessed on:

https://github.com/raphaelbfs/DataScience-BlogPost-001