I’ve run 10,000 product manager vacancies through AI and that’s what happened
By: Nikita Artemchuk, Никита Артемчук
Tags: AI, Artificial Intelligence, Product Management, Management, Self Development

Hi, my name is Nikita, and I am the Head of Product in one big product company in CIS region. If I was given a dollar for every time other product managers asked me “What path to choose in career development?”, then I would have accumulated $50, or even $100.

Since I am a lazy person by nature, at some point I set myself the goal of answering such questions automatically. Somehow. Plus, the quarantine brought me back to the basics of programming for which, as you know, honest product managers have no time because they need to set tasks.

The first version of my service allowed everyone to answer a list of questions on the topic of product management. As a result, the person received a report, which highlighted the strengths and weaknesses of his / her professional skills. The results were presented in the form of a radar chart, which successfully demonstrates distortions and irregularities.

Also, in addition to the report, there was a comparison with the average answers of all those who answered earlier.

The assessment was carried out in 5 sections: Analytics, Design, Strategy, Interaction, and Team/Processes. A couple of sample questions are attached.

After a bunch of people passed the survey, there were several problems, based on feedback:

I’ve come up with questions by myself. I was not sure that the test was checking the right skills.

Comparison data was only available from a limited sample of previous answers. To make them objective, you have to engage numerous product managers to pass this test. And to make them answer honestly somehow.

At first, I tried to run paid advertising, but honestly speaking, the results were modest with a high price for a lead. The competition for the attention of product managers is quite serious :)

I started thinking about where else I could get product managers’ skill data at a completely affordable price.

There were 3 available sources:

An official description of the skills required to take a certain job position. Such data is stored in large open data reports, for example here. The main problem is that the majority of it is about blue-collar jobs, and it is quite difficult to find product managers among the tons of data. Plus, such data is mainly only about the United States.

The second source were the resumes of those product managers who are looking for a job. Good source, but there aren’t many of them, plus they don’t reflect what employers are looking for or what is key for professional development. Here people simply write about how great they are. There’s no guarantee that this is what companies demand.

Finally, the most important source were the product managers vacancies. There is a variety of requirements, moreover, there is a link to the professional level (junior, middle, senior).here is information about the region, the company, and the field of work. Sometimes even there is the salary level. This was exactly what I needed.

The project was divided into 3 logical parts:

Parser. It was necessary to collect data. Parser selected product vacancies from job search sites and added them to a database. I parsed the job description, position name, company name, region, and salary level if specified.

Data processor. For data processing, I used the method of “topic modeling”. Its essence lies in breaking the text into single words, bigrams, and trigrams (stable phrases of 2 or 3 words long). It is important because if the sequence of words changes, but the essence of the bigram/trigram remains the same, it is counted as one mention. An example of a bigram would be “UX design” or “product manager”. An example of a trigram is “make the roadmap”. I will talk about the logic of data processing further since this is the most important part.

Interface. The place where the user enters his data and receives a report in the end. In the report his data is compared to the parsed set from the database by running it through a special handler. Thus, the user gets an average list of the most wanted product manager skills in his region and a conditional comparison of his profile with industry standards.

How did it work out?

The model finds the most frequent and most “cited” words and phrases in the parsed dataset. The calculation logic is next: if we have 3 vacancies for product managers and in the first request for web analytics was mentioned 1 time, in the second — twice, and in the third, it was not mentioned at all, then the “web analytics” skill was encountered 3 times, and was cited in 2 vacancies. By adding the necessary coefficients, you can understand how often this or that skill is required in general and in what percentage of all vacancies it is mentioned.

Further, the model breaks down all the collected skills and divides them into large clusters — “topics” (hence the name of the method “topic modeling”). Examples of such topics/clusters can be like “Finance, sales and pricing” where the model relates the requirements of working with P&L, responsibility for the sales plan, trade activities, etc. Or, say, a “Leadership qualities” cluster which includes such skills as the ability to lead a team, give feedback, etc.

After initial processing and creating a model, such clusters are easily calculated for various sections, for example, country, professional level, sphere, etc.

Now to the point, I will share some of the data that I managed to aggregate (it is the average data on vacancies from all available countries of the world):

A request for at least one personal quality is found in 99% of product vacancies. The most commonly used word is “able or ability” — the ability to do something

Requests for teamwork and process management skills are found in 98% of vacancies. The most common ones are the ability to plan and prioritize.

Surprisingly, a request to work with commercial tasks is found in 90%, here the most common request is understanding P&L reports and working with sales

Marketing skills are requested in 68% of vacancies. The most popular requirement is to be proficient in digital marketing

58% of vacancies want UX and/or Design skills.

In 98% of cases, a product needs strategic management skills for its product — vision, value management, etc.

What was odd — among research skills, requesting user interviews, etc. are much less common than market knowledge and market research.

To summarize, the ability to focus, result delivery and market knowledge are much more in demand than design, UX, and marketing. Knowledge of the commercial side is more important than user research.

The top 10 personal skills that are most frequently required in the global product management job openings are Product Management, Solution Delivery, Delivering Results, Productivity / Performance, Improvement, Building an alignment, Decision making, Creativity, Change management, and Flexibility.

There is a similar analysis in each skill group within a personal report. Average reports are available here — https://compatista.com/reports

What’s next?

I have always perceived this as a pet project and did not seriously think about monetization. Most likely, it will remain in this state, if nothing is fundamentally changed.

There are plans to do the same for other similar vacancies: project managers, business analysts, marketers, and designers. And to finalize the other language versions of the report. So far, the test and report are available only in English.

You can get your report for free here https://compatista.com/