European business analytics in 2018: lack of specialists and Agile
Tags: Big Data, Business Intelligence, Qlik

Today we are talking about the trends that we have seen in the outgoing year in the business analytics services market. These trends have began to emerge in 2018 and are likely to determine the future digital transformation of most SMB companies.

Historically, the European market of business analytics was formed due to constantly growing demand for corporate data analysis from large Enterprises. Usually, the customers for BI systems and data processing services were IT departments, which were responsible for both IT infrastructure as well as for implementation and development of business applications. However, recent technological trends are able to dramatically change the direction of the market.

For several years now, a process that can be described as data democratization has been rapidly occurring worldwide and in Europe. Today even a small company collects and processes a lot of data concerning sales, transactions, employees and customers. The usage of business analytics helps to optimize processes and to lower costs dramatically. Additionally, data analysis opens the way to identification of growth points and search for new business models, which turns BI into the basis for enterprise digital transformation.

1. Specialists’ shortage constrains the market growth.

In 2018, the increasing use of business analytics by companies was hampered due to the shortage of data professionals available in the market. And there was a lack of all types of specialists, such as developers, engineers, data architects. They typically form the project team to create and support BI applications. The most probable reason for the sharp shortage of staff in 2018 is the steps to form their own expertise, which are being taken by most major companies from 2015.

“There are few good teams, and this constrains market growth,” says Dmitry Kiselev, CEO of Qlever Tech, “educational institutions cannot graduate competent specialists in working with data. The students are required to have experience in some successful projects”.

The data specialists shortfall will remain in 2019. The market is likely to adapt and the cost of hourly rates of qualified developers and architects will begin to grow. Service providers may significantly change their corporate management priorities.

Today their efforts are aimed at maintaining their technical expertise, yet next year they will have to focus more on developing a project management culture. This will help service providers to start reducing the time spent by the most qualified developers and include novice specialists and graduates into the project teams.

Conversely, tools for managing data are becoming simpler and today no special knowledge is required to learn the basics of any of the popular BI-platforms independently. The common opinion that business analytics is only available to large enterprises will begin to change. Consequently, small businesses will take an interest in analyzing their data in 2019.

However, the needs of traditional and the largest customers of business analytics in Russia will shift towards complex services. This segment will face a shortage of specialists.

2. Flexible project management methods allow to build compicated BI-solutions.

The upward trend of 2018 has become the spread of flexible methods (Agile) in BI project management. Using Agile provides customers with the ability to start using the results of individual stages without waiting for the project to finish.

This approach has a number of advantages. Project risks are lower, resources are easier to plan, project progress and investments are more predictable. In addition, if the requirements change, the application of flexible methods allows you to adjust the project flow — change the priority of tasks and involve the necessary specialists.

In the field of working with data, flexible methods are quickly becoming a standard industry approach to creating BI-systems. This is no longer just a method that can accelerate development and save money.

“Previously, a typical task was a report or dashboard,” says Andrey Kharlak, CTO of Qlever Tech, “the customer knew exactly what result he needed. The specialist was in search of an answer to the question how to get the required report from raw data. But today it is the other way around. The company has a lot of data, but nobody knows what can be extracted from it and who of the business users will need a certain indicator. You have to rely on the data — to check its quality and completeness, its sources, the needs of the particular business users and their readiness to apply some or other information. The traditional approach, where the result is already known at the start, does not work here”.

During 2018, Qlever Tech began to provide services based on Agile methodology to all its customers with more than 1000 workplaces. Within a year, the specialists responsible for project management in Qlever Tech were trained in ICAgile (The International Consortium for Agile), and the internal information system was changed according to the principles of Agile methods.

3. The second wave of interest for BI: solving a backlog of problems requires a technical expertise.

The challenges that customers addressed to Qlever Tech during 2018 were technically more complex than before. Some queries were related to dissatisfaction with existing BI systems because customers weren’t pleased with the speed of reporting and response time of BI applications. The number of requests for creating BI systems increased, and their development requires a large project team from various specialists — developers, administrators and specialists in the field of Data Science.

The first problem that customers face is too time-consuming period from the incident to its reporting.

A few years ago, an event was reflected in the report a few hours after it occurred. Such reporting speed was considered acceptable. Nowadays, the requirements are changing. Customers need to download and transform data in a few minutes. The speed of report delivery is particularly important for large scale companies with geographically distributed branches and multi-level management structure.

“Let’s imagine a chain of restaurants in 6 hour zones with thousands of employees,” says Dmitry Savvin, Qlever Tech data architect, “at 9:00 a.m. local time, area managers hold a meeting with their restaurant directors. They demand updated data, not the ones of the day before yesterday. Otherwise, they will not be able to solve anything during the meeting. Such uncertainty costs a lot throughout the chain”.

The second challenge is the rapid growth in the complexity of the corporate data storage and processing structure. Companies have accumulated a lot of data that is created in different ways, stored in separate systems and very different in terms of quality and suitability for analysis. Thus, the amount of data in the companies has increased, and their quality has fallen.

As a result, a simple solution with functionality consisting of loading data from several sources and turning it into tables or diagrams is likely to be meaningless if it has been designed without considering the quality of data in different systems, its completeness and peculiarities of storage.

“The era of superficial BI is quickly passing,” says Dmitry Kiselev, “the starting point of a BI project is not dashboards but data for now. It means that a project have to start with data in terms of finding answers to the question whether there is enough data to achieve business goals, or how to obtain and integrate the necessary data”.

4. The role of IT departments is changing: they lose an initiative.

Until 2018, IT managers usually became the initiators and key participants in those projects that involve data processing.

Previously, analytical project teams were mostly technical. Nevertheless, the number of non-technical specialists involved in the projects increased dramatically in 2018 and began to exceed the amount of technical staff. The role of IT departments began to turn into a service one — they provide the infrastructure for data processing and manage the compliance. Business users are more frequently becoming owners of data.

Product owners will increasingly appear on the service provider’s side. Understanding the customer’s business processes allows the product owner to consider and combine the requests of managers from different cross-functional areas. For example, this will greatly simplify the applications development for different departments which using the same data.

5. BI for Big Data: the use of common BI-platforms to analyze large data sets.

Traditionally, BI-platforms have been used by companies to analyze their internal data, i.e. data that is created and processed in corporate information systems. These are data about sales, customers, logistics, KPIs. However, in 2018, business leaders increasingly felt the need to learn how to benefit from combining their internal with external data. This may include data from open sources, geodata and data collected by the companies in parallel with their core business.