How to build your product startup from a project
By: Nikolay Ganyushkin
Tags: Venture Capital, Startup, Monitoring, Product Development, Product

How to build your product startup from a project

Our story of how the monq AIOps platform emerged from a huge project for monitoring business applications Nikolay Ganyushkin Follow Jun 17 · 11 min read

Copyright Roskosmos

In this article I would like to tell the story of the development of our product — an AIOps platform for IT and business monitoring. And also share some lessons learned during the run for the product. This article will be of interest to both those who are building a product and those who are occupied with IT monitoring in a large organization, since our platform is built for automation, all-in-one end-to-end monitoring, synthetic monitoring and predictive analytics.

How the idea was born

In 2014, there lived a team of techies who worked in a system service provider company and dreamed of making their own product and entering the market with it. No, not even that. The team never dreamed of making its own product, we first saw an opportunity to help our colleagues maintain their information systems more efficiently.

At that time, this service provider company was mainly engaged in supporting highly loaded systems with a huge number of users and a large number of processor cores per system. The support was in the best traditions of the genre 24/7, in a single shift 20–30 people could be working simultaneously. Someone looked at the screens of infrastructure monitoring, someone followed the requests of users, someone continuously tested the functionality by manually filling forms in a browser, someone supported autotests on Selenium, etc.

There were two problems that required to be tackled:

It was very difficult and expensive to check the functioning of the systems at user level — the procedures were done manually and had to be automated. An insane number of alerts from monitoring systems were flying in, and we needed to find a way to reduce the number of alerts and identify only the “important” ones among them.

Solutions were found for both problems.

The first problem was solved by deploying Jenkins that was used to run test builds on Selenium, then the data was parsed and displayed as metrics in Zabbix, on which triggers were built. On fired triggers notifications were sent to the support team.

The second problem was solved by writing a small application, where all notifications from monitoring systems dropped in RabbitMQ, then they were processed according to rigidly set rules, and the results were added to the PostgreSQL database. A small dashboard was built on the Bootstrap template where these events were displayed. The application was named “Service Monitor” (this is the previous name of the monq product). After that, another Zabbix stood that monitored emerging events and sent the preprocessed events to the monitoring team by email.

For the service provider company, the problem was solved relatively quickly and inexpensively. And the team was proud of what they did. At that time few could boast of such automation.

And then one day, at a meeting with one of the major customers, the project manager from the service provider company’s side showed the customer this dashboard. He boasted how they get informed about the crashes now. The customer asked for access to the dashboard and asked to be notified about critical failures that were not resolved in 1 hour.

Then the customer’s manager saw this dashboard and asked to set up an alert for him if a critical failure lasts more than 2 hours.

And then just like in that cartoon about Winnie the Pooh, “They stayed a little longer, and then a little more”. The customer asked to set up a few more rules: a special rule for weekends, a special rule if the incident is massive, a special rule for handling night events. It was very inconvenient to write rules in the code by hand. The number of rules grew, and the time to maintain the system grew.

This is how the service provider company’s internal product actually received the first external customer. It was then that the thought came: what if you make a product out of this, with which you could then enter the market? It was still a very long time before that but the team liked their work and the fact that they were able to do something really useful.

Lesson 1: Seize opportunities

If you have technical developments that help you, show them to your clients who may even have a higher need for them.

Business application monitoring

In 2015, this customer operated over a hundred information systems, worked with twenty contractors, and employed more than 50 internal engineers and 300 contractors. Monitoring costs were very high, but efficiency was low due to understandable difficulties in coordinating all different processes, and, of course, due to the human factor, which made the quality assessment system opaque. At the same time, monitoring was often in the hands of contractors who did not find it profitable to register incidents in order not to spoil their KPIs by underestimating the SLA.

So, the organization was looking for a solution to two of the most severe pains it had:

System users stumbled upon problems in the operation of IT services before specialists from the IT operations department. Contractors sometimes did not register incidents and overestimated SLA.

Since the customer was familiar with the “Service Monitor” system, he chose it as a solution. During the first six months, 87 information systems were connected to monitoring (on average, 3–7 tests, 10 steps per system), more than 7,000 metrics and more than 2,600 triggers were used. Over 50 different escalation rules have been written. All this made it possible to answer the most important question: whether the customer’s services work and which of the contractors is “lying”. It turned out that some incidents went unnoticed not only at night but also during daytime, if users did not write about problems.

There is one common problem often encountered when someone tries to implement synthetic transaction monitoring or directly verify functionality by simulating user actions. The system wants to check the real production environment but the test user is not allowed to make changes. This happened with the central accounting system built on the basis of 1C and Parus packages. The problem of how to test a fat client was quickly resolved, but it was not clear what to do with scenarios where payments should be made but they are not allowed.

Where it comes to financial and payroll accounting, the system can view the data, but no changes could be made. Imagine accidentally changing the salary of an employee or firing him. And the level of wages in general is a very sensitive thing. Such data should be treated with care and security. We had to establish a pre-production environment for each division, on the same infrastructure, to make a mirror of the system with similar but transformed data. In this environment it was possible to make changes and see how the mirrored system would behave, and if everything worked fine (the input data were practically indistinguishable) we knew that it would work on the production side too. In fact, the case was 100% confirmed: failures on both the pre-production and production environments occurred simultaneously.

But what about monitoring logs and monitoring real transactions? It would not be very informative, since some operations were rarely performed. It was necessary to identify the problem before the user noticed it. And monitoring real transactions would only show the problem when a user already detects it.

For example, this is how a report on the performed checks looks like:

As the saying goes, appetite comes with eating. Although the problem with the reliability and control of contractors was solved, the load on the situation centers has increased many times since they received another monitoring system.

At that time the “Service Monitor” was not perceived by customers as a means of combining data from different monitoring systems. It performed this function within the service provider company, where it really helped to reduce the number of “garbage” alerts and increase the efficiency of the duty shift.

When you have 10 engineers and 200 notifications per day within several service provider company projects, this is not the same thing when you have 100 systems, 350 engineers and 10 thousand notifications from monitoring systems per day for a large organization with millions of customers and tens of thousands of employees.

In 2016, the “Service Monitor” was brought to the customer’s situational center and it was clear that the product was still very, very raw. It was impossible to implement it as it was.

Lesson 2: Appreciate criticism.

The most valuable suggestions on product development come from those who refuse it.

The service provider company made a decision that the product should be developed faster, and additional funding was allocated to expand the R&D team. All the requirements from the customer’s situation center were taken into consideration. They were formulated as follows:

Role model of access to various infrastructure objects. Writing escalation rules within the interface, preferably in a visual designer. Alert templates for different user groups., Ability to connect different monitoring systems such as Zabbix or SCOM. Automatic registration of incidents in the Service Desk system, not through the email channel. Viewing information on the incident (protocol) from the system interface.

In 2016 the customer saw the architecture of his monitoring system as follows:

The requirements were reasonable and useful for product development.

In 2017 all the requirements were met to one degree or another. Continuously we were showing what we were getting to the head of the situational center who showed interest in the product development, and it instilled confidence in our team that the final product would be useful.

Lesson 3: Stay in touch and be persistent

We were very lucky that the customer did not choose another more mature product from a well-known western vendor over us. Most likely, if we did not keep reminding the customer about ourselves, that is exactly what would have happened.

This is, for example, how rules in the constructor of rules and actions look like in the end:

All-in-one end-to-end monitoring

The customer had a lot of information systems written by different developers in different languages, and each information system often had its own monitoring that generated a huge stream of events, not all of which were necessary and important (maximum 10%). Usually contractors were responsible for this higher level of monitoring the operation of applications, runtime environments, and databases.

At the lower levels: virtualization, servers, networks, there were monitoring systems that were already on the side of the customer. Data from these systems flowed to the situational center. The situational center also monitored user requests. When a massive incident happened, engineers started looking for a problem at the network / server level, then they called upon contractors who would say that everything was fine on their side and the problem was in the data center. Does this sound familiar?

The shifters had to exchange screenshots from their monitoring systems, trying to find the source of the problem, while time passed, the incident hung unresolved, and clients were left without service. And such situations kept happening for several years and were perceived as normal.

And then our hybrid umbrella analytics system comes in and says that all their problems will be quickly resolved, that one informational dashboard will quickly reconcile everyone, that it will be possible to quickly investigate any incident.

On this topic I have a picture that says that progress, if people need it, will win.

The person in the picture with a red flag (or lamp) warns people that a car is approaching. This is a requirement of the Locomotive Act 1865. It was canceled in 1896, 31 years after the launch of the first car on the road. Imagine what kind of protest the first car makers went through.

Probably, the head of the situational center then also looked ridiculous as this man with a red flag. Two years later, by the way, the situational center was transformed into the operational center where process and project managers worked, not operators. Out of 10 screens only two remain now: the resource-service model with a set of filters for business areas and the health dashboard for the main systems.

Several years ago when the product was installed in the situational center, some contractors covertly sabotaged the use of the Service Monitor. And some of the situational center specialists have chosen the tactic of “at all costs to find flaws in the work of the product,” which, as they thought, might someday replace them. This way the product had its first several hundred users. And it was very valuable as they helped to develop the product although working in such conditions was not easy. We had to fight for each release, stay awake at nights fixing bugs.

Lesson 4: Don’t be afraid of hard work

There will certainly be difficulties. Only hard work and striving for results will help to overcome them.

For example, this is the most hated screen for so-so operational contractors, where the customer now evaluates the level of service by himself. Here is the calculation used to determine the penalties for the contractor if the SLA is not fulfilled:

For a year of operation of the system in the customer’s environment, in addition to two thousand functional robotic tests, several hundred rules, more than a thousand users, the system took control of the resource-service model. This is how the graph of the resource-service model looks from the outside — as for me it is practically a piece of art.

And this how the system health screen, that appeared not so long ago, looks like:

Product development

Each functional user of the “Service Monitor” system wanted something of his own. Unbeknownst to the system, the system has become a “desktop” for a large number of specialists involved in support processes, changed the organization’s business processes and changed people’s habits. What in 2015 seemed like “cosmos”, in 2016 it seemed to be a very difficult task, and in 2019 it was already in operation routinely.

The product has evolved along with the development of customer requirements. In 2019 automation appeared, a large refactoring of the backend and frontend was done, in 2020 log analysis appeared, analytical models using ML. Now we have an understanding of who our competitors are and what we are doing, that we can position ourselves as an AIOps platform. In 2019 we also changed the name, began to be called the MONQ platform, brought the product into a separate spin-off, and received the status of a Skolkovo resident. In 2020 we finally became operationally profitable, and won in the Startrup Village competition (this is the biggest startup event in Russia), but that’s another story belonging to a different stage of the startup.

If you are reading this article now and you have some developments that make life easier for someone, try to tell others about them. Perhaps someone else needs them. Do not waste opportunities, the worst thing that will happen, you just waste some time talking about them. Listen carefully to criticism, it will help you focus your product on customers’ needs . Do not give up hope and do not stop communicating with those who gave you feedback on your idea, especially with those who spoke in a negative way. So, if you have an idea, a prototype, or an understanding of people’s needs, get ready for hard work. The road is made by walking.