Some casual notes about BPM based applications. Oracle, Camunda and others…
By: Victor Alekseev
Tags: Software Architecture, Oracle, Camunda, Software Development, Bpmn

to attract attention only ;)

This article is dedicated to some fundamental choices that stand before the developers, who have decided to separate the implementation of their business processes from the distributed source code in the individual artifacts.

In my opinion, the most important of them are the following:

“BPM Monolith” vs “engine per service”

“Remote engine” vs “Embedded engine”

“Enterprise-ready solutions from respectable vendors (Oracle BPM, for example)” vs “Open-source solution (Camunda / Activity)” vs “Home-made engine”

I will try to provide arguments pro and contra each point, as well as point out the important features of each opportunity.

“BPM Monolith” vs “engine per service”

For a long time, there was a “BPM as service” approach, which implies the deployment of one server (cluster of servers) that is shared by all other services — “BPM monolith”.

Even though this approach is considered as common and “enterprise-ready”, it also has many considerable drawbacks.

Deployment and administration such a centralized service require a lot of additional effort.

It provokes a waterfall project organization with the inevitable big bang at the end.

Business analysts try to describe all processes as thoroughly as possible, so as not to miss anything and get a beautiful process landscape.

In parallel, the IT department carefully designs and implements a service repository, which is abstractly beautiful in a vacuum and satisfies all SOA principles as they are described in thick books.

Then, in the end, these two elegant abstractions meet with each other and it suddenly turns out that one badly matches the other.

During the development process, it is necessary to somehow share this big resource between individual developers who interfere with each other. Similar problems with launching tests by developers and CI

As a rule, deployment together BPMN based descriptions and supporting code/libraries are not provided at all or only in the form of scripts (for example, groovy based)

On the one hand, such code is very difficult to develop and test in isolation from the server itself. They can use a quite limited set of tools and API.

On the other hand, such a “text-based” approach provides automatic co-ordinate versioning of both the process itself and the code, implementing its task, which can be quite attractive

Usually, the integration of the main application with such service, taking into account security and transactional contexts, is not simple.

Some problems with monitoring — it is the only standard predefined non-extensible set of nonstandard features, provided by the vendor. Integration with them usually requires to support the complex vendor-specific, badly described, and an outdated stack of libraries.

External access to the engine itself and instances of executable processes is only possible through a proprietary API

It can be very nontrivial and these will likely be web services that are not currently being widely used.

Similar problems with the transaction, security, monitoring, impotence, and so on…

On the other hand, these are common problems for the remotely implemented BPM engine.

Some problems with the division of server resources between separate types of processes and separate tasks within one process:

Usually, there is only one scheduler with one pool of threads shared by all activities. So, if there is a slow process or a process with some slow task, everyone suffers

If we have a cluster of servers, it may not be very painful.

Such a monolith belongs to everyone, which means no one is responsible for anything.

Fortunately, now there are lightweight solutions like Activity / Camunda, which allow us to provide a dedicated embedded process engine with its database for each service of the system — “engine per service”.

Advantages:

Better division of the concept

Each service is independent of the others in terms of technology, version of the engine itself / its database / specific settings, the lifecycle of the service itself, deployment, administration, and so on.

Problems with the performance of one service do not affect the others

The data for each service is stored independently. This is more secure, reduces the risk of accidental changes / intentional attacks / mistaken changes.

Only data/processes related to one service are visible during the administration process. There is no need to differentiate data between administrators of individual services in a single interface. The risk of possible errors is reduced.

Disadvantages:

It is necessary to perform the deployment/administration of several engines instead of one. It’s a little easier if we use an embedded option, which itself is automatically deployed. It is of course more difficult when the highlighted “per-service” engine is deployed separately, for example, if the service is not JVM based. Docker containerization can help in this case.

Global business processes are divided between separate engines. It becomes difficult to administer, collect, and aggregate information for monitoring and so on — there is no central access point. It can help to implement in each process data providers for centralized storage, for example, based on ELK stack. However, it must be written by yourself and it will not be as detailed as direct access to the engine’s data archive. On the other hand, sooner or later it will have to be implemented, especially in the case of work with external executors of tasks.

Partially the problems of multiple “per-service” engines can be solved if they all use a single centralized database, but each service runs its instance of the engine. A single UI can be used to administer/monitor all processes, where everything will be visible. This is both a plus and a minus.

Camunda provides some specific tools for this approach.

Deployment aware — each of the “per-service” engines can see only the tasks related to those processes that are deployed by the service and for the support of which the service has the appropriate classes implementing the service task

— each of the “per-service” engines can see only the tasks related to those processes that are deployed by the service and for the support of which the service has the appropriate classes implementing the service task It is guaranteed that the engines of any two “sequential” versions will be able to work with the same database structure. In this way, it is possible to gradually migrate between versions, switching separate services as they are ready.

Multi-tenancy — usually it is applied to separate storing and processing data of individual organizations. But nobody prevents you from using it to separate data/processes of different client services. Data separation can be implemented at “table-level” as well as at “schema level”.

Of course, this hybrid approach has some disadvantages: less isolation between microservices. They become more dependent on the implementation details of the BPM engine and versions of it and a database.

Deployment: Remote vs Embedded

In case “embedded deployment “ we use the engine just as an ordinal library:

is it only possible for JVM based services

it allows us to close and almost seamlessly integrate with it in both directions, perform configuration of the engine during the startup, store the process described as part of the service source code or generate it programmatically, add custom retry policy, perform easy monitoring through such advanced tools as Java Melody type, etc.

it is easy to participate in the transaction/security contexts and pass them to the called services

usually, we can put the engine’s specific tables in the application’s database schema and have benefits of consistency backup of all business data.

It is easy to test, mock the necessary process with the service and a lot of other bonuses

The main drawback: correct bootstrapping requires a lot of effort because it is necessary to properly integrate with the application server providing for the engine as a minimum: Datasource, transaction manager, and thread pool. Especially it is essential for performance tuning.

It makes sense mainly in the traditional mode when BPM based processes call other services of application in synchronous mode — orchestration model.

Close to this variant is unique for Camunda architecture of “container-managed engine”, when a BPM engine is deployed “per server” and shared with several services, each of which provides its process description and backend java beans that implement the tasks, listeners and other business logic

Available on tomcat, wildfly, and other application servers. Correct integration with the application server comes out of the box, but only in the commercial version

The engine is available to web applications through registration in a centralized JMX repository. By the way, even if each war starts its engine, all of them will be visible there and available for inter-service communication.

Recently, joint deployment of several service WAR on one application server is not recommended in favor of Docker containerization, so this approach is more a subject of the past.

In the case of remote deployment:

This is the only option when the service itself is not Java-based. Accordingly, we call the BPM engine through the REST API.

It is a sample of good separation of concepts. The engine lives in a separate independent process, so it’s easier to identify and isolate errors. The process has its separated lifecycle.

Unfortunately, this means an additional artifact, separated from the service, with a description of the process and maybe some additional scripts — doubling of service deployment/administration procedures.

Some common problems with remote calls:

Shared transactions are not available. That is, it is not possible to have the whole transaction including some service, BPM engine, and some services that will be in turn called from the process.

Again problems with passing on the security context, end-to-end monitoring of calls chain and so on

we have to think greatly more about impotence and retry policies of remote calls. Camunda / Zeebe provide special client libraries, but it’s still more complicated than before

Unavoidable complexity of testing. Especially when it comes to the mock of other services, and when the test needs to run the process from the middle of the of one.

However, this is a fairly common option. Especially in case of high-performance systems when it’s not BPM that calls services but on the contrary, BPM is called by services through to External Task API.

Some problems with the fundamental enterprise-ready solutions from respectable vendors

In most cases it is a “BPM as service” product that implies centralized installation as a separate server and brings already described disadvantages, but not only ones.

First of all, it’s a high cost of entry:

skills of an already available developer (e.g. java developer skills) are not very applicable

training is usually expensive and not all employees are eager to learn

already trained developers specializing in these technologies are rare in the market, expensive and can not always be 100% loaded during the whole precinct

the company’s existing development practices (CI, testing, version/release management, code review) should be redesigned and adapted to the specifics of the new solution

high entry cost not only for developers but also for the team that performs maintaining the solution.

The more universal and fundamental the solution, the more difficult it is to integrate with the current application stack of technologies. Such solutions are not interchangeable, so we have a dependence on the current vendor:

and quite not only in the BPMN specific part but also in the rest of the infrastructure that this solution requires — instead of the tool we buy (and depend on) an entire development and administration stack

Such an approach is acceptable for auxiliary business processes of the company but not for the main business process of the company

Great problems if the vendor leaves the market or stops actively developing a solution. Especially critical if you’re not a large customer

Such solutions usually use dedicated database storage, so we meet with problems of consistency backups of all business data. You also regularly have to meet a situation when the crash of such storage or its unsuccessful update leads to significant data inconsistency. The structure of such storage is usually very complex and it is impossible to perform manual operations with it.

Normally (for example, Oracle BPM) such software pack includes some advanced specialized IDE allowing to develop processes “without programmer’s participation”:

Promised “zero-code” is a deception from the very beginning, as business analysts are simply unable to cope with the complexity of development and plenty of technical details. This approach is strange to most programmers, difficult to use, and not transparent. Therefore, an attempt to force them to change the usual common development style to a graphic proprietary one is guaranteed to fail in advance.

Development through 3–5 levels of dialogues and wizards (“death-by-properties-panel”) is complex and slow in itself. With large projects, the opening time of each next dialog is about a few seconds. Often (custom code can only be added with scripts that can only be run and tested inside the engine and the list of available APIs is strictly limited.

Metadata and descriptions are usually saved in a set of XML (and this is still a good thing since they are at least somehow understandable), which cannot be edited without a special IDE. The total size of the source code is more than 100 times. A lot of additional files, the purpose of which is not transparent. A nightmare with code review, merge, reverse engineering, and other similar standard development operations Debugging of processes, implemented in this way, are quite complex and time-consuming.

Supplied testing tools require a dedicated server with complex infrastructure, don’t allow to mock required external services, and are not compatible with existing test libraries and CI tools.

Such IDE is also usually organized on the “all or nothing” paradigm so it forces to develop all code related to the BPM project in this IDE. As a result, we have at least two IDEs for one project.

There is no access to the source code, which helps identify problems faster, especially performance/bandwidth related ones. It is quite difficult or even impossible to customize the engine. Normal debug operation cannot be performed, especially in malfunctioning situations.

And of course, a quite considerable price. Moreover, since the common deployment case is “BPM as service” the price is often increased by the dependence on the number of processors.

In short, this type of solution is well suited for a one-time solution of simple tasks over the existing infrastructure but is categorically not suitable as a basis for the independent development of complex and responsible systems.

Home-made engine

In many cases using full BPMN description for business process modeling is redundant and we can do with a simple state machine.

Of course, some functionality will have to be reduced, for example: nested and multiple states for one process instance, asynchronous timer/message interruption/continuation, business transactions with compensatory actions, reuse (call activity) some common fragments, versioning of descriptions and so on. Nevertheless, it is a simple and effective tool for uncomplicated enough services in situations where high performance, throughput, and concurrency are required.

The author has an extremely positive experience in implementing a similar solution for financial documents flow application for banks processing several million documents a day. The principle of organizing such applications is usually quite simple:

the document contains a reference to a process instance

UI asks for a list of possible operations with a document as a list of outgoing transitions from the current state

all business operations are called on the process instances only ( UI => process instance => business code ). Business code can perform some operations (locally or remotely) and return the next transition name or another standard outcome to the process instance

In the case of an asynchronous call of the external system, business code sends a message and switches the process instance in some intermediate state, inaccessible from UI. After receiving an answer from the external system, the message handler calls the corresponding operation on the process instance, which moves the process into the next state.

There are a lot of open-source projects which can be used as a base for such approach: Spring StateMachine, EasyFSM, Akka FSM, State Machine Compiler, Stateless4j, StatefulJ, JState. All of them provide their API for description of schema but from the author’s point of view, the best combination is groovy based DSL which in turn calls the underlying API of state machine builder. Similar DSL allows combining in one document meta descriptions of states/transitions with executable code and provides the next benefits:

Escaping from XML/CDATA hell

Avoiding necessity to switch often between XML file with process layout and java files with business callbacks

Well organized foldable descriptors, keeping code searching and navigation, refactoring and debugging ability, other IDE sugar

Code can be kept together with descriptions or separately depends on complexity and convenience

All text properties can automatically be extracted for localization

Any fragment of business code can be replaced or modified during customization/modularization

Some common disadvantages

Incompatible with existing business analysis and process design tools.

At first, the costs of such a solution seem insignificant, but there is a risk that as the costs grow when we will have to implement a big part of the advanced functionality on our own.

Monitoring functionality, SLA / KPI calculation and so on will have to be implemented independently)

Generation process description in BPMN notation is complex enough, but it is quite easy to apply Graphviz for visualization in general way:

Event-based approach

It means the use of JMS like infrastructure for asynchronous state switching. Every execution of business operations can send a message in the dedicated queue to continue asynchronous processing by another message handler, responsible for the execution of the business operation.

Implementation of such an approach can be performed based on about 50–100 classes and take about 1–2 man / month.

Advantages:

Lightweight, transparent approach with easily predictable behavior

Full-fledged clustering from the start — scalability + fault protection

Flexible performance control: how many servers we run and how many threads we allocate for each handler/operation

It is easy to monitor the performance by control the length of queues for each business operation

Сompared to database+scheduler based solutions, there is no issue in the form of a continuous query flow even if no new documents are received for processing. Moreover, there are no frequent issues when a lot of documents are involved simultaneously in the processing and the corresponding tables are overcrowded

Tasks of different “complexity” do not compete for the same pool of scheduler flows.

Disadvantages:

Additional infrastructure for deployment, monitoring and so on

Event-based architecture is not always familiar enough to architects and business analysts, as well as to developers

The simplest implementation requires a shared database and JMS transactions.

The main objection to using a JMS when implementing a state machine is that it does not keep the message order. However, in this case, it is completely irrelevant since each transition can be inited only by the event sent by the previous state. So there is only one message addressed to the process instance simultaneously in the queue. The relative order of the messages addressed to different instancies is not so important in this case.

Database-based approach

In this case, the processing can be implemented based on the dedicated table with not-completed states and processing procedures implemented as “select for update skip locked” query.

It is quite simple, flexible, and reliable enough for throughput of about 1000 operations per second provided that the duration of each operation does not exceed 10 milliseconds (long transactions kill database performance).

If the count of active processes at the same time is not large, the database server keeps them all in memory, working almost a real message broker. It is therefore recommended that the minimum amount of data required is transmitted through this table.

Advantages:

we can implement very flexible message prioritization policies based not only on timestamp but also on some business-related properties

data processing is multithreaded on the one hand and strictly transactional on the other hand

automatic retry in case of any exception

we get the strong one-place persistency and well-known programmatic model

Disadvantages:

every transition has to be processed very fast. It means the inability to make remote calls inside a transaction or complex processing logic with intermediate states and lock’s timeouts

consumers who constantly re-requests data even if the message queue is empty.

The schema with one table it’s not very scalable in terms of throughput and the number of participants.