Microservices, Industry API’s and the Inverse Conway Maneuver
By: Kyle Gene Brown
Tags: Microservices, API, Software Architecture

Kyle Brown, CTO Cloud Architecture, IBM Cloud Garage

Shahir Daya, IBM Distinguished Engineer, IBM Global Business Services

In our jobs we see a lot of proposed reference architectures and even more client architectures. As a result, we’re often asked to give our opinion on one aspect or another of a particular architecture, and recently, that opinion often revolves around the use of what we’ll generally term an “Industry API”.

Industry API’s are not a new idea. They’ve been around for decades, dating all the way back at least to the SWIFT system, which was founded in 1973 to handle interbank financial transactions, followed by the first EDI (Electronic Data Interchange) specification in 1975. The manufacturing industry soon followed, resulting in today’s OPC (Open Platform Communications) standard, built on top of years of lessons learned through implementing other standards and platforms like SCADA. Today these standards include full-fledged service architectures like those from BIAN (Banking Industry Architecture Network) and the OTA (Open Travel Alliance).

First of all, let me state that we are all in favor of Industry-specific, common API’s for interconnectivity. That is the problem we have faced for decades, that businesses need to be able to talk to each other electronically in a common way, and many industries have solved that problem admirably with these API’s. What’s more, over the years, these standards have evolved with the times, such that most of them (such as BIAN) now implement RESTful API endpoint definitions to make it easier for developers to implement them with modern tools. The result is that in those industries with wide adoption of these common industry API’s that integration between partners is more straightforward, cheaper and easier than ever before. An interesting side effect, which is particularly appropriate in these economically uncertain times, is that the technology component of Mergers and Acquisitions is simplified when both companies have adopted the same industry API standard.

But that’s not really the subject of this article. What we instead wanted to discuss briefly is a slightly worrying trend that we’ve seen multiple times in clients that adopt these API’s — and that is a tendency to not only treat them as a tool for improved interconnectivity but to treat them as the specification for their entire domain, from front to back. So what’s wrong with that approach? That will take us into a discussion of architectural decisions, a sidebar on how committees operate, and the problem of how to innovate in a constrained environment. But first, a bit of history…

The problem with specifications

In the early 2000’s one of us (Kyle) spent quite a bit of time helping out with the development and specification of first IBM’s VisualAge and then Rational Application Developer and Rational Software Architect products. We were on the front lines of teams using these tools in large-scale development projects with the early versions of Java Enterprise Edition. At the time, we were excited about one of the coolest features that we had built into the tools — the ability to write an XML specification of the software, which would then be generated into all of the appropriate Object-relational mapping code, external API interfaces, and template code for the intermediate business objects. To top it all off, we added a feature on top of that that allowed the user to draw the desired outcome as a UML diagram, which would then feed into the XML specification, and the result would be your shiny new Java code (all as EJB’s!) coming out the end.

However, despite having a better mousetrap, the world didn’t beat a path to our door. In fact, there was quite a backlash against the whole idea of Enterprise Java (particularly Enterprise Java Beans, or EJB’s), resulting in the widespread adoption of what is now the de-facto standard for Java frameworks, Spring.

So what the reason why things didn’t work out as we thought they would? Part of the fault lies in the overly-complex nature of EJB’s — specifying multiple interfaces, XML configuration files, and having to take both clients and servers through complex code generation steps was a bit much. However, there was another problem that we believe was the nub of the issue. In essence, it was far too difficult to take something built as a complex specification and adapt it to the needs of changing business requirements as those requirements changed.

We had several assumptions about the way software would be written and used that proved to be untrue:

1. We thought that our EJB’s (particularly Entity EJB’s, which for those of you too young to remember, were a form of object-relational mapping into fine-grained remote objects) would be reused as-is throughout enterprises. In fact, that almost never happened — it turned out to be far more common that an EJB (representing a Database table or View) would be used only once in the entire Enterprise than would be used by multiple teams. We had missed the true power of relational databases — that the way you store your data is different from the way you use your data, and that the uses vary widely.

2. We thought that once you understood how a business functioned that you could design your software once, and it would just work for all time — this also turned out to be false since we failed to account for how quickly business conditions changed. For instance, the downturn of 2007–2009 wiped out not only entire businesses but also whole lines of business within existing enterprises. Those that survived often had to drastically revise their business models. The idea of a software model as a static entity rang particularly hollow as the ramifications of the downturn spread. In today’s rapidly changing business climate (being changed by outside forces like a global pandemic) this is especially true again.

3. We thought we could specify an entire business in excruciating detail, and then that specification could be magically transformed into the appropriate source code. What’s more, there was even an undercurrent that this could be done by the business users themselves (lest you think that the current low-code/no-code fad is new — it’s actually the third time in our careers we’ve seen it repeated). In fact, the source code you ended up with was often difficult to follow, hard to maintain, and often full of complex abstractions that had to be put into the code to minimize the number of different potential paths that the code generators could follow in interpreting the specification.

So in short, we missed several of the main lessons that Agile methods (which were being developed at about the same time) have now taught us. You can never specify something in enough detail to run as code without it becoming code. Big Design Up Front (BDUF) usually foundered on the level of detail that was required — you had to pity the poor developer who often was brought in to “fill in and clean up the last few pieces” of a project developed from a BDUF specification. They would often find that the pieces didn’t generally fit together as well as you had anticipated, that the glue code necessary to make them fit often outweighed the generated code, and that by the time you finished the process, that the backlog of new requirements not anticipated in the original specification was twice the size as the original specification.

If we now know that the Big Design Up Front process simply did not work as we thought it would and that it’s better to do incremental architecture and design based on what you know, solving only today’s problem as you encounter it, what does this imply for the question of specifying an entire enterprise model from an industry specification? Well, that comes back to one of the points we made earlier, about how the specifications are made, and what that leaves businesses who need to be able to deal with changes in the business environment.

We can speak from experience about how industry specifications are written. Kyle is listed as a contributing author to one of the late, unlamented WS-* specifications, and has his name listed on patents that were contributed to form the base of two more. Having participated in a number of these specification processes, there’s one thing we can tell you for certain about the outcome — you’re trying to build something that is as general as possible, so that you can facilitate communication across products from as many different (competing) companies as possible. Because the industry standard specification process is so exacting, you end up specifying things in great detail; you have to in order to ensure interoperability not just between products that implement the spec but to also remove ambiguity from how the specification should be interpreted by the final consumers of those products.

But that specification process requires compromise. Lots of times in the specs we’ve been involved with a committee member has brought up a possible extension, or neat feature that they’re thinking about in their product, and the committee as a whole ends up rejecting it from the spec for one reason or another. Often it’s because other committee members find it hard to see how they would implement that feature, or because they want to include a slightly different, competing feature in their product. That’s just the nature of working in a committee of “frenemies”. You end up agreeing on the things that are essentially not differentiating for your business, and disagreeing on the things that you think are your company’s “secret sauce”. The outcome is the part that everyone agrees on — the most general and common aspects of whatever problem you’re solving.

Now, for interoperability and inter-business communication, that’s exactly what you want! You want to make sure that competing businesses can communicate so that the ecosystem as a whole can flourish. We have found that businesses need to expose their data assets via APIs to be a candidate for inclusion in marketplace/ecosystem platforms and aggregator services to reach customers that take advantage of the value-added services that such platforms and services have to offer. If Industry APIs don’t exist, some aggregators resort to screen scraping, which is not ideal. By lowering the barrier of entry for new players, the established players benefit, and the overall community grows as a result. When APIs are based on an industry-standard, it makes it a lot easier for these 3rd party platforms and services to integrate. That’s one of the benefits of adopting an industry API as a part of your architecture: you are more likely to be integrated with value-added service providers as part of a greater ecosystem.

But the problem is that if you view the specification as your entire business model, then you’re essentially saying you’ll only do the general-purpose, generic things that you probably do now. What’s more, if you adopt it as your development model, then you’ll probably align the reporting structure of your software development organization (this is an application of the Inverse Conway Maneuver) to match the structure of the specification. In that case, you’re not leaving open the possibility of change or disruption. In a development organization organized by the standard, there’s no one whose job it is to work outside the standard. What’s more, your business model can only change at the rate that the specification changes, which is going to be at least partially dictated by your competitors!

So instead, we take a slightly different view of Industry specifications. You should definitely let them inform the way in which you think about the business, but you shouldn’t let it dictate that this is the only way you think about your business. They should form the outer model of how you communicate with the outside world, but you should allow a separate, inner model to be built around the unique way in which your business operates. You would build up that inner model using techniques like Domain Driven Design (DDD) to reflect your particular take on the industry. No two companies in the same industry are the same, be they airlines, banks, or manufacturers. They offer different products. And even when the products that they offer that are similar, they offer them in different unique ways; with each company trying to differentiate it’s offering from the others to gain more of the market.

So we will offer up a simple model to capture that uniqueness below:

Industry APIs wrapped around a unique core

By just changing the way you view the industry-standard — as a mechanism for communication and interoperability with the outside world, but not as the definition of your core business, then you leave open the possibility that you can innovate and keep ahead of the competition as new business opportunities arise. In many industries, the core business capabilities are implemented using one or more COTS products. Many modern COTS products have adopted Industry Standards and expose REST APIs that conform to the industry-standard API. When an industry-standard COTS product is a good fit for the core business, the industry-standard can be used for communication within the company as well as with the outside world. That way if you need to adopt new industry standards, or even lead in creating new standards you have the opportunity to do so without revealing too much about how your particular business operates as a result. Now, to a software architect, this idea should look familiar — this is the tried-and-true Adapter pattern approach first described in the book Design Patterns. It’s been behind every integration product that has arisen in the last twenty-five or more years and it’s at the heart of many software architectures.

In the Microservices world, there is a slightly different interpretation of this that we’ve used successfully in several client architectures. If you read Sam Newman’s book Building Microservices, one of the patterns he advocates is Backend for Frontend (BFF). It’s nothing more than an adapter that deals with a particular front-end application whose expectations differ a bit from the overall Microservice API that it adapts. In many of our implementations we’ve had to add them to a design to deal with the exigencies of either a Mobile or Web application that had some unique requirement in terms of format, sequence or size of data that it could accept — an external API is nothing more than the same problem writ larger, and writing API BFF’s is similar to writing them for Mobile or Web Apps.

What’s more, when you start looking at developing a Microservices architecture for your unique core, you will probably find that the mapping between the industry standard vocabulary and the microservices that represent your unique core is not perfectly 1–1. That’s good — that’s an indication that you are doing something innovative if a DDD exercise has shown you think about the world differently than everyone else. Of course, being able to map back to common concepts is important, which is why adapter variations like BFF are so useful for the outer layer.

Summary

Wrapping this up, we can summarize our opinion of adopting Industry Standards as an architectural principle as “Adopt but don’t slavishly embrace”. Using an industry-aligned REST API to provide access to 3rd Parties should definitely be a part of your software architecture — an important part, but not the only part. You need to make sure that you begin from principles like DDD, and benefit from techniques like Event Storming in order to really and truly define what makes your approach unique, and to preserve that uniqueness, before you begin writing the software to adopt the standard. This gives you future-proofing in case the standard changes (as has happened far too often), or in case a new industry standard arises that you need to adopt instead.

Now on, the other hand, you should definitely let the industry-standard inform your approach toward your business model while performing DDD and Event Storming. What’s more, if you work in a very large enterprise, you can even think about the outer model as being a way of reducing the friction between different aspects of your own business. For instance, on a recent project at a Large Bank in North America, what we found was that adopting BIAN as part of that model led to a lot of efficiencies because of the common unified data model and vocabulary across the bank. Everyone from the business and development teams to the analytics and data science teams all used the same terms and benefited as a result.

So think carefully about industry standard models when forming your core, and then adapt them on the outer edge. Use them as a starting point and as a guide, but don’t make them your entire approach.