A better way to API
By: Igor Lobanov
Tags: Software Architecture, Distributed Ledgers, Integration Architecture, Enterprise Architecture, API

I believe we have a problem with APIs.

Not in principle, nothing wrong with the idea of application programming interfaces. They are helpful to make a front-end talking to the back-end, and get systems interacting in a useful way. Nor do I have any problems with public APIs intended for the end-user. I can check where my train is on the go because I have an app that can invoke an API published by Transport for London, and Open Banking makes my life admin slightly easier. Admittedly, API technologies of today are also better than they used to be.

My grudge is towards the use of APIs in business transactions, and especially relatively recent obsession with the use of Web technologies for that. When I talk about business transactions, I mean an automated and technology-enabled equivalent of completing an order form, raising a service request, signing a cheque, proposing contract amendment —anything that has a real world effect, contractual implications, and, more often than not, a monetary cost. Systems integration in an indispensable part of an inter-organizational process automation, and, it always happens via an API of some sort, regardless of whether the underlying technology is REST over HTTP or an FTP server. If anything can be certain in our increasingly interconnected world, is that it is only going to get more interconnected and require more integrations to be designed and built.

In this post I am touching upon a spectrum of problems that have been making implementation of APIs for business transaction problematic and very expensive. I will also touch upon most common practices used to alleviate the pain. Finally, I am sharing hope that some recent technology advances — specifically, distributed ledgers — could make some of the issues finally go away.

First, what do I know? Throughout my 20+ years career in software engineering I have been building systems integrations of all kinds, and seen fads come and go. I authored a fair number of CSV file formats, XML schemas, IDLs, WSDLs, Swaggers, ProtoBuffs, AVROs, and GraphQLs. I also wrote and debugged plenty of code talking to AT modems, FTP servers, SQL databases, IBM MQ Series middleware, JMS libraries, and HTTP endpoints. I have battle scars and nightmares. At some point, because of an error in API implementation that escaped testing, I managed to wipe out passwords for millions of online customer accounts of a major retail brand. At a different time, because I incorrectly implemented my own spec, I made tens of thousands of product license keys incompatible with a license server. These are fascinating stories which I will leave for another time, but it feels I have seen enough to warrant strongly held views.

Speaking in pure functional terms, executing a transaction over an API isn’t a rocket science. You send a request, get a response, and that’s it. However, in a real business-to-business interaction, because of the real world effects, an inordinate amount of effort — perhaps 95-99% — is applied to make sure that in all eventualities the things stay under control for all parties involved. And that is what makes business-to-business integrations expensive.

Broadly speaking, what makes business transactions over an API a hard work is the need to maximise the chances of all involved parties agreeing about the facts. Often because of intermittent technology issues — but also occasionally due to a spec misinterpretation, an implementation error, a third-party blunder, a natural disaster, or a malicious action— there could be a disagreement. In bilateral integrations it takes one of three forms:

You are certain you made a transaction with your counterparty, but they deny knowing anything about it. You counterparty insists they made a transaction with you, but you don’t know anything about it. Neither you nor your counterparty are aware of a transaction, but the transaction actually happened and there is a real world change affecting others (these most commonly occur when you try to ‘smartly’ recover from one of the first two).

Worth noting that if a business process involves more than two parties — and most do — the number of scenarios when things go wrong, as well as the probability of an occurrence, raise exponentially. Another point worth making is that it’s irrelevant whether a durable record of transaction exists somewhere in logs when one of above failures happen. The record would aid a recovery for sure, but does not do anything with the fact that there is a problem that needs resolution.

Speaking of which, the gold standard of an error resolution — people from both sides getting involved — is prohibitively expensive for most integration scenarios which were deemed important enough to justify investment in an automation in the first instance. If you fail in 0.1% of cases, an inter-organizational human intervention is likely to take at least an hour to run its course, which means that a million transactions will likely to take a thousand hours to investigate and recover from errors. That is if you leave aside the real world effects, and their legal, reputational, and monetary implications.

Over years, a lot of engineering ingenuity went into thinking about the failure modes and designing solutions around them. The overarching goal was to decrease the percentage of failures that require an intervention and thus reduce the ongoing cost of an integration, so, in practice, probabilities of a failure necessitating an investigation are lower than 0.1%. Covering emerged best practices in detail is out of scope for this post, but I simply mention a few most effective ones:

Request idempotency. Many technology failures are transient in nature and tend to resolve themselves over time. An ability to send the same request repeatedly without a risk of double-counting helps to automatically recover in a good number of failure scenarios. Compensating transactions. When a failure is of a well-known type and has limited consequences, it might be easier to simply send a reverse instruction to restore status quo after the failure. (cancel an earlier order, refund a payment, reverse a change). Needless to say, compensating transactions are transactions still, and all the same failure modes apply. Transaction reconciliation. Exchanging records of transactions that both parties believe they undertook with each other using different means with the intention to find any unexpected ones.

None of these methods is a silver bullet, and, besides, they are solution patterns, rather than reusable building blocks. For example, you cannot download a library that takes care of reconciliations for your API. Applying patterns correctly requires qualified engineering effort (read: cost and risk of errors) to assess suitability and adapt to unique integration task at hand.

When APIs for business transactions use Web technologies — mainly that’s using DNS protocol and X.500 certificates as an identity layer, HTTP over the public Internet for transport, and JSON as a payload format — it adds another layer of misery. Designed mainly for human-driven interaction with an aid of a web browser, the technology stack lacks built-in support for asynchronous delivery, receipt acknowledgements, durable records, rich payload schemas, and simply have too many moving parts that could be misconfigured and go wrong or exploited by a bad actor. Have you ever clicked ‘Accept risk and continue’ when seeing an expired SSL certificate warning? How many teams actually audit the list of trusted CA roots on their outbound integrations?

In my slightly controversial view, as far as business transactions are concerned, AMQP almost always gives a more elegant solution compared to HTTP. In fact, any well thought-through RESTful API for business transactions ends up with a lot of asynchronous communication features like correlation identifiers and polling for long running operations, which are unnatural for HTTP and, in the absence of a message-oriented protocol, are cumbersome to implement. Pushing it even further, in my view, two of the most hugely successful protocol stacks for high-value business transactions — SWIFT (banking) and FIX (securities trading) — are practically anti-Web because of the use of private networking and, in case of FIX, binary encodings. Neither is without problems, don’t get me wrong on this.

And I have not even begun talking about the pains of changing a published API once it has live consumers who you have a contractual relationship with. It is not uncommon to give counterparties notice years in advance about a change that potentially breaks compatibility. In a world where product teams are compelled to make daily releases into production, the need to maintain all those layers of backward compatibility does not come cheap.

All these deficiencies could be accounted and compensated for, and in some cases the benefits outweigh the costs, but the fundamental reality of a business-to-business APIs remains: they are very expensive to build and use, because they need a lot of engineering effort to be efficiently designed, safely implemented, and reliably maintained. As a consequence, perhaps only the top 1% of all real world inter-organizational business processes that could be automated, actually get automated. This must be one of factors contributing to the productivity paradox of recent decades, whereby despite all the advanced technology, economic output per hour worked has not been increasing — presumably because a lot of people are stuck in menial operational roles that cannot be profitably automated with the present-day technologies, or, in fact, were automated badly and carrying operational overhead.

We need a bigger boat.

This is why I am excited about the potential of distributed ledger technologies in many areas. Perhaps it’s not surprising given my background that I have spent time reflecting on what it means for automating business processes. My firm conclusion is that that eventually distributed ledgers will become a dominant paradigm for implementing APIs for business transactions, and that it will allow considerably more inter-organizational business processes to be automated in a cost-effective way.

When making such a bold claim I have to hastily clarify that I do not mean any of the current public permissionless blockchains like Bitcoin or Ethereum. Even though they technically are distributed ledgers, they aim at solving very different set of problems. They may have a profound impact on how we go about our everyday business over a long term, but in the narrow sense of automating business transactions between organizations they are not a good fit. No, I am specifically talking about permissioned platforms such as Hyberledger Fabric, R3 Corda, ConsenSys Quorum, and Tendermint Core which allow a group of organisations to form a private network with controlled admittance, strong privacy, and a well-defined legal and governance framework. I don’t think we are necessarily going to be building on these specific platforms in future — they will come and go — but the features make such a compelling case that it’s hard not to get curious about them.

In it’s most basic definition, a distributed (or shared) ledger is an append-only immutable record of state that is replicated, shared, and synchronised across several nodes in a peer-to-peer network without a central authority. In other words, there cannot be a record committed to a distributed ledger that is reflected differently by different nodes. This ‘you know I known you see what I see’ principle goes long way to eliminate a broad range of failures that integration engineers used to grapple with in any business transaction.

What makes distributed ledgers such a compelling option for designing APIs of the future? In broad terms, there are two major features deserving careful consideration.

Firstly, any distributed ledger platform comes with a protocol stack that covers the basics of inter-organizational communication, but also ventures much further into what traditionally is in the application domain: transaction atomicity and validation, business rules enforcement, record-keeping, etc. For an application developer any additional feature that the underlying protocol provides means one less thing to worry about, and distributed ledgers come with a lot of plumbing. Moreover, not only the stack is ‘thick’, it is also highly cohesive, which makes it more effective (end-to-end principle), evolvable (less need for coordination across layers), and secure (fewer independently operating layers reducing the number of attack vectors). Designed for Byzantine fault-tolerance, distributed ledgers hit the sweet spot by preventing the same very issues that plague most business transaction API designs and implementations: the potential to disagree about facts.

Further, unlike best practices of API design I referred to above, which require a skilful interpretation in each individual case, distributed ledgers are actual software implementing the features in a reusable manner. To be specific:

Idempotency is a fundamental feature of any distributed ledger protocol, (although in an upside-down way better known as the ‘double-spend problem’). If a transaction is accepted by a distributed ledger, it cannot possibly be double-counted.

Compensating transactions are not required to overcome communication faults, because there are either identical copies of a record across all nodes of the network, or it is not in any of the nodes.

Transaction reconciliation is built into the ledger itself, where every party has a local record of every historical transaction they were involved in, which is guaranteed to be identical to any other party’s record, and, in fact, carries their counterparties’ digital signatures proving their authenticity. Parties may still have consistency issues between the ledger node and other internal systems of record, but it’s a lesser problem by at least an order of magnitude.

The other principally important feature is that distributed ledgers are inherently multi-party. This is arguably their most important characteristic as far as business integration is concerned. Unlike most APIs in use today, especially those based on Web technologies, there could be any number of entities involved in a transaction via a distributed ledger, and they are all guaranteed to see the facts identically. Different ledger architectures have different privacy models (e.g. network-, channel-, or transaction-level privacy), but as long as it is correctly chosen for a given business process, it will take care of the data synchronization and information protection needs out of the box.

I said above that in an inter-organizational business process both the probability of a failure and the number of different failure scenarios increases exponentially with number of parties involved. The truth is that we live in a world where the majority of practically useful (and valuable) business transactions, when considered in their entirety from start to finish, involve at least four players: two transacting entities (to establish what the transaction is about) and two banks (for the money need to change hands). Whilst most APIs are bilateral, there are, in fact, four different bilateral relationships that need to be addressed. To put simply, they go somewhere along the following lines:

a) Actual transactional interaction between party A and party B, which gives raise to a financial obligation from A to B, carried over a specific business-to-business API,

b) A instructing its bank to make a transfer to the account of B in their bank according to the invoice, probably via a SWIFT message,

c) B receiving an assurance that A’s money are safely in B’s account, again probably via a SWIFT message, and finally

d) both banks settle mutual obligations using some sort of interbank facility, such as RTGS or BACS.

A lot of things could go wrong here, and the fact that these interactions follow different lifecycles does not help to reduce complexity. As a result, small armies of operational personnel are employed by all participants, which adds a lot of drag and stifles innovation by making changes hard and risky. And this is just a simple B2B trade. Business ecosystem of securities markets, especially when conducted internationally, involves at least a dozen different types of organizations, but it’s not too far from other industries with complex supply chains and information flows.

Distributed ledger in theory allows all parties to be privy to the same set of facts related to a transaction, which makes it possible to contemplate drastic simplification of the integration landscapes, reducing the cost and risk for everyone involved. Because of complexities inherent in it, it is no wonder that financial services industry seems to be trying so hard to bring in the innovation driven by this technology, but pretty much anyone stands to gain from it eventually.

Building a business-to-business API on top of a distributed ledger protocol — once you unlearn the old habits of the engineering mind — is a liberating experience. Granted, the technology is still new. There are still many ways to do something wrong, and not many textbooks around. It is also not a silver bullet, it’s just a technology. Moreover, many people, when you mention distributed ledger, give you a blank stare and mumble something about Bitcoin. Nevertheless, I am convinced this is all going to get easier going forward, and many engineers and their business stakeholders will see the benefits of the new technology. Let’s embrace distributed ledgers as a legitimate way to build APIs for business transactions, give it a fair appraisal, and do the right thing for our clients.