Towards a native higher-order RPC

We present a new abstract machine, called DCESH, which models the execution of higher-order programs running in distributed architectures. DCESH implements a native general remote higherorder function call across node boundaries. It is a modernised version of SECD enriched with specialised communication features required for implementing the RPC mechanism. The key correctness result is that the termination behaviour of the RPC is indistinguishable (bisimilar) to that of a local call. The correctness proofs and the requisite definitions for DCESH and other related abstract machines are formalised using Agda. We also formalise a generic transactional mechanism for transparently handling failure in DCESHs. We use the DCESH as a target architecture for compiling a conventional call-by-value functional language ("FLOSKEL") which can be annotated with node information. Conventional benchmarks show that the single-node performance of FLOSKEL is comparable to that of OCAML, a semantically similar language, and that distribution overheads are not excessive. 1. Native RPC and transparent distribution Remote Procedure Call (RPC) [7] is a widely used mechanism for higher-level inter-process communication. However, the RPC mechanism tends to be bolted on top of a pre-existing language, as a library for example, rather than be seamlessly integrated into it. This leads to significant syntactic differences between calling a local library function and a remote function. Even if these syntactic differences can be smoothed using stubs that wrap remote calls into local calls [6] important differences still persist, of which the most important is that arguments must be of ground types. Generalising RPC to all types and incorporating it seamlessly in the language has been considered but dismissed on grounds of potential inefficiencies [41]. However, considering that a number of technologies that trade efficiency for convenience have been rejected on similar grounds (from machine independent languages to functional programming, garbage collection or automated program verification – to name only a few), we decided it is time to revisit this issue. Because the execution of applications is increasingly and rapidly moving from single devices to networks of (often heterogeneous) devices a case can be made that such revisiting is timely. [Copyright notice will appear here once ’preprint’ option is removed. ] A native RPC system, offering as an immediate benefit transparent and automated distribution, can be useful in domains where programmer effectiveness is more important than machine efficiency. We attack this problem in a principled manner. We want the seamless integration of the RPC in the language to be not merely syntactic but semantic as well: the RPC is a native call of our language, on the same level as a conventional local call. This is realised by introducing a new kind of abstract machine which extends the conventional SECD machine [26], or rather a modernised version of it, with communication primitives. These are not generalpurpose low-level communication primitives but are especially designed to support the implementation of RPCs. Even though we aim for simplicity first, some of the technicalities, especially the proofs of correctness, are quite intricate. For this reason the abstract machine net framework and its correctness properties are fully formalised using the Agda language [34]. The technical challenge is not just one of handling complicated formalisms but also mathematical, the key correctness proof requiring the adaptation of the step-index relations technique [2] to bisimulation Note that we are language rather than system oriented. We assume the existence of a run-time infrastructure to handle systemlevel aspects associated with distribution such as failure detection, load balancing, global reset and initialisation, and so on. From a systems perspective we only deal with failure handling, which is particularly important in a distributed setting, using a general transactional approach. The abstract machine nets can serve as a target for the compilation of a conventional call-by-value language which we call FLOSKEL. The interesting new thing about it is that there is almost nothing new about it. It uses a HASKELL-like syntax but it has the semantics of the pure fragment of OCAML. RPC calls are indistinguishable from native calls except for an annotation, which can be applied to any sub-term, indicating that it is to be executed on a different node. From the point of view of the language, and of the programmer, this annotation has no syntactic, semantic or typing implication. It is a mere pragma-like directive. Prior work on native RPCs and seamless distribution are specialised to web programming [11], thus domain-specific. We aim to be as generic as possible in this context. Some existing work uses as a starting point interaction-based semantic paradigms which lend themselves naturally to a communication-centric implementation: Geometry of Interaction [17] and Game Semantics [18]. Such approaches have two significant disadvantages. The exotic operational behaviour makes it impossible to apply known optimisation techniques, and to interact with code compiled conventionally. This can be seen in the low performance of single-node execution of programs compiled using such techniques. On the other hand, the single-node compilation of FLOSKEL is very similar to the conventional compilation of a language such as OCAML, and the benchmarks indicate that the overhead required by the RPC run-time is not excessive.