The Walled Gardens within Elixir
By: Marcel Otto
Tags: Elixir

Case 2: SPARQL function extension registry

For the next occurrence of the problem, this wasn’t an option, unfortunately. In this case, the problem occurred in SPARQL.ex. The W3C spec for SPARQL specifies an extension point for graph database vendors to extend SPARQL with vendor-specific functions. The following example query shows the usage of an extension function with the id http://example.org/geo#distance which one or more RDF graph store vendors might support:

In order to support this in SPARQL.ex this extension function could be implemented with the SPARQL.ExtensionFunction behaviour.

So, whenever we encounter SPARQL function call for this id we want to get the respective module implementing the behaviour and call the function.

Again we have the need for a registry and this time no excuse to actually tackle the problem since we definitely don’t want to maintain a list of all possible extension functions. Users might want to implement application-specific extension functions and should not be required to request their extension to be added to the list or fork the project for this. They also shouldn’t require listing their extension functions together with all the ones they might be using from other external libraries in some configuration.

Before discussing the solution that was chosen, let’s look at a couple of characteristics and requirements for this instance of the problem:

Extension functions aren’t structs, but just simple modules of functions. That means the use of protocols isn’t an option. Having the registry available for reflection at runtime would be sufficient since we don’t need to generate code for the registered extension functions at compile-time. Accessing the registry to retrieve a function is a performance-critical operation since it might be called many, many times during the execution of a query (for any possibly matching query solution) and we don’t want the registry access to add that much overhead on top of the actual execution of the function.

Solution 2: BEAM introspection

The solution for our problem in SPARQL.ex is to make use of BEAM introspection by getting all modules and check their metadata if they implement the behaviour in question. Every module in Elixir comes with a module_info function that gives access to various informations about the module. In particular, we can ask for all the module attributes with mod.module_info(:attributes) , which will include the list of all behaviours a module has specified with the @behaviour module attribute under the :behaviour key. So, we can write a function which checks whether a module implements a certain behaviour like this:

Now, we’ll have to get all the modules to apply this function, which is a problem, unfortunately. The closest we can get to that with the builtin Erlang functions is :code.all_loaded/0 . But as its name says, this function returns just the modules that have already been loaded and at what time modules are loaded depends on the mode in which the BEAM runtime system was started.

The modes are as follows: In interactive mode, which is default, only some code is loaded during system startup, basically the modules needed by the runtime system. Other code is dynamically loaded when first referenced. When a call to a function in a certain module is made, and the module is not loaded, the code server searches for and tries to load the module. In embedded mode, modules are not auto loaded. Trying to use a module that has not been loaded results in an error. This mode is recommended when the boot script loads all modules, as it is typically done in OTP releases. (Code can still be loaded later by explicitly ordering the code server to do so).

— https://erlang.org/doc/man/code.html

So, in interactive mode the results of :code.all_loaded/0 is dependent on when the call happens and which modules were accidentally loaded before :code.all_loaded/0 , so it must be considered to return non-determinant results in interactive mode.

An alternative approach that also works in interactive mode goes a level deeper and directly looks into the metadata of the compiled .beam files of the Mix project and filters the modules in a similar way as our implements_behaviour?/2 function by checking the :behaviour attribute. This approach is used for example by Paul Schoenfelder for plugins in exrm:

This approach, however, has its own limitations. First, it relies on Mix which isn’t available in releases normally ( i.e. unless explicitly added to the :extra_dependencies ). It also doesn’t work with behaviour implementations in .exs files, most notably in test files, as these are only compiled in memory and don’t produce .beam files.

But the limitations of this approach actually complement very well with our first described method of using :code.all_loaded/0 , since releases usually run in :embedded mode where all modules are loaded by the boot script upfront. It also seems that modules defined in the tests already seem to be loaded always, at least I didn’t experience any problems yet (if you know why, I would be eager to hear an explanation in the comments).

So, the solution finally taken for the SPARQL function extension modules in SPARQL.ex is to combine both approaches. It is implemented in a separate Hex package behaviour_reflection.

With its Behaviour.Reflection.impls/1 function it seems we now have a candidate that could be used for the __behaviour_impls__/1 pseudo-function in our generic example. But this function just works at runtime, which means our registry can no longer be implemented with module attributes and we’re losing all the benefits this brings with it like their performance or their possibility to be used in guards. We are also losing the possibility to use the registry in macros to generate code for the available adapter implementations. We already noticed that for this instance of the problem a runtime solution would suffice, but that we also need to have an eye on the performance characteristics.

Before digressing a little bit from our actual problem by looking at alternative ways of storing the modules at runtime, a remaining problem with our solution must be mentioned. The careful reader might have noticed already that there’s still an edge case where modules aren’t caught by both approaches. When modules implementing a behaviour are dynamically loaded later at runtime and this happens after both of our detection methods were run, they still aren’t found. This means whatever our runtime registry looks like, when such modules are dynamically loaded at runtime, we have to reinitialize the registry with the results of another Behaviour.Reflection.impls/1 call.

Excursion: Runtime registry

Let’s rethink our initial generic example registry as a runtime store for cases when our __behaviour_impls__/1 function does not work at compile-time. The natural choices for this are of course processes or ETS. Here’s an example implementation with ETS:

But if our registry lookups are a performance-critical operation for our library, as it is the case in our SPARQL extension function example, even ETS might not be acceptable. In this case, a library implementing the mochiglobal objects could be used.

The mochiglobal objects technique utilizes an optimization in Erlang for constant pools for functions that return static data. The method is named after its first occurrence in the mochiweb library. There are several libraries implementing this method, the most popular ones being Discord’s FastGlobal and Basho’s Fling. Here’s an example implementation for our initial generic example with the FastGlobal library: