Mix-and-match websites using Rails engines.
By: Agnieszka Figiel
Tags: Ruby on Rails, Data Visualization

Data is ingested via an admin panel , which is a typical ActiveAdmin interface. It is divided into sections, each of which allows you to upload files for a particular dataset. The fields on the upload form and the respective validation logic are dataset-specific. Positively validated files land in an S3 bucket, organised in directories named after the datasets they belong to.

Next, data is read from S3 and processed by importers. This happens asynchronously and in background, as some of the files take a long time to process. The importer classes are namespaced by dataset, for example HistoricalEmissions:: or NdcContent:: . They take care of parsing files, validating data and storing it in the database.

Data access and persistence is managed by ActiveRecord models, also namespaced by dataset. The underlying database tables have their names prefixed by dataset name, for example historical_emissions or ndc_content (which works well with namespaces on the models). There are no cross-references between tables across datasets, only within (with the exception of the locations table, which is a globally shared dictionary of countries).

Finally, the web API reads data from the database and serves it to the frontend, formatted in a way to easily populate the various visualisations on the website. The routes to the endpoints are namespaced by dataset here as well.

I used colours to show how we could easily slice through each layer of the application by dataset. It would be easy not only because everything is consistently namespaced; it’s also because it’s conveniently naturally decoupled. For example: tables and ActiveRecord models are cross-linked only within a dataset, and not across datasets (except for the special case of the locations table). Data importers are independent of each other, as they implement specific dataset’s logic. Finally, the web API was designed to align with sections of the website, with separate endpoints for each of them.

Interestingly, the potential for splitting the code in this way was not our objective at the beginning of the project when we introduced this organisation. It was because we envisaged we would end up with lots of models, many of which would have the same name, but different meaning in different datasets (for example “sectors”), so we established namespacing on the database object level and applied it throughout.

Packaging code.

It looked like it would be possible and relatively easy to vertically slice the monolithic Rails application. If only there was a way to package such a slice into some kind of pluggable building block, that could be easily applied to other applications! That, in essence, is exactly what Rails engines do.

Rails engines are effectively gems, the standard for packaging code in ruby. That means they can be used to share code easily across Rails applications, using the same mechanisms that we use for managing other dependencies, such as bundler . But they are also miniature Rails applications in themselves, so they can contain code from all layers of a Rails application which attaches itself almost seamlessly to the host application. A well known example of a Rails engine is the authentication solution Devise. With a few lines of code it adds models, controllers and views to fully handle authentication, with numerous customisation options.

Extracting code into engines from Climate Watch turned out to be conceptually very easy, mostly because code was already namespaced throughout. Therefore, every extracted namespace could very easily become an engine. There were a few obstacles to tackle though.

First, the API controllers—like in many Rails applications—relied on functionality from the base controller, such as setting http headers or providing customised error responses. There was not a lot of that shared functionality, but it was important to keep it consistent across all platforms without having to copy and paste. Ideally, there would exist only one version of the ApiController across all the applications and all the engines, so the first instinct was to create a separate engine with the base controller, which would then be a dependency for all applications and engines. However, that didn’t feel right, because inheritance comes with a lot of risks for future maintainability, so having the base class of an inheritance tree external to the application would open the door to technical debt. Therefore, we decided against a single base class and instead packaged the shared bits of functionality into modules, which need to be included by the base controllers. Those modules ended up in a separate engine, alongside other utilities (such as logging or S3 access).

module Api

module V1

class ApiController < ActionController::API

include ActionController::MimeResponds

include ::ClimateWatchEngine::Cors

include ::ClimateWatchEngine::ExceptionResponses

include ::ClimateWatchEngine::Caching

end

end

end

Next, we needed to deal separately with the only shared database table / ActiveRecord model which represents locations. Conceptually it is also a separate dataset, with its dedicated admin panel section, importer and API endpoints. Therefore, it too was packaged as an engine. The difference from other datasets is that this one is referenced not only from host applications, but also from all the other engines which depend on it. Things worked mostly as expected when running the applications, but all hell broke loose when running the automated tests. We use rspec with FactoryBot for tests. In general, there are a few surprising things to remember about when writing specs for engines, such as explicitly having to point controller specs to the routes. Those are covered in the documentation. The bit we got stuck on was using factories from the Locations engine in other engines’ specs. We ended up loading them explicitly in rails_helper.rb of the host engine:

# Add additional requires below this line. Rails is not loaded until this point!

Dir[Locations::Engine.root.join(‘spec’, ‘factories’, ‘*.rb’)].each { |f| require f }

Finally, we struggled with how to publish the engine gems. We didn’t want to publish them via ruby gems, but instead used bundler’s ability to load them directly from git repositories. The natural way to do it would be keep each engine in its separate git repository, but that was something we wanted to avoid, as we originally thought we would have many such engines, and we didn’t want to end up with an unmanageable number of repositories. We went for using a single repository with multiple .gemspec for all the engines. In the end this turned out to be false economy, as we didn’t end up with that many engines as we envisaged, but we did end up with a lot of problems with versioning gems!

Not quite IKEA, but still a powerful addition to our toolbox.

As we got familiar with using engines and progressed with our work on the platforms, we spotted an opportunity to cut down on code duplication in the admin panel. The ActiveAdmin controllers (which handle file uploads) were extremely repetitive, and we realised we could easily generate them instead of writing them, allowing us to add new ones very easily as new datasets are integrated. The generator was built as a Rails engine, and successfully applied to the global platform and the national platforms.

Even though in the end we didn’t manage to extract as many reusable “slices” of backend code as we had hoped, Rails engines proved very useful for decomposing the original Rails application for reuse. Given it was the first project in which we applied engines, they proved relatively easy to work with, mostly because the documentation is excellent. The fact that in the course of the project we came up with another application for engines to build a configurable ActiveAdmin back office generator proves how flexible the solution is.

The downside of extracting code into engines is having to manage versions of engines and the gems they require as the host applications evolve. For example, upgrading core dependencies such as ruby or Rails needs to be applied not only to the applications, but the engines as well. Dependency management is, however, not a problem specific to engines, but maintaining ruby gems in general, which we didn’t have experience with before.

The end result? Have a look at the country platforms for Indonesia and South Africa!