Continuous Integration at Jahia
By: Francois Gerthoffert
Tags: Platform, Docker, E2e Testing, Continuous Integration

At Jahia, our teams build and maintain a fairly large number of modules, each of these modules has its own unique functionalities which sometimes vastly differ from one another. Each module is implemented separately and has its own development and release lifecycle, but needs to co-exist within the Jahia environment, which also has its own release lifecycle.

One of the big challenges when working with such a modular environment is to make sure the module being worked on, not only meets its quality requirements but also doesn’t negatively impact other modules nor is negatively impacted by newer versions of its own dependencies, in particular when the time comes to release a new version of our Jahia distribution.

We will talk onions — Photo by Jene Yeo on Unsplash

In this article, we’ll go over a possible approach for tackling such challenges, pros & cons, and how engineering practices both at a high level as well as down to the nitty-gritty details can help define a continuous integration strategy.

This article will focus on continuous integration from a testing standpoint, but more articles should follow on Jahia’s tech blog on topics such as releasing, dependency management, and more.

Jahia release cycle

Before diving into continuous integration, it is important to understand the nature of Jahia’s architecture and its release cycle.

Jahia can be seen as a distribution (if you’re familiar with GNU/Linux you understand the concept), it’s a Java application composed of multiple modules, each with its own release cycle, bundled together at regular intervals into a Jahia release, which contains Jahia core and the latest stable release of a set of modules (not all Jahia modules are included in a Jahia release).

Jahia’s release cycle

Jahia users are free to install a more recent version of some modules, remove modules, or install new ones. Jahia provides dependency management (module B requires module C with version 1.2.3 or above), but there is still a fairly wide set of possible deployment scenarios (not all users will update to the latest version of Jahia or its modules).

This creates some interesting challenges for us on a wide variety of topics, such as validating during the active development phase of a module that its upcoming release is still compatible with older versions of Jahia or older versions of its dependencies and updating required versions accordingly. But also validating that a module that didn’t receive much activity recently is still fully operational when it gets bundled in the next Jahia distribution.

Continuous integration boundaries

As we begin covering the continuous integration strategy, it’s important to define clear boundaries around what we’re trying to achieve.

We want to provide development teams with tooling to help them be confident that the new (or modified) code being written does meet the technical and functional requirements of the features being developed. For our strategy, this will take the form of unit tests and integration tests, automatically executed at a Pull Request level. At that point, we want a failure to clearly point to new code being pushed.

We want to provide teams with (automated) tooling to help them prepare the release of a module by running integrations (or E2E) tests suite against a snapshot version of that module alongside various target dependencies (jahia versions, other modules). At that point, we want the failure to highlight an incompatibility issue between the snapshot module version and other elements of the stack. Incompatibilities that might not have been caught during development.

We want to provide teams with (automated) tooling to validate release artifacts in staging. Tested against a narrow set of dependencies, this aims at validating that the artifacts about to be released are actually operating properly (for example, proper artifact signature, jar file actually runnable, …). At that point, we’re not looking for incompatibilities issues but for a glitch in our packaging/releasing process.

We want to provide teams with (automated) tooling to validate a Jahia distribution as a whole. This includes validating that a released (and/or snapshot) Jahia distribution is properly working with newer installed modules and ensuring all modules operate properly on the same platform. At that point, we’re looking for issues while running tests towards a “real-life” cluster environment, to catch things such as performance or unknown incompatibilities issues. This doesn’t aim at testing “known” and narrow-focused scenarios or validating “known” dependencies, which should have been tested earlier in the flow.

Another important step while designing tooling supporting our strategy is not to lose sight of the characteristics of our modular architecture. For us, one of these characteristics is the large number (100+) of independent codebases (or repositories) each corresponding to its own Jahia module.

We want to design a strategy addressing the above points while giving us as much flexibility as possible and a good level of consistency across our codebases. But a future modification of our strategy MUST NOT require us to push changes to 100+ codebases.

So how do we juggle with all these constraints?

One could build one (or many) large testing codebases fully separated from the actual module being tested, but this will bring its own set of challenges. Similarly to a product codebase needing to reflect the nature of what is being implemented, a testing codebase must reflect the nature of what is being tested. If the primary purpose of a codebase (or part of a codebase) is to test a single module, then this codebase shouldn’t be co-located with code serving another purpose. Similarly, if the primary purpose of a codebase is to test a set of modules and handle this testing as a whole (i.e. not executed in smaller pieces), then it probably shouldn’t be co-located with the codebase of a single module.

You can find additional reading on codebases co-location here and there.

Layer up, Neo

The key to addressing this challenge is to break down your continuous integration strategy into multiple layers with light, but strong, interfaces (or contracts) between layers and a careful balance between using off-the-shelf products and building your own tooling.

You could see it as the application of the “Onion Architecture” but focused on just a portion of the domain.

Our continuous integration layers

“Using contracts allows each layer to set its expectations onto the next and couples it to only what it requires to be. In addition, the implementation specifics of each layer can be refactored at any point as long as they meet their contractual obligations with their neighbours making it easier to respond to changes and increasing the testability of our code.” — Barry McAuley

When it comes to off-the-shelf vs building, the right balance usually comes down to using existing tooling for everything within a “layer” and building your own for interfacing between layers. And since we want the interfaces (or contracts) to be as light as possible, this often results in low development effort and can actually serve us in the long run if designed properly.

Although building your own tooling (for everything) is always tempting (who doesn’t like a nice engineering challenge?), you definitely do not want to create your own testing framework (think Cypress, Jest, Selenium), create your own execution platform (think CircleCI, Jenkins, Bamboo), or anything of that nature… Except if you have a massive software engineering team, don’t go that route, that’s some major red flag.

Red pill? blue pill?

At the core of everything, will be the development effort to build an actual set of shippable features. This corresponds to what the entire team is doing on a day-to-day basis, leading to the creation of Pull-Requests.

Although aiming for consistent practices is always preferred within engineering teams, it is critical not to lose sight of the primary purpose of a testing codebase, which is to serve the validation of an actual set of features. And if the team determines that the best approach for doing that is to use framework Y, there is probably a very legitimate reason for that. The team shouldn’t be dictated to use a specific technology if there is something more applicable to meet the codebase’s purpose, as long as it meets the contract.

Photo by insung yoon on Unsplash

The above statement doesn’t mean it’s the wild west, teams don’t operate in isolation within an overall development organization, there is a consistent communication stream with other teams via multiple channels (architects, tech leads, guilds, demos, …)., with commonly agreed guidelines and practices. And in a well setup organization, it’s pretty unlikely that a team would come up with something so exotic that it’s entirely inapplicable to other teams while creating a large amount of technical debt.

Worst case, if that does happen once, as long as the team abides by the common contract, the exposure to the overall organization to more technical debt should be limited to that single module.

Docking the Nebuchadnezzar

At that point, docker comes to the rescue and can be one of the foundational elements towards reproducibility.

This can simply come down to a simple statement: Use any framework or techs, as long as your testing codebase is Dockerized.

Actually, we should say containerized more than Dockerized, today the containerization technology of choice is Docker, but maybe something else will come in the future. What you want here is the ability to execute your test with as few dependencies and as much reproducibility as possible without technical considerations for what’s inside the container. You want to know that the containers include the test suite for module A, but shouldn’t care much about how those tests are implemented (this is not this layers’ concern).

Photo by Kelly Sikkema on Unsplash

Performance aside, two machines executing these containers must receive the same outcome, without considerations to the local version of node, java, or others installed on the machine.

For us, the interface between the first and second layer is facilitated by a tool we built to handle the provisioning (installation of modules and configuration) of Jahia environments. Since it’s none of a testing framework’s business (first layer) to prepare the environment, the first layer should have a very narrow focus towards testing the actual features being implemented.

Jack-in Morpheus

The third layer detailed in the diagram deals with the automated execution of the dockerized environment, based on various triggers. It will execute a set of actions when new code is being pushed, at a schedule (nightly, weekly), or reacting on triggers (adding a label to a PR), using a diverse set of dependencies and/or configurations.

This is the layer that actually addresses most of the boundaries listed above and can be composed of one or more execution platforms, based on what makes sense for the DevOps in the team.

For example, we can test staging artifacts for the latest Augmented Search release on Jahia 8.x using this payload (using CircleCI’s Pipeline API):

And the staging for the latest Augmented Search release on Jahia 7.x using that payload:

Launching the tests for different dependencies is as simple as modifying the API payload, without needing changes in the code or configuration.

WyldKyss — Brick Fair 2012

The actual execution platforms used for running the tests (CircleCI, Bamboo, GitHub actions, others…) do not matter since we have defined a standard interface (or contract) when dealing execution of the runs and the handling of the artifacts.

The interface dealing with run artifacts is handled by jahia-reporter, a custom tool we built that ingests testing reports (Junit XML, Mocha JSON) and takes care of communicating the results, in a standardized way across teams (Slack, Testrail, …).

A standard error message during the development of Augmented Search

And if a team wants to use a new testing framework with a different reporting format, we would simply have to extend jahia-report to ingest this new format.

Flexibility vs vendor lock-in vs consistency

Although we’re almost done with this article (and with referring to The Matrix), finding the right balance between flexibility, vendor lock-in, consistency of practices is challenging, and different solutions might work for different people.

A heterogeneous stack

Looking at the above diagram, one might wonder, why not use multiple source code repositories instead of just one, why not multiple container registry, why not one single execution platform?

A first guideline would be to try to avoid segmentation on widely “visible” elements. For example, an entire software engineering department is interested in the product itself and its state. The above diagram advocates for one single source control platform and one single test tracking platform. As long as the tests are executed, the way they’re executed (the testing framework) and by whom (the execution platform) does not matter to the department.

The choice to introduce segmentation in specific parts of the stack can be taken individually (container registries, execution platforms). New additions to the stack could be evaluated in the context of a PoC, which could sometimes result in a progressive technology consolidation, sometimes in more segmentation in the stack. In that context, segmentation is not necessarily an issue as long as you have clear and common contracts between the various elements of the stack.

Another guideline is to maintain a clear separation of concerns between the various components of your stack and the actions being performed within each layer, relying as much as possible on common interfaces across various platforms or vendors.

For example, we could have set-up our testing frameworks to directly submit test results to Testrail, instead of using our own tool jahia-reporter. But we want a diverse set of frameworks, using various languages, to report all tests in a consistent manner. Maintaining reporting within the test framework would have created some overhead in ensuring all the frameworks remain consistent in their implementation. If we were to change vendors, we would have to rewrite, for all the modules, the test reporting feature.

Instead, by using a jahia-reporter, we’re able to prevent such situations from happening. The contract (or interface) for submitting tests is as follows: “Produce a test report summary, in a format parsable by jahia-reporter”. No need to be concerned about where to connect, credentials to be used, … The test framework can focus on its primary purpose: executing tests.

Conclusion

As we’re wrapping up this article, I will insist on the fact that continuous integration is a system, and as such, should be designed as a whole, using the same engineering principles as any of the other elements being designed and built by the teams.

It may be dealing with multiple platforms/vendors, environments, and stacks, but in the end, it remains a system. And as with any codebase, this system needs to continuously evolve over time.

Over the past few years, you might have observed codebases progressively re-gaining a central position in the software engineering stack. Concretely, this has been taking the form of CI/CD configuration, orchestration instructions, documentation, project management tooling, and more… co-located with (or next to) the codebases. In that context, the external platforms primarily become stateless elements executing tasks or presenting views based on instructions coming from the codebase itself.

Ultimately, by considering the integration of a new building block as part of a greater system and by making sure its configuration (or its code) is co-located with the codebase the team works on every day, we end up in a situation in which all aspects of the software development lifecycle are handled in a similar fashion. Whether changing a feature, modifying a test, updating a parameter in an automated execution setup, or updating documentation, the process will be similar, helping teams gain and maintain control over their delivery.