Why We Stopped Using So-Called Best Practices in Our CI/CD Process
By: Luc Claustres
Tags: Javascrip, Web Development, Programming, Continuous Delivery, Continuous Integration

When Some Best Practices Break

Package manager

npm is the de facto solution to manage modules with Node.js. It’s simple and works great for independent production-grade modules.

However, it has some drawbacks when it comes to managing packages during development, particularly on a large ecosystem.

First, the process to use unpublished (e.g., under-development) packages is quite different from the process to use released packages. While a simple npm install is sufficient in the second case to get your full dependency tree, you need to clone all repositories of your dependency tree that are currently under development and link them in the first case.

Second, as the link process occurs at the global level of your Node.js installation, you can’t easily have multiple versions of the same module in your environment.

Last, when you have a large dependency tree and release deep-level modules, you have to perform an upgrade operation on all the dependent modules, which can be painful. Indeed, if you’d like to have predictable builds. you must rely on lockfiles. When dealing with a large code base, this means having to make changes across dozens of repositories.

Note: Due to some changes in the way npm manages linked modules, we actually prefer to use Yarn as a package manager, but this doesn’t restrict the scope of this analysis.

App stores

App stores are really designed for native apps and not web-based apps, where you have a physical separation between the front end (e.g., client) and the back end (e.g., server).

As a consequence, the natural process of releasing an application is to promote the same app bundle (e.g., APK or IPA) from one publication state (e.g., private alpha release) to the next publication state (e.g., public beta release). This means you can’t change the default configuration of your package depending on the publication state.

As your client app should be able to at least run without being connected to your server (because the user doesn’t point a browser to the right server URL, in this case), it’s, for instance, natural to make the target server URL part of its configuration.

But if you have different endpoints for your different versions, then you have to rely on a fixed third-party server to relay requests (not simple) or a built-in GUI allowing users to change the target server URL (error-prone and not really user-friendly). More challenging is you can have only one version of an app installed at any point of time, since all of these share the same application ID. If some of your users are using the alpha version, they can no longer use the production one.

Sensitive data

As an application relies on third-party services, its configuration must include sensitive data like API keys, SSH credentials, etc. to access the required services. Of course, this shouldn’t be pushed under source control if your repositories are public.

First, you can use encrypted environment variables set either in the build file or the repository settings. When you have a lot of environment variables, it starts becoming tricky and not flexible — as changing a value requires additional operations to encrypt data.

Moreover, this doesn’t work well when you have different configurations of the same application, like for alpha, beta, and production versions. You have to duplicate each variable for each available configuration using a naming convention (e.g., VARIABLE_ALPHA , VARIABLE_BETA , VARIABLE_PROD ).

Last but not least, this doesn’t work if some of your secrets aren’t environment variables but rather files. In this case, you can create a secrets.tar containing all secured files and encrypt it to secrets.tar.enc using Travis CLI. This file will be decrypted before the build or whenever you need something inside.

However, the same restrictions apply: You have to duplicate each file for each available configuration using a naming or path convention, and changing any file requires additional operations to encrypt data. Moreover, you still need to find a secured data store for the raw secret files.

Front end and back end separation

This conceptual split has evolved into specialized developer roles for each, which is still the norm throughout the industry.

Among many, one of the most fallacious arguments is probably that changes in the back end won’t affect the front end — and vice versa. This is only right if you keep the API backward-compatible. The fact is we develop features, and features are neither back-end or front-end. Most of the time, features are both.

And in this case, effort increases with the division of the front end and back end, particularly if you have a lot of modules. In order to commit any changes, two synchronous commits are made rather than one. In order to release a new feature, the same process is applied twice. When dealing with a large code base, this means having to make changes and actions across dozens of repositories.

Team scalability is also an argument for the separation, but when working with a small team (like we do), it’s better to reduce miscommunications, facilitating smooth application development.

Last but not least, some parts of our modules are written using Isomorphic JavaScript, which run both on the client and the server, so the separation doesn’t make any sense on these parts.

SemVer

A strict use of SemVer actually requires a public API, which is great for libraries or modules to be integrated in a third-party application but not really for the application itself.

In this case, users expect minor releases to be backward-compatible, with data produced by previous minor versions but with a couple of new features or enhancements in the workflow of existing features. Major releases might break this backward compatibility by removing existing features and defining a new data format.