10x Faster CI with Nx and Github Actions
By: Yannick Galatol
Tags: Continuous Integration, Engineering, Github Actions, Monorepo, Nx

By reading this, I assume you already know about CI, Github and the JavaScript ecosystem.

Nx is a set of tools to help working with monorepos and build scalable enterprise grade applications.

Since 2018, Nrwl — the company empowering Nx — released major improvements of Nx to keep continuous integration fast despite of the increasing number of applications or libraries of a monorepo.

From simple tasks execution through to affected tasks, parallelized runs, distributed CI and computation cache features, and the native capabilities of Github Actions, the CI time is able to go down from 45 minutes to less than 4 minutes.

Standard use case

A common CI flow often consists in three steps: lint , test and build .

We could add format , e2e or any other task but that would not make much difference in the way you write it.

A Github Action workflow using Nx would look like below:

name: Nx Standard Use Case

on: push

jobs:

ci:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v2

- run: yarn install

- run: yarn nx run-many --target=test --all

- run: yarn nx run-many --target=lint --all

- run: yarn nx run-many --target=build --all --prod

These commands will test, lint and build all the available projects present in the workspace.json file (or angular.json if working with Angular).

On a large repository, this may take several minutes, up to hours… In our example repository, a whole CI run lasts around 45 minutes.

Affected tasks

One of the most interesting feature provided by Nx is the ability to re-test, re-lint and re-build only the libraries and applications which were impacted by code changes (read the official doc for more information).

In the dependency graph below, changing a file in the admin-feature-permissions library will affect the admin and the admin-e2e applications, and obviously the admin-feature-permissions library itself.

That’s how the Github Actions workflow looks like after implementing the Nx affected command. Tasks will be run only for projects which need to (i.e. affected by changes occurred since origin/master ).

name: Nx Distributed Tasks

on: push

jobs:

ci:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v2

- run: git fetch --no-tags --prune --depth=1 origin master

- run: yarn install

- run: yarn nx affected --target=test --base=origin/master

- run: yarn nx affected --target=lint --base=origin/master

- run: yarn nx affected --target=build --base=origin/master --prod

This will drastically improve our CI for the most of the runs, but it won’t change the worst case scenario when every project is rebuilt.

Parallelized runs

Tasks of a same target can be run in parallel instead of sequentially, maximizing the usage of the processor and reducing a bit the CI time.

name: Nx Distributed Tasks

on: push

jobs:

ci:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v2

- run: git fetch --no-tags --prune --depth=1 origin master

- run: yarn install

- run: yarn nx affected --target=test --base=origin/master --parallel

- run: yarn nx affected --target=lint --base=origin/master --parallel

- run: yarn nx affected --target=build --base=origin/master --prod --parallel

In practice, on large codebases, the CPU consumption often rises to 100% and the machine becomes quickly bottleneck.

Concurrent targets

Splitting the job into three — in order to get one job per target — will parallelize the tasks.

name: Nx Distributed Tasks

on: push

jobs: test:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v2

- run: git fetch --no-tags --prune --depth=1 origin master

- run: yarn install

- run: yarn nx affected --target=test --base=origin/master --parallel lint:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v2

- run: git fetch --no-tags --prune --depth=1 origin master

- run: yarn install

- run: yarn nx affected --target=lint --base=origin/master --parallel build:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v2

- run: git fetch --no-tags --prune --depth=1 origin master

- run: yarn install

- run: yarn nx affected --target=build --base=origin/master --prod --parallel

As jobs are run onto different virtual machines, the CI time is reduced to the longest target which is a good improvement. However, each of the three jobs could impact hundreds of projects, leading to the slowest target reaching dozens of minutes.

Distributed CI

It’s now that the power of Github Actions acts 💪.

Github Actions provides a way to create a bunch of jobs regarding a matrix of values which jobs will be given to as a variable.

Often used to run jobs across different platforms or software versions, this feature is particularly useful to run our tasks in parallel on different machines.

name: Nx Distributed Tasks

on: push

jobs: test:

runs-on: ubuntu-latest

strategy:

matrix:

jobIndex: [1, 2, 3, 4]

env:

jobCount: 4

steps:

- uses: actions/checkout@v2

- run: git fetch --no-tags --prune --depth=1 origin master

- run: yarn install

- run: node ./tools/scripts/run-many.js test ${{ matrix.jobIndex }} ${{ env.jobCount }} $GITHUB_REF …

Where the run-many.js script, inspired from the one written by Victor Savkin in Blazing Fast Distributed CI with Nx Workspaces, looks like:

const execSync = require('child_process').execSync;



const target = process.argv[2];

const jobIndex = Number(process.argv[3]);

const jobCount = Number(process.argv[4]);

const isMaster = process.argv[5] === 'refs/heads/master';

const baseSha = isMaster ? 'origin/master~1' : 'origin/master';



const affected = execSync(

`npx nx print-affected --base=${baseSha} --target=${target}`

).toString('utf-8');

const array = JSON.parse(affected)

.tasks.map((t) => t.target.project)

.slice()

.sort();

const sliceSize = Math.max(Math.floor(array.length / jobCount), 1);

const projects =

jobIndex < jobCount

? array.slice(sliceSize * (jobIndex - 1), sliceSize * jobIndex)

: array.slice(sliceSize * (jobIndex - 1));



if (projects.length > 0) {

execSync(

`npx nx run-many --target=${target} --projects=${projects} --parallel`,

{

stdio: [0, 1, 2],

}

);

}

Let’s see what’s going on. We pass four arguments to our script:

target : one of our workspace targets: test , lint or build .

: one of our workspace targets: , or . jobIndex and jobCount : the index is a variable coming from the matrix substitution, while the count is hardcoded as a job environment variable. Array of indexes and the count must always be synchronized.

and : the index is a variable coming from the matrix substitution, while the count is hardcoded as a job environment variable. Array of indexes and the count must always be synchronized. $GITHUB_REF environment variable: provided by Github Actions, it refers to the branch or tag that triggered the workflow (in the format refs/heads/branch-name ).

Depending on the target and the Github reference, the print-affected command provided by Nx CLI returns the list of projects affected by the target.

And then, we extract a chunk of the computed projects thanks to the job index and the total number of jobs.

This way, each job index generated from the matrix will launch a group of projects to run for the given target.

Feel free to adjust the number of jobs per target to create but, once again, keep sure the array of indexes and the count are always in sync. Otherwise, you risk to run the targets multiple times for a same project, or in the worst case, to simply skip a bunch of projects 😱.

Unfortunately, Github does not provide a native way to get the size of an array to avoid this pitfall, at least for the moment.

Why not a matrix of targets?

In the last snippet, I only described the job for the test target, but in reality, each part should be copied/pasted for both the lint and the build jobs.

Even if the workflow would remain quite simple — this remains an example workflow — it can become really longer in real-life cases while dealing with private NPM registries, more specific git refs comparison, and so on.

To avoid duplicating the whole workflow definition, we can go a step further and apply the same logic of matrix projection for the workspace targets as we did for jobs indexes.

name: Nx Distributed Tasks

on: push

jobs:

distributed-task:

runs-on: ubuntu-latest

strategy:

matrix:

target: ['test', 'lint', 'build']

jobIndex: [1, 2, 3, 4]

env:

jobCount: 4

steps:

- uses: actions/checkout@v2

- run: git fetch --no-tags --prune --depth=1 origin master

- run: yarn install

- run: node ./tools/scripts/run-many.js ${{ matrix.target }} ${{ matrix.jobIndex }} ${{ env.jobCount }} $GITHUB_REF

Much better, right?

We drastically improved our CI time. But we can do a bit more...

Caching dependencies

Like most of the continuous integration platforms, Github Actions suggests caching and it works very well for dependencies caching like our node_modules folder.

name: Nx Distributed Tasks

on: push

jobs: install-deps:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v2

- name: Cache node modules

id: cache

uses: actions/cache@v2

with:

path: node_modules

key: cache-node-modules-${{ hashFiles('yarn.lock') }}

- uses: actions/setup-node@v1

if: steps.cache.outputs.cache-hit != 'true'

with:

node-version: 12.x

- name: yarn install

if: steps.cache.outputs.cache-hit != 'true'

run: yarn install --pure-lockfile distributed-task:

runs-on: ubuntu-latest

needs: install-deps

strategy:

matrix:

target: ['test', 'lint', 'build']

jobIndex: [1, 2, 3, 4]

env:

jobCount: 4

steps:

- uses: actions/checkout@v2

- run: git fetch --no-tags --prune --depth=1 origin master

- name: Cache node modules

uses: actions/cache@v2

with:

path: node_modules

key: cache-node-modules-${{ hashFiles('yarn.lock') }}

- run: node ./tools/scripts/run-many.js ${{ matrix.target }} ${{ matrix.jobIndex }} ${{ env.jobCount }} $GITHUB_REF

We first need to factorize the dependencies installation in a dedicated job.

Github Actions uses the key to determine the uniqueness of a cache. If the key matches an existing cache, then it will be downloaded for the job but the cache will not be updated nor overwritten. If it does not exist yet, then it will be created at the end of the job.

Using hashFiles('yarn.lock') (or hashFiles('package-lock.json') ) in the cache key allows to create a new cache every time a dependency is updated.

Moreover, the cache action outputs a cache-hit variable to inform if an existing cache strictly matched the given key. In our case, it’s really helpful as we don’t need to execute the installation again if the dependencies are found.

Finally, we just need to ensure every job operates after the install-deps job using the needs property.

Get more information about caching on the Github doc.

Nx cache

Nx computation cache and Nx Cloud definitely changed the game. The motto is: don’t re-test or re-build the same thing twice.

In more concrete terms, Nx offers a way to cache the output of any target using source files, environment (os, node version, etc.) and arguments as cache key.

Nx cache can be used locally or remotely — uploaded and shared in the cloud — with Nx Cloud for example.

Using Nx Cloud would work natively in Github Actions while it’s more complicated to get relevant results with locale cache and the previously discussed Github cache system.

Caching dependencies perfectly fits the Github caching strategy: pulling only if a key matches, pushing only if no key matched before.

However, Nx cache aims to store all tasks results in a single cache directory, regardless of the target, the branch being checked, the workflow run… And that becomes problematic as we can’t both retrieve possibly existing cache and append missing ones for currently running tasks.

A common use cases that Github Action can cover is:

a rebuild of a fully or partially checked pull request

the master branch build after merging a checked pull request

This can be achieved with the following cache key matchers:

- name: Cache Nx

uses: actions/cache@v2

with:

path: node_modules/.cache/nx

key: cache-nx-${{ hashFiles('yarn.lock') }}-${{ matrix.target }}-${{ matrix.jobIndex }}-${{ github.sha }}

restore-keys: |

cache-nx-${{ hashFiles('yarn.lock') }}-${{ matrix.target }}-${{ matrix.jobIndex }}-

cache-nx-${{ hashFiles('yarn.lock') }}-

This time, we also used the restore-keys property of the Cache action. It allows to download a cache even if the key didn’t strictly matched. The patterns are resolved in the order and the first matching key will restore its cache.

As this is not a strict match, a cache will still be created for the key and the cache-hit will be false.

Let’s see some use cases…

During a pull request rebuild, the key should strictly match — as the git sha would remain the same — and the Nx cache would be retrieved.

While on a master build, the first of the restore-keys should match as the sha would probably be different this time because of the merge commit (depending on your Github merge strategy of course).

The last restore-keys matcher is a default case which can help retrieving cache for certain tasks. Not really useful here but it helps understanding the possibilities offered by Github about caching retrieval patterns.

You got it, the best way to get benefits of a fully working Nx cache with Github Actions is to use remote caching like the Nx Cloud platform.

Final result

And this is how the workflow finally looks like:

name: Nx Distributed Tasks



on: push



jobs:



install-deps:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v2

- name: Cache node modules

id: cache

uses: actions/cache@v2

with:

path: node_modules

key: cache-node-modules-${{ hashFiles('yarn.lock') }}

- name: yarn install

if: steps.cache.outputs.cache-hit != 'true'

run: yarn install --pure-lockfile





distributed-task:

runs-on: ubuntu-latest

needs: install-deps

strategy:

matrix:

target: ['test', 'lint', 'build']

jobIndex: [1, 2, 3, 4]

env:

jobCount: 4

steps:

- uses: actions/checkout@v2

- name: Cache node modules

uses: actions/cache@v2

with:

path: node_modules

key: cache-node-modules-${{ hashFiles('yarn.lock') }}

- name: Cache Nx

uses: actions/cache@v2

with:

path: node_modules/.cache/nx

key: cache-nx-${{ hashFiles('yarn.lock') }}-${{ matrix.target }}-${{ matrix.jobIndex }}-${{ github.sha }}

restore-keys: |

cache-nx-${{ hashFiles('yarn.lock') }}-${{ matrix.target }}-${{ matrix.jobIndex }}-

cache-nx-${{ hashFiles('yarn.lock') }}-

- run: git fetch --no-tags --prune --depth=1 origin master

- name: nx run-many:${{ matrix.target }}

run: node ./tools/scripts/run-many.js ${{ matrix.target }} ${{ matrix.jobIndex }} ${{ env.jobCount }} $GITHUB_REF

What next?

It’s possible to make the CI workflow even more optimized in many ways. For example:

Adapting the number of chunks to create in order to minimize the final build time and/or the Github Actions remaining time for your organization.

Calculating the affected tasks in an earlier stage to avoid such a computation in each job (even if its cost is not really significant).

Creating different number of chunks depending on the target. Because we often test more than we build (only apps are usually built, while both apps and libs are tested), it might be interesting to create more chunks for test jobs and less for build ones.

Don’t forget the power of Nx: distributed tasks and caching would also work for any other custom targets in workspace.json 🎉.

Feel free to pick ones from the web (eg: docker-build , deploy-netlify ) or even create your own 👷.

Conclusion

Thanks to affected tasks, distributed CI and computation cache features provided by Nx on one hand, and jobs parallelization, matrix strategy and dependencies caching features from Github Actions on the other hand, we could divide our CI time by 10 🍻.

You can get the whole working example on the repository: https://github.com/yannickglt/nx-github-build.

If you’re looking for a version which uses pull request references to calculate the affected projects, you can take a look at the modified workflow and run-many script.

Sources