How much CI/CD does a startup engineering team need?
By: Wei Wei
Tags: Continuous Integration, Continuous Delivery, Engineering Practices, Lean Startup, Startup

At the end of 2020, I’d like to record some experiences the Flux Engineering team has had in the past two years.

CI/CD (Continuous Integration/Continuous Delivery) is a well-known concept in the industry now. It sure feels nice to see committed codes triggering automatic tests and click one button to deploy to production, but it takes good efforts to build and maintain a good CI/CD system.

Is it for me? How much effort is enough?

Flux’s main component is a web app. We started with an API backend (written in Node.js) and a Single Page Application frontend (written in Vue.js). We also have an extensive data backend, but we can talk about its setup another day. I’ll focus on the web app in this post.

CI/CD Tool Choice

There are a lot of CI/CD tools available these days. I considered these criteria when picking one two years ago:

Pipeline definitions can be stored as code along with app code

It can be written as a list of shell commands

Has basic functions built-in, e.g., Slack notifications, running Docker containers

No need to maintain servers

Flux picked Bitbucket Pipeline because it had a bonus point of having strong integration with other Atlassian tools. Several other products are compelling options, too.

Keep Git Workflow Simple

CI/CD workflows are tied to Git workflows. We trigger pipelines on code commits, branch updates, and PR (Pull Request) updates. A key factor in keeping CI/CD workflow simple and easy-to-understand is keeping Git workflow simple first. A “customized” git workflow is usually a cause of confusion. And problems that were supposed to be solved by customizing git workflow can usually be solved in other ways.

Atlassian has good documents describing and comparing common Git workflows. Flux started every repo (code repository) with the dead-simple centralized workflow, meaning “everyone pushes to master!” For experimental, pre-launch projects, and projects that one person worked on, this is usually the best way. When a repo needs more collaboration, we often quickly switch to the Feature Branch workflow to enable code reviews. When a repo gets mature and needs to have hotfixes as a separate process as feature development, we upgrade the repo to the popular Gitflow. And we stick to the vanilla definition of Gitflow so everyone can learn how it works easily from any document, and it works automatically with any tool that supports Gitflow.

Testing

Testing is the first thing we did in CI. We run faster testing on every code push and slower testing (browser testing) on PR updates and main branches (develop and master).

API tests are straightforward. We use mocha + chai +supertest. Most API tests are written as integration tests that use a test database (Postgres) and send API requests then check API responses, sometimes other side effects such as database records. We also write some unit tests for functions that have slightly complex logic. But we do find integration tests gave us more coverage and confidence with not much extra effort.

We use Cypress for e2e (end-to-end) testing. Basically, we need to start 3 services: Postgres, API, Front-end, and Cypress test specs in Chrome. This can be done locally, and on CI pipelines, we use a simple docker-compose file to start these services and run them. We’ve been delighted with Cypress for its easy-to-use API, stability, and good documentation. E2E tests run slower than API tests (~15 mins vs. ~3 mins), so we don’t run e2e tests on every code push but only run them on PR updates and main branches.

End-to-end tests are the heaviest tests we have. It helped capture issues numerous times, but it’s also more tedious to troubleshoot when issues happen. We explored lighter testing options for the frontend, e.g., unit testing or component testing, and actually wrote some Vue component testing using Vue Test Utils. But so far, we don’t feel it’s worth the effort because it requires learning a new set of test APIs which are not so straightforward, and it rarely catches real issues. I’m looking forward to Cypress’s support of Vue component testing.

CI is not just testing; there are several other things we do in CI pipelines:

Lint. Check code styles.

Security/dependency check. Fail pipelines when there are high-risk vulnerabilities.

Test coverage and code quality check. We use Sonarcloud.

Build artifacts (docker images for API, code packages for frontend) for deployment.

The goal of CI in my mind is to “integrate quality code, fast.” If problematic codes are merged into the main branch, we think if we can improve CI to prevent such incidents; if codes are held on branches for a long time because of CI, we also need to think about the CI setup to make good codes available to other developers as soon as possible.

Continuous Delivery

Flux’s CD setup (snippets in Bitbucket pipelines) is so simple that I don’t have a lot to share… It’s usually just a few commands that call AWS CLI to deploy codes.

We deploy API on AWS Elastic Beanstalk. So the deployment step is to call an AWS CLI command to update the app version on EB (Elastic Beanstalk). We recently switched to use Bitbucket’s EB deployment pipe because it does a better job of “waiting” for EB env health status changes.

We deploy the frontend on AWS S3 + CloudFront. The deployment step is to sync compiled code packages to S3, then do a cache invalidation of the index.html and a build stamp file on CloudFront.

One important point in thinking about deployment is to clarify the “artifacts” of each component and store them properly. For example, our API server artifact is an immutable docker image, which is then used for e2e testing and also for deployment. This ensures we deploy the same thing that was tested.

The goal of CD in my mind is to “ship quality code, fast.” One metric to evaluate is the time from a feature is merged into the main branch to the feature is shipped to production; the other metric to watch is how much time and effort a team spends on release, whether it’s an “all-hands-on-deck” situation or a one-person casually clicking a “deploy” button, or even not a button. I think Flux did fine on this front. We still deploy components separately, so someone needs to click not only one “deploy” button but multiple “deploy” buttons. And a light collaboration happens in a Slack channel to check with senior engineers to ensure release dependency (ideally none). But this is an ROI-conscious decision since the current CD pipelines took not much effort to set up and already work well for us.

What Would I Do Differently If I Started Again, Now?

Keep in mind a lot of choices were made in 2017/2018 when Flux started. So if you’re starting from scratch, it’d be good to evaluate options based on the latest.

I’d recommend now doing a monorepo setup to put API and frontend codes into one repo. So e2e tests can be more easily triggered by changes on both sides, and changes about a feature can be reviewed as one PR. Monorepo tooling has advanced a lot.

CI tool landscape has also changed (e.g., GitHub Actions). So although we’re happy with BitBucket Pipelines, and it was, in fact, a reason that we picked BitBucket back then, I’d probably take a deeper look into GitHub if starting today.

I had some experience writing AWS CloudFormation and dealing with AWS. So I wrote CloudFormation templates for the frontend setup. We started using Netlify later for feature preview deployments, so each front-end PR can be previewed at a URL before merging. I think in most small team use cases, Netlify is a better solution for frontend deployment.

Flux is still a growing company and our practice is far from perfect. If you have any suggestions, please feel free to comment here or reach out to me directly. Thanks for reading!