How Travis CI killed the productivity of a 100+ team in 2 hours
By: Dick Tang
Tags: DevOps, SaaS, Continuous Integration, Travis Ci

The story starts with several strange notification emails, reminding us that there are credit balance issue.

We felt weird since we are using the plan with 20 concurrent job. We have been a happy Travis CI users for years and never saw email like this.

Of course we are also aware of some upcoming change on Travis CI. An email (with subject “Travis CI — Roadmap update and new pricing” , sent on 1st Oct) mentioned

For our Enterprise customers like you, we have been working on a new pricing model that will better connect your usage with the price you pay each month. Starting in November 2020 we will be moving from concurrent job pricing to a metered based billing that will allow you to buy credits instead. These credits will be used to run your builds which will be metered on a per-minute basis, meaning you will only be charged for what you use. Unused credits will carry over month to month allowing much more flexibility in how you choose to build compared to the current system. We will have more details to share with you on exactly how this will work in the coming weeks.

In the past month, we kept monitoring to our email (for email announcement, if any) and also Travis CI pricing page. None of them has shown the sign immediate threat to our organization account.

Hence we were quite confident that our plan is still working (which is a false assumption) when we saw that credit issue email and believed that it may be some bugs in the recent release of Travis CI panel (we are developers, so we know there are bugs). We submitted a support ticket and thought Travis CI support team were busy in handling enquiries similar to ours.

After a while, the Travis CI dashboard showed we were lack of credit (since we ran so many CI jobs and spent all of the bundled build credit in 2 hours) and refused to run further build job.

Builds have been temporarily disabled for private repositories due to a negative credit balance.

So Our CI pipeline stopped, and for the first time, due to a non-technical issue.

Stop the world

Since our engineering team relies on automation and the CI pipeline so much, they reported the error immediately after we were aware of the issue and started asking about what happened. They cannot release backend code, build mobile app and even simple configuration change (if it is backed by Travis CI).

Luckily, there are only roughly 2 working hours left since the CI is down. During these hours, the team are busying on (i) working with support team, (ii) identifying if any release is urgent and we need to do manual release without CI; (iii) try to explain to the stakeholder that what potential impact of this single incident to the product release roadmap.

The unnoticed new pricing model

Later, Travis CI support team reply to our ticket, and pointed us to blog article “The new pricing model for travis-ci.com” (which we never read before). There are some fundamental change to the pricing model. It is expected previously. But let me highlight the most surprising part

Beginning on Monday the 2nd of November 2020, customers impacted by pricing changes listed above will be gradually upgraded to the new pricing scheme upon their plan expiration date.

So my understanding to it is like that:

We (Travis CI) tell you (via blog article) that we are going to change the pricing model, and being effective tomorrow, in case you are so unlucky that your billing cycle starts from 3rd Nov.

Personally I do understand the reason behind the change in pricing model and it is reasonable to me. We also observed some of Travis CI competitors, like Circle CI and GitHub Action, mostly switched to credit based model.

But it is quite uncomfortable for me that, the “short” notice comes so invisible. It makes me feeling being forced to buy those credit without options (as we are running it for production), and even without know how much we need to pay for (no usage report was sent out). In practice, there are some real budgeting issues, since most budget was previously allocated in fixed way (by concurrent job) and now it is various cost (usage based). The actual usage depends on the (and which we did not have data in hand to give a good estimation at day 1). It will takes a few day to do estimation and the approval from account department (if any in the company).

In short, it sucks.

Last word

As of the writing, we are still working with Travis CI team to see what options we have (eg. rollback to the previous pricing model for a short period, or assistance to give an estimation of how we need to pay).

And I will possibly continue to use Travis CI in short term. But honestly, I am not sure how long we will still use Travis CI, given such this poor experience with it and meanwhile there are tons of CI alternatives.