Why you shouldn’t use Guerrillamail for automated E2E testing
Tags: Programming, Test Automation, Testing, Email, Development

First of all, a little bit of context:

At Koa Health, we create digital health apps, and this means they are required to complete critical quality standards to certify they are ready for the market. This implies lots of manual testing, exploratory testing and documentation. We manage all this inside our Quality Management System, but this will be part of a future post. However, prior to that, our main goal is to optimize the process of development as much as we can, which is why we have a testing automation strategy built on top of a continuous development ecosystem. As a summary of this, we work with Flutter to develop multi-platform mobile applications and node(js) for the backend + frontendish stuff.

From the quality team, we encourage developers to take care of the testing as much as they can, because we believe that quality is not just the last step of the chain but a critical part of each step. The culture we build around this small Quality Engineering team (QE), compared to a big development team, is that everyone is responsible for their tests, while QE provides the tools, the guidance and the support to maintain everything in shape. Then, of course, there is a final layer of End to End tests that are completely owned by the QE team. This post is about these E2E tests that are basically reproducing a real user scenario from the login page to the creation of another user, as the last safety net before something gets deployed. When we first built the E2E tests we were using an external service (Guerrillamail) to create random email accounts with a public API and use them to successfully log in and create new users in our dashboard application (the control panel for users management). Life was good (before pandemic kind of good) and the grass was always green as well as the CI pipelines. But then, this happened:

Well, you get what you pay for! But, although this was probably something unfair, you really don’t want external affairs affecting your E2E tests (and getting all your developers crazy for more than 4 days with the service down). This is why you want to have a service that can provide you some reliability. After the incident, we decided to investigate some alternatives to Guerrillamail that can provide an API for creating random email accounts and reading emails. Easy-peasy. But lesson learned, now that we were actively thinking about it, we wanted to make sure we would get the right capabilities:

Non-total public exposure (guerrilla is just a take and go available to everyone, everywhere). We want some level of privacy for this type of service so not everyone has access to our random uncontrolled emails.

Good API, good documentation, plug-and-play if possible! We don’t want to spend a lot of time on configurations or whatever. Because, you know, we are testers, not developers.

Good price! We don’t want to pay too much for something as easy as one-time-use email accounts. But at least we want some support service if things go wrong at some point.

After some research and asking the marvelous community of ministry of testing, we’ve got some options:

Mailinator

Some level of privacy

Decent API

Support looks reliable enough

The free version doesn’t support API calls (and the emails are totally public). At some point, this can be helpful for other purposes but the main pain point here for the starters subscription is the price. After the free version, the next price is 160$ per month, so it’s out of our scope. It seems to have some nice extra features, but not for our use case.

Gmail API

That was our first option, as we already have the accounts for our company. Everything was looking fine until we reached the authentication point. First, trying to use Oauth2 there is always a step where the UI is required to authenticate, which was not the best option for our tests. We also tried to use API keys but they are only allowed for some specific services, not including Gmail. Either way, it seemed too complex for supporting E2E test cases. Maybe in the future, we will give it another try, but it feels overwhelming at the moment for having to set up the Google API development console, Oauth, client services, etc. This option is really focused on web applications, not on testing. But maybe is exactly what you are looking for. Who knows?

MessageBird

A service that offers a lot of capabilities but it’s not useful for only testing emails. However, it would be good to check for other use cases, as they describe it: Customer service software and automation to talk to your customers on channels like Live Chat, WhatsApp, Voice and more.

MailSlurp

The winner, for our requirements. A private service that can handle our workload for the E2E tests with the basic subscription (20$). Impressive documentation, thanks to swaggerUI, that helps you understand what is the scope of the tool and how you can use it. Also, the official SDK for different languages and usages was just plug-and-play (with already built-in functions to wait for messages for example). A breeze to use, and with typescript support. What we need to test now is the reliability and the service, to see how support responds to any inquiry, but for now, we will give it a try.

Mailosaur

Well, this arrived after we’ve done the POC and decided about MailSlurp, but it looks pretty impressive. Good pricing for starters (10$), SDK with good documentation and very similar to what we bought with MailSlurp. Absolutely recommended to give it a try, but you know, we are in a rush here so it’s all in your hands 🤷‍♂

Mailgun

Mailgun was on the table also, with similar pricing and capabilities as MailSurp. But, they don’t have an official node SDK, so during the POC, we’ve just moved into Mailslurp as it was fitting more into what we needed: documentation, SDK to go! Also, it’s a little bit more expensive than MailSlurp but it seemed focused on more than just testing and receiving emails. The API capabilities are pretty impressive, maybe too impressive for what we are looking for. Also really powerful sending functionalities. In any case, a good contender if at some point we are not happy with MailSlurp service.

Conclusion

We’ve tried different services, each of them with different capabilities and we’ve decided on the easiest, less expensive and good enough of all of them. Some are too complex or have a bigger scope (Gmail, MessageBird), others were discarded because the entry point was too high (Mailinator price), and some others were sexy enough but we didn’t have the need to get in deep (Mailgun, Mailosaur), due to an instant feeling with MailSlurp. Also worth noting, we are very biased towards node/typescript, so MailSlurp just offered something really juicy for us and that might not be your case. But I hope this is a useful first filter for you if looking for different options.

The only thing I would take for granted in this whole proof of concept is the following: don’t put your E2E tests on a dependency of an external service that is free, totally public whose only answers come from a Twitter account. Lesson learned.