Solution For Common Problems When Setting Up Test Automation
By: Mohsen Nasiri
Tags: Test Data Generation, Selenium, Web App Testing, Cypress, E2e Testing

When it comes to dynamically seeding your test data, which is in most cases my method of choice, you can do it either via the existing API endpoint, or directly writing to the database. My method of choice is the latter for the reasons that sometimes and for some specific test data, there are no clearcut endpoints to create those data, which again, might not be the case everywhere. Either way and whether using the API of your backend services or running SQL queries on the database, dynamically seeding your test data is what I will be talking about below.

How would this data creation look like and where would it reside? My current approach is to have a service either on a remote Lambda function or as a standalone service, accessible via a secure SECRET_KEY that’s only available to the tests. This service receives a YAML file (or a JSON conversation of it) and returns a JSON object. This JSON object then contains the test data that are necessary for the execution of the test cases, as well as for the clean-up later on.

As an example let’s say you want to write a test case in which ”a user goes to an online car rental app and out of two existing Toyota Corolla car, they should only see the available car and not the one that’s rented out already”. Here is a simple YAML file for this example:

---

template:

car_default: &car_default

manufacturer: 'Toyota'

model: 'Corolla'

year: 1998 rows:

car_available:

table: cars

parameters:

<<: *car_default

rented: false

output: [id, uid] car_unavailable:

table: cars

parameters:

<<: *car_default

rented: true

output: [id, uid]

And its JSON conversion:

{

"template": {

"car_default": {

"manufacturer": "Toyota",

"model": "Corolla",

"year": 1998

}

},

"rows": {

"car_available": {

"table": "cars",

"parameters": {

"manufacturer": "Toyota",

"model": "Corolla",

"year": 1998,

"rented": false

},

"output": [

"id",

"uid"

]

},

"car_unavailable": {

"table": "cars",

"parameters": {

"manufacturer": "Toyota",

"model": "Corolla",

"year": 1998,

"rented": true

},

"output": [

"id",

"uid"

]

}

}

}

Then the mentioned service after receiving the test data file goes through it and starts creating the following database rows for two instances of car_available and car_unavailable

// car_available

INSERT INTO cars ('manufacturer', 'model', 'year', 'rented')

VALUES ('Toyota', 'Corolla', 1998, true) RETURNING id, uid // car_unavailable

INSERT INTO cars ('manufacturer', 'model', 'year', 'rented')

VALUES ('Toyota', 'Corolla', 1998, false) RETURNING id, uid

Which conveniently creates two entries in the cars table. Then this service, having the returned values from the query and the tables’ name, construct and create the following JSON object, and returns it to the running test:

{

"car_available":{

"table": "cars",

"id":1373,

"uid":"xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",

},

"car_unavailable":{

"table": "cars",

"id":1374,

"uid":"xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",

}

}

Afterward, the tests can use these data to access these created entities accomplish different tasks, such as:

Going directly to a url using the returned id / uid (i.e: http://www.mycarrental.com/cars/1373)

using the returned / (i.e: http://www.mycarrental.com/cars/1373) Check if certain elements or values exist based on the properties of the added entry ( name , title , etc)

, , etc) And cleaning up the data, using the table and id value of each returned object.

You might ask why not having the test data written in JSON in the first place, and the answer is in the template part in the YAML file:

template:

car_default: &car_default

manufacturer: 'Toyota'

model: 'Corolla'

year: 1998

This way we can create a default template of the columns and values that might be repeated for several objects, and with this syntax of <<: *car_default , we can easily access these common values which makes the test data easier to write and maintain.

In my Cypress setup, I store the seed files under /fixtures directory and by the same name as the corresponding spec file for better readability:

Cypress structure and the seed file

And roughly this is how the test would incorporate the seeding mechanism:

// globally defined data to be used by all tests

let data = {} describe('my test suite', () => {

before( function () => {

// path to your seed file

const seedPath = 'cypress/fixtures/car_rental.yml'; // call your seeding API and assign the result to data

date = seed(seedPath)

} it('my test case', function () => {

// now in your tests you can access the data object, i.e:

cy.visit('rental_cards/' + date.car_available.uid) // or any other usage you see fit

cy.contains(date.car_unavailable.id).should('have.length', 0)

}) after( function () => {

// here you can clean up the data if you like

cleanup(data)

}

}

This approach is very flexible and there is a good deal of possibilities that you can achieve with your YAML test file. For instance, you can create custom conventions and set some unique characters to deliver specific tasks. Below you can see three use cases for these special characters that I tend to define in my tests data files to add more flexibility to my tests data and make my life easier:

Case 1: Randomisation (special character I chose: ^)

When I need to have some values unique in every test execution, I’ll put a ^ character in my test data followed by a digit that later on and in the seeding service I can replace with a unique randomized string with the length of the digit the comes after while creating the query.

For example, having:

...

name: AutomatedUser_^4

email: automated.^6@domain.com

...

in my YAM test data can be translated into:

INSERT INTO ... (..., 'name', 'email', ...)

VALUES (..., 'AutomatedUser_h7m5', automated.y8wb3c@domain.com, ...)

This way every time the tests are executed, unique usernames and emails are generated since most authentication services have a unique database constant over a username and email columns.

Case 2: Handling relative dates (special character I chose: $)

Another trick that I often do is passing timestamps relative to the current time of test execution, in cases that I need to create a database entry with a date column, that needs to have a timestamp either in the past, present, or future, based on the nature of the test case.

Let’s say I want to create two rentals cars, one that has expired 5 days ago and another one expiring in 10 days. Therefore in my YAML file I can have:

...

rows:

car_1:

...

expires_at: $-5d

...

car_2:

...

expires_at: $+10d

...

...

Which again while processing the SQL query, I can replace $ and any +/- , digits, and letters s/m/h/d/... that comes after it with dates relative to the moment of test execution.

Here are some other examples of using this format:

$now // current time

$-2m // 2 minutes ago

$+3h // 3 hours from now

$+36h // 1 and a half day from now

$-8d // 8 days ago

In case you’re using NodeJS or just want to see which library I’m using to transform this syntax to relative dates, you can have a look at this node module.

Case 3: Handling foreign keys (special character I chose: =)

You will definitely come across cases wherein your test file you want to create an entity, like company and then create a user for that company, which essentially means using the ID of the created company as a foreign key for that employee.

To handle foreign key references I use another special symbol, an equal sign:

company_google:

table: companies

parameters:

...



employee_1:

table: users

parameters:

...

company_id: =company_google

When I create and execute queries, I keep track of the returned ID of all my previous queries along with the object’s name (like company_google in this example) and while creating new queries, whenever I come across a = symbol, I look up in my previously executed queries, for the object name that comes right after = and use the returned ID of it as a foreign key for my current query.

Cleaning up the test date

Deleting the test data that your tests create is not something that you have to do. Depending on the volume of the generated data and the capacity of your database, it might not even be worth spending time for.

However, for anyone whom this might come in handy for I will briefly explain the way we usually clean up our tests data:

Every time a row of test data is created, in the same endpoint, we save the primary key and the Table name of the created data in a table we call test_data . Then once a week on a Friday night we execute a cron job that iterates over that table and DELETE CASCADE s all those data.

That’s basically it!

3. Testing emails in user flows

We often come across user flows in which an email being to the users as a result of an action, a.k.a transactional emails, which are crucial for us to ensure not only these emails are being sent to users, but also the content of the email is fully correct as well. User flows like signup, forgot password, and invitations are some examples.

Of course, one way to test emails are being correctly sent and received is to access the inbox webpage in the tests, look for the right email, open it, and check its content, but this approach comes with plenty of overhead. First of all, this approach is testing the email clients, such as Gmail, which is not in the scope of the test case, and secondly, this practice is too expensive to maintain.

One thing we can do here is using an email testing solution, such as Mailosaur, that enables us to send email to specific inboxes and using an API_KEY access the inbox, and get specific emails via its REST API . With Mailosaur or similar tools, you can have full control over the transactional emails the are sent by your application and received by your test user, by simply looking for the exact email address.

As an example consider this scenario: You want to try the registration endpoint of your application. All you need to have is a unique email address like test+2dgx97d@xxxx.mailosaur.io , use that emails address to register, and in your tests wait for an email to appear in your Mailosaur inbox, with the SentTo value matching the unique email address that you have.

2dgx97d could be a unique hash that you create while running your tests, which will be different and unique in every execution

could be a unique hash that you create while running your tests, which will be different and unique in every execution xxxx is your Mailosaur’s SERVER_ID

As an example, here is a part of our current test suite that checks if our application is sending emails to users in case they’ve forgotten their passwords:

it('Should send email when user clicks Reset Password', function() {

...

cy.mailosaurGetMessage(Cypress.env('MAILOSAUR_SERVER_ID'), {

sentTo: data.test_user.email,

subject: loginPO.resetEmailSubject,

}).then(email => { // Checks the subject of the email

expect(email.subject).to.equal(loginPO.resetEmailSubject); // Checks the body of the email

expect(email.text.body).to.contain(loginPO.resetEmailBody); // Checks the link in the email is pointing to the right place

expect(email.html.links[0].href).to.contain('/user/reset-pw/');

...

})

4. Locales and translations

In some tests, you want to check if certain text or messages appear or not, for instance in the case of users entering the wrong email or password:

...

cy.get(#emailInput).type(emailAddress)

cy.get(#passwordInput).type(wrongPassword)

cy.get(#submitButton).click()

cy.contains('Your email or password is wrong.')

...

But the problem with this approach is that if someone in the team, like the product owner or the marketing, decides to modify the message Your email or password is wrong , then the automated tests would fail, which is far from ideal since the main focus of our end-to-end tests should be the functionality and not some minor copy changes.

A solution to this problem could be using the message key that’s being mapped to the message value instead of the value itself, so changes will not be breaking the automated tests which their goal is to test the functionality of the web app and not minute details like the message.

This can be done in two very simple steps:

Step 1: In most cases, these messages, or commonly referred to as translations, reside on 3rd party services such as Phrase, and while your app is being built, set or sets of translation files are downloaded within the project directory. For instance, the current project that I’m currently working on has its Phrase translation in this path, which are basically JSON files, one for English translations and one for German:

➜ project: ls src/locales/langs

➜ project: de-DE.json en-US.json

And all these files contain is simply key and value pairs, like:

{

...

"login.password.wrong": "Your email or password is wrong.

"login.password.empty": "Please enter your email."

...

}

Therefore the first step is as easy as copying these files under cypress/fixtures , which can be added as a step in your CI pipeline and before the test execution step:

cp -a path/to/translations/. cypress/fixtures

You might ask: Why under fixtures directory? And the answer is found in Cypress’ documentation:

Fixtures are used as external pieces of static data that can be used by your tests. Fixture files are located in cypress/fixtures by default,

This basically means any file JSON , YAML , etc formats that are set in the fixture path can be directly used within the tests.

Step 2: Now all you have to do is reading from those files in your tests by using the cy.fixture() command:

beforeEach(function() {

// name of the fixture file, without the extension (.json)

cy.fixture('en-US').then(translation => {

// this will contain all the key:value pairs

this.translation = translation;

});

}); ... it('my test case', function() {

// here you can access fixture values

cy.contains(this.translation['login.password.wrong']);

...

}

And that’s it. From now on, if the translations change by a developer, product owner, or anyone else within the team, the tests will not be affected by it!

Conclusion

If you are about to set up an end-to-end testing infrastructure for your web app, you might benefit from having a look at this article. There might be topics that you haven’t taken into account yet, or you have still open problems on your mind that you are not sure how to tackle.

Either way, here I talk about the common problems that I faced when setting up testing infrastructure in multiple startups, and the way that I overcame them, hoping that these past experiences of mine might come in handy for others heading down the same road.