Why changes in ignored tests might affect CI build? How to correct it?
By: Reghill J Manuelraj
Tags: Practice, Continuous Integration, Tdd, Tutorial, Unity3d

Unity Test Runner Window

I am working on an Unity project in which the CI build failed due to some addition / removal of code in the ignored play mode tests. You can learn more about testing in Unity here .

Tests are not strictly part of CI , but it is typically included.

What are ignored tests?

Usually , tests are written to make sure that any future changes in the functionality does not affect the previous functionalities and ensure proper building on top of it.

Test 4 is an example of typical ignored test

Ignored tests are tests that might fail in the CI environment due to variety of reasons such as absence of drivers or any components in CI environment that are present in local machine. These tests would pass locally.

Why to ignore tests?

Continuous Integration(CI) usually contains many plans/tasks which are chained together to form a pipeline. If you wonder what CI is , just check this high level overview of CI. Usually, a plan has to pass so that the next plan is triggered . Consider, you have 3 plans chained together. The second plan will be triggered/run only if the first plan is executed successfully.

Let’s say, you don’t ignore the test that has to be ignored . It might be in your plan 1 or plan 2 in your CI. It will result in failure of the plan , which eventually results in blockage of the other plans to be triggered. It blocks the proper availability of the deliverables of your software/product.

How to Ignore tests?

It is quite simple . Just use the annotation [Ignore(“Any reason as you wish”)] available in NUnit.Framework either in class level or function level based on your requirements.

Why changes in ignored tests affect CI?

In the CI plan , you would have to include a script which does all the necessary actions that you desire. That script might contain details of any diff/patch files that contains details of your ignored files. Let’s call it ‘ignored_tests.patch’.Usually, when you take a pull from git, it makes use only of the difference between two versions of the same file to update the changes in your local system.

In addition to that , git verifies the code line by line internally to see if there are any changes . Let’s say , you have included the [Ignore] annotation in the 10th line of the code . The file ignored_tests.patch is assumed to be pushed to the repository.

Now, consider, you are adding new code into that class file in which some tests are ignored. Let’s call it IgnoredTests1.cs. It may/may not have external dependencies for smooth operation. Assume that you have an external dependency in the new piece of code that you have added. You add those dependencies / external packages. Now after addition , your [Ignore] annotation might have moved to 11th line(or any other line).

When your CI plan is triggered , it would look for ignored_tests.patch . The patch file in CI doesn’t know that you made some changes in that file. It still remembers that 10th line has [Ignore].The triggered action would check in your code “Hey, Look for that guy called Ignore in the 10th line”. Unfortunately , your code might not have it in 10th line and it is in 11th line . The triggered action will fail instantly after investigating upto 10th line. The poor guy ignored_tests.patch did not know that you pushed the guy [Ignore] to 11th line.

This will cause in failure of CI build.

How to avoid it?

Whenever you make any changes to these ignored files , you just make sure you update ignored_tests.patch.

Consider , you are making changes to the test class file IgnoredTests1.cs.

After making changes , enter the following command in git bash

git status

You will see the IgnoredTests.1 listed.

Now ,

git diff > ignored_tests.patch

This will update the ignored_tests.patch file. After updating the patch file , push it to the repository using the following command

git push

Thus , any issues regarding this in CI could be avoided.

And the patch file would also be happy that it did not disappoint the CI plan and of course, Devs.!