Setting up a Selenium Pipeline In Azure DevOps
By: Sai Prasanth
Tags: Azure Devops, Ci Cd Pipeline, Selenium, Automation Testing, Front End

Setting up a Selenium Pipeline In Azure DevOps

Thank you, Ankit Thakkar, for co-authoring the blog and adding your valuable inputs to improve its quality.

At Clairvoyant, we have been working with different CI/CD tools to integrate automated tests so that they can run in a scheduled fashion. Very recently, we explored Azure DevOps pipelines to run our tests and performed extensive research to configure the pipelines for our automation tests using selenium. As part of our research and learning, we realized that setting up these pipelines is comparatively challenging and different from the other CI/CD tools.

As a way of documenting and sharing our learnings, in this blog, we will discuss a few different ways to create an automation pipeline in Azure DevOps using selenium with Maven.

Introduction

Azure Pipelines is a feature that provides Continuous Integration and Continuous Deployment.

Continuous Integration — Continuous Integration is the process of integrating code changes from multiple contributors to a single project.

In this document, we will go through the steps to create a pipeline, execute the tests and view the results.

Here, we will be using Git as a Repository.

Setup of Azure Pipelines

There are 2 ways of creating the Pipeline -

First -

A) Selecting a project and connecting to Git

Navigate to https://dev.azure.com/{projectName} and select Pipelines.

2. Click on the “Create Pipeline” button. Select the repository where the code is present. In our case, it would be Git.

Once the “GitHub” option is selected, the user will be navigated to the GitHub login page. Provide the required credentials for the user to be displayed with projects as the user now has access.

Select the desired project for which test cases have to be executed. Once a project is selected, the user will be navigated to the authorization page. Select the required project and click on the “Approve and install” button.

B) Configuring the pipeline

As the project is a maven project “Maven” should be selected from below.

Once maven is selected, the user will be navigated to Review Screen where a template of the yaml file will be displayed. In the next section, we will understand the yaml schema.

Second -

The second way is using the classic editor for which the YAML file isn’t required.

To do that, we need to click on the ‘Use the classic editor’ link at the bottom.

Selecting Repository

Once the user clicks on the link, it’ll navigate to the Repository page. It will show all the above options for the user to select the Repository. Once selected, it will navigate to the sample template page.

2. Selecting the Template

After selecting the Repository, the user will now have to select the template. It has the Yaml file template as well, but in our case, we will use maven.

3. Selecting the OS

Once the user clicks on ‘Apply’ for maven, the user will be navigated to the final setup page. Here, the user can update the Name of the pipeline and select the OS on which the pipeline should execute the test cases. Azure DevOps by default provides the following OS setups:

After selecting the OS, the user can add variables if required, for example, if we want to select the browser while executing the pipeline, then the user can add that as a Variable. By default, it will create some variables, and we can add our own variables by clicking on the ‘Add’ button.

From the Triggers tab, the user can select the auto-scheduling of the pipeline.

In the above screen, the setup is done to execute the pipeline at 6 AM every weekday.

Once these are done, the user can click on the ‘Save & Queue’ button to complete the pipeline set up and it gets triggered once the user clicks on the button.

C) Understanding the yaml schema

Suite file on the pipeline will be executed in accordance with the keywords specified in yaml file.

trigger: This keyword is to specify when the pipeline should execute. By using this keyword, we can execute pipelines only when commits are made to certain branches.

Example –

In the above image, the pipeline would get triggered when commits are pushed to the master branch alone. It would not get triggered when any commits are pushed to branch1.

4. If the pipeline should get triggered when commits are made to any branch, below is the screenshot

Example: -

P. S — If an exclude clause is specified without an include clause, the pipeline would trigger for any commit made to any of the branches.

pool: This is to specify the execution of tests on a specific VM machine.

By default, Microsoft has created some VM machines which can be used by anyone in the project to execute test cases. Below is the list of VM images created by Microsoft:

ubuntu-latest (for Linux)

macOS-latest (for MacOS)

windows-latest (for Windows machines)

Follow the below example to use the above VM images:

Example —

pool:

VMImage: ‘ubuntu-latest’

Apart from these, there would be private pools that can only be created by Administrators. When private pools are used, follow the below example:

pool: ‘{poolName}’

steps- Steps can be tasks, scripts, or references to external templates. Any task, such as executing a maven or extracting a zip file and storing it in the destination folder should be specified under steps. tasks- Tasks names should be specified after a task keyword. Below are a few examples:

To execute maven tasks, we can specify it as a task — task: Maven@3

To unzip a folder, we can specify the task as — task: ExtractFiles@1

To publish results, we can specify the task as — task: PublishCodeCoverageResults@1

3. inputs- All the commands required for the above-specified tasks should be specified under the inputs tag.

Let’s look at the mandatory commands to execute a maven task:

mavenPomFile — This key is used to specify the location of pom.xml in the project directory.

Example 1 — mavenPomFile: ‘pom.xml’

Example 2 — mavenPomFile: ‘{folder Name}/pom.xml’

mavenOptions — This key is used to specify the mavenOptions. The ‘Maven options’ variable is used to send command line arguments to start JVM. The ‘Xmx’ flag specifies the maximum memory available in JVM, the default value is Xmx1024m.

Example — mavenOptions: ‘-Xmx3072m’

goals — Specifies the maven command to be executed.

Example — clean install

‘clean’ would clean the target folder. Once the ‘clean’ command is executed, all the libraries would be erased, and the user will be required to start on a clean slate.

‘install’ would reinstall all the dependencies.

‘test’ would execute the tests without deployment.

Example to specify goals with suite file as dynamic value- clean install test — DSuiteName=$(SuiteName)

P.S — Any other variable which has been specified as profiles or properties can be declared under goals.

Example — clean install test -DEnvironment=$(Environment) -DSuiteName=$(SuiteName)

4. timeouts — This variable is used to specify the amount of time a specific job should run. Setting the value to 0 would allow the job to run on the below conditions:

For a self-hosted agent, it would run indefinitely by setting the value to 0.

The maximum time limit for a public project or public repository on Microsoft Hosted agent is 6 hours.

For a private project or private repository on Microsoft Hosted Agent, the maximum time limit is 60 minutes (until additional capacity is paid for).

D) Add Variables

Variables can be user-defined and there are also a few system-defined variables.

To view the complete list of system-defined variables, please click here.

As discussed above, if the goals specified in yaml file have dynamic values, we need to add variables in the pipeline so that the pipeline would append those values and execute the goals.

To define a user-defined variable, click on ‘edit pipeline’ and then on ‘add new variable’. Add the name and value of the variable.

Example –name of the variable -SuiteName

Value for the variable — testng

In the above example, the variable name is in accordance with the above-specified maven goal. So the pipeline would look for testng.xml in the project directory and execute the test cases specified in the suite.

E) Scheduling tests on Azure DevOps

Executing test cases on a schedule is very important to check the functionalities. It also helps the team identify defects or any code changes that need to be made. We can schedule tests on azure devOps by either using yaml or azure devOps UI.

P.S — If schedules are set both on yaml and UI, UI will override the settings of yaml.

i. Scheduling tests from UI

Below is the way to set up a schedule using azure devops UI.

Select a pipeline and click on ‘Edit’.

Click on the 3 dots icon and select ‘triggers’.

Click on ‘Add’ beside scheduled.

Select a time and day(s) for the scheduled job to be executed

Select the required branches and click on ‘Save’.

Note — To view the scheduled runs for a pipeline, select a pipeline, click on the 3 dots icon and select ‘Scheduled Runs’.

ii. Scheduling tests using YAML

Below is the procedure to schedule tests using the yaml schema:

schedules — This keyword should be used to add a schedule to a pipeline.

a) cron — cron syntax should be used to schedule a pipeline.

<minute> <hour> <day-of-month> <month> <day-of-week>

Use the above format to define a cron timing.

Below are some of the variables that can be used to specify a cron syntax:

* — Specifies all possible values for a field

? — Allowed only in the ‘day of month’ and ‘day of week’ fields. Used when a value needs to be specified in one field but not in the other

Example — If a pipeline needs to be trigged on 1st of every month but the day of the week doesn’t matter, then ‘?’ will be used.

‘-’ — Specifies a range of values

, — Specifies a list of values

/ — Used to skip a given number of values

Examples for cron syntax are:

https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.html

https://github.com/atifaziz/NCrontab/wiki/Crontab-Expression

b) displayName — This key is used to specify a name for the scheduled pipeline

c) branches — Branches that need to be executed on a scheduled basis are to be added here

Example –

The above example will execute the tests on the master branch on the specified schedule.

We can also exclude a specific branch if we decide that code changes to the branch need not be executed. Below is the example:

d) always — This key is to define if a pipeline should run when there are no code changes.

If always is set to true, the pipeline will execute at the specified time even if there are no code changes

If always is set to false, the pipeline will execute only at the specified time even if there are code changes made to the branch

Example for schedule:

F) Executing the test case and viewing the build result

Once all the variables are set, click on ‘run pipeline’. The pipeline will execute and display the results. All the JUnit results would be under Test Plans -🡪Runs.