Automation Reporting
Tags: Selenium, Testing, Automation, Quality Assurance, Reporting

Hi Guys,

Today we are starting on a topic which is very important strategically in the long run for both testers and the organization as a whole. Why we are dedicating a whole article to Reporting is that it is a critical aspect for a better testing scenario for the fact that clear and comprehensive reports help us make inferences and reach a conclusion post script execution.

These reports allow us to analyse our test results by breaking down the results leading to a better identification of the areas that need intervention and should be focused upon. When we say break down we mean dividing the test results based on the environment or configurations. The aforementioned factors help us gather a faster feedback which pays in the long run. In the real world testing we come across large tests where end to end testing is done to check how all subsystems integrate. keeping a track of our test runs helps in drawing comparisons, tracking progress, and identifying flaky tests.

Contents of an Automation Report

Suite Level

Browser Details — Name and Version of the browser on which the test was executed on.

— Name and Version of the browser on which the test was executed on. Platform — OS Details of the machine on which the test was executed. Eg: Linux, Mac, or Windows.

— OS Details of the machine on which the test was executed. Eg: Linux, Mac, or Windows. Test Group Executed- Group targeted for execution. Eg: Regression or BAT or Smoke.

Group targeted for execution. Eg: Regression or BAT or Smoke. Total Test Count — Count of total tests executed for the given execution.

Count of total tests executed for the given execution. Total Pass Tests — Count of total tests passed during during execution.

Count of total tests passed during during execution. Total Fail Tests — Count of total tests failed during during execution.

— Count of total tests failed during during execution. Total Skip Tests — Count of total tests skipped during during execution.

Count of total tests skipped during during execution. Start Time — Time at which the test suite was started.

Time at which the test suite was started. End Time — Time at which the test suite was ended.

Test Level

Test Name — Name of the Test Method Name.

Name of the Test Method Name. Test Status — Result of Test Execution — Pass/ Fail/ Skip.

Result of Test Execution — Pass/ Fail/ Skip. Test Duration — Duration of test execution time.

Duration of test execution time. Failure Reason and Exception — In case of failure, the reason of failure captured by assertions or exception handling.

In case of failure, the reason of failure captured by assertions or exception handling. Screenshot of the screen for failed tests — Screenshot of browser/ device on which the test was executed in form of png/jpg format.

The details in the reports are not limited to above mentioned ones and can be customised to cater business requirement(s).

Type of Automation Reports

TestNG Report — TestNG library brings a very convenient reporting feature. Once you execute the tests, TestNG generates a test output folder at the root of the project.

Extent Report — Extent Reports are the most effective and highly used automation report when it comes to automation. It is a configurable report which provides customisation based on user needs. For more details go to http://extentreports.com/

Cucumber Report — This generate html format report summarising features and scenarios level details. It comes with multiple customisation and support for languages. For more details refer to https://cucumber.io/docs/cucumber/reporting/

ReportPortal — ReportPortal is a service, that provides increased capabilities to speed up results analysis and reporting through the use of built-in analytic features. It provides real time reporting along with historical test execution analysis. For details refer https://github.com/reportportal/reportportal

Allure — Allure Framework is a flexible lightweight multi-language test report tool that shows a very concise representation of what have been tested in a neat web report form. For more details refer to http://allure.qatools.ru/

Now that we have discussed the details of reporting in test automation lets once again understand as to why reporting is critical. Though we know that automation itself is a time saviour for us and saves a lot more energy and efforts as compared to manual testing but capturing the test results and connecting them with individual test cases is a big task. Here is where we need proper reporting to be able to trace the test scripts back to the test requirements and their respective test cases.

The underlying point here is that being able to draw conclusive information from a test run is very critical and for that we need to reach to the source or to say the root of the problem, if any. Here is where numbering or tagging exits or failures might come as a rescue.

The outcome of reporting basically is data, data that would later help us draw inferences. These analytics would assist us in finding out the problems and acting on them. Numbering the latest revisions would also help us in narrowing down which change caused an issue.

The report should have clear and crisp failure details which should make debugging of the script failures less time consuming, hence focusing on delivery.

So, we feel we have stressed enough on the advantages of reporting and how to even get it into practice. We would love to hear your views, opinions a or queries here so that we can together enrich the idea discussed.

Have a good day folks.!!!!