Think Like a Python Automation Engineer
By: Prashant Kumar Pathak
Tags: Robot Framework, System Testing, Pytest, Python3, Automation Testing

I read many articles about the use of Python in software development, but very few of them focus on using Python for test automation. I have been working with test automation for more than 9 years, and my main programming language has been Python. I have decided to share my experience on how to think like an automation engineer. With this knowledge, you will be able to develop a framework for any given test scenario.

Photo by Andrew Neel on Unsplash

Introduction

Verification of software is considered as one of the key phases for the entire software life cycle. Without verifying software quality, it is not possible to deliver it to the end-users/customers. Verification helps identify the critical bugs during the software development life cycle and helps improve the software quality to a great extent. Due to the frequent release of the software (sprint releases), it’s become tough to test all the major and minor software functionalities within a limited time frame. This brings the thought of applying Test Automation in the field of software engineering.

Applying the Automation in the software verification adds the following benefits:

It helps in a quick regression check for every major/minor build release.

Improves quality.

Manual execution of test scenarios reduces, which helps find more bugs.

Complex scenarios get automated through various tools and data sets.

Quick test results and reports.

Many more.

Just a few words about quality

Quality is considered the backbone of the entire software irrespective of the field it belongs to. Few major sectors like Avionics, Automotive, Healthcare give more focus and budgets to the quality of products. It will be worth reminding the historical incident that caused a huge loss to the industry and the research called A space error: $370 million for an integer overflow.

The history reminds us how important verification is for a product before it goes into operation.

My experience is that the following three metrics are the most important:

Independence Rule: States that the code developer can not be a tester, and a tester can not be the test reviewer and related outputs. SCA needed to verify the code has been covered 100% and if not, then the proper analysis of the code uncovered. Tracing matrices can be tests →requirements →codes (called bottom down approach)

A good starting point for measuring the quality of a product by focusing on the following metrics

Does the Software verification life cycle follow the Independence rule?

Has the code coverage analysis been performed?

Has various tracing matrices, i.e. bottom-up tracing or top-down tracing, in place?

Has the software verification results plan been created?

Steps Towards Developing a Test Automation Framework

In a project applying automation is not that straightforward, before choosing this approach various factor needs to be considered like

Various tools cost

Engineer’s experience and skills set

How frequently are test runs required (daily, weekly, biweekly or so on)?

Efforts in developing the framework from scratch

Challenges to cover a complex manual step into automation.

How does this automation framework set into the organization-wide automation infrastructure?

Considering the above challenges, it’s always considered a good idea to do a brainstorming session (developer and tester). This helps to identify the feasibility of test automation, deciding on the available open-source tools, languages and other infrastructure.

Scenario to Automate test for:

Software consist of various independent components, such as:

One component for storing the logs from various sources and then forwarding it to the next component for data analysis

The next component receives the logs from the previous one and then applies the various analytic Models (data science) to these logs and forwards the report to another component.

And the last component receives these analytic reports and displays them on the UI for end-users like security analysts.

Find the overview of the above application in the block diagram view

Application Under Test Block Diagram

Test cases for the different scenarios

Define the structure of a test case and explain the different sections.

Test Scenario#1:

Requirement-ID : Logs_receives_from_various_log_sources

Description: Verify the component_1 can receive logs from various logs sources. It can be a Syslog server, site collector or any cloud service.

Test Type: Automated

Test Steps :

ssh to component_1 and check the required docker services are up and running by using the docker ps command on CLI.

command on CLI. Able to get an API session for REST API calls.

To achieve the reception of logs from various sources, the “netcat (nc)” command can be performed by creating a log.txt file with some unique string logs.

cat raw_logs.txt | nc localhost <port number> #port number where the component_1 receives the logs at the eth interface.

Expected Result :

Verify the Syslog logs at the path rsyslog folder is growing and grep for a unique string inside rsyslog logs using the below CLI command.

user@prashant:cat /opt/app/data/input/rsyslog/Internal.syslog.log | grep ‘your unique string from log you netcated’

Test Scenario#2:

Requirement-ID : Alert_rules_ran_on_log

Description: Verify that component_2 can pull logs from component_1, and alerts are triggered based on the alert rule applied on these logs.

Test Type: Automated

Test Steps:

ssh to component_2 and check the required docker services are up and running by using docker ps command on CLI.

command on CLI. Able to get an API session for REST API calls.

Configure an alert rule through a POST API call; this rule will be applied to the logs coming from component_1 to component_2 to identify the alerts.

A Sample Robot File With REST API POST Call

Expected Result :

Verify the file “syslogs.log” at the path rsyslog folder is growing and grep for a unique string inside rsyslog file.

user@prashant:cat /opt/app/data/input/rsyslog/Internal.syslog.log | grep 'your unique string you have sent through NC command'

Verify the alert logs inside the alert_engine application logs. The alert logs will contain the rule name configured on the component_2.

user@prashant:cat /opt/app/data/output/alert_module/alerts.log | grep 'alert_type'

Test Scenario#3:

Requirement-ID : incident_created_based_on_alerts

Description: Verify the component_3 can receive alerts logs from component_2 and create the incident report ticket for further use.

Test Type: Automated

Test Steps:

ssh to component_3 and check the required docker services are up and running by using docker ps command on CLI.

command on CLI. Able to get an API session for REST API calls.

Check the configuration files are in place by running the find with the file name on CLI.

Expected Result :

Verify a new incident id is created on the incident responder application logs.

user@prashant:cat /opt/app/data/output/incident_responder_module/incident_service.log | grep 'incident_id'

Verify the response of the GET REST API call contains an incident id.

How to monitor the output logs?

There are various ways to verify the output logs for a specific value. Like

Through grep command on current growing logs.

Applying a polling method on the logs.

Both can be used to achieve the task but applying a grep method may not always give a consistent output because data logging is sometimes delayed. Therefore I prefer the polling method as a safe option to go with. A polling method ensures to try some specified number of attempts with a specified interval to look for the value, and if it’s not successful, it returns Fail.

A sample robot framework polling keyword has been attached below for reference.

A Robot Framework Keyword

NOTE: Here, I have used “grep” by doing a Linux system call only for demonstration. You will see that several system calls are done in the example code, such as using grep, sed etc.

This is fine if you have a well-defined system, but the code becomes less portable, and it is difficult to handle the error codes/scenarios. For example, you can find the desired string from the log file using regular expressions in a production version of the code.

Test environment

Follow the steps in the below link on how to create the test framework:

Test Environment Setup

Test Case Demonstration

This code assumes that you have three running Nodes for each component, and it’s provisioned to access through the SSH terminal, and your environment is set up as per the instruction provided in this article.

Both PyTest and Robot Framework style code snippets were added as a reference. Use the below link to see how a test case can be developed using PyTest and Robot Framework for the Application under Test.

Using PyTest

Robot Framework

Conclusions

In this article, we looked at developing an automation framework for an Application Under Test using Python. Once you start developing the framework for your application, you will get a better understanding.

If you have some tricks you would like to share, feel free to share them in the comments.

I hope this helps you; thank you for reading!

Take care, and I hope to see you soon.