Performance and Load Testing with JMeter
By: Semih Arslan
Tags: Software Testing, Load Testing, Jmeter, Performance Testing

ABOUT JMeter

The Apache JMeter™ applications is an open-source software, a 100% pure Java application designed to load testing functional behavior and measure performance. It was originally designed for testing web applications but has since expanded to other test functions.

You can use JMeter to analyze and measure the performance of web applications or a variety of services.

Features

· Open source license: JMeter is free, allows developers to use the source code for their own applications.

· Platform independent: JMeter is a 100% pure Java desktop application. So it can run on multiple platforms

· Full multithreading framework: JMeter allows concurrent and simultaneous sampling of different functions by a separate thread group

· Visualize Test Result: Test results can be displayed in different formats such as a chart, table, tree, or log file

· Easy installation: You just copy and run the *.bat file to run JMeter. No installation is needed.

· Multiple testing strategies: JMeter supports many testing strategies such as Load, Performance, Stress, and Functional Testing.

· Support multi-protocol: All basic protocols such as HTTP, JDBC, LDAP, SOAP, JMS, and FTP are supported by JMeter

Requirements

JMeter has minimum system requirements.

Java Version

JMeter is compatible with Java 8 or higher. You can’t start Jmeter if you don’t have Java installed in your system.

Operating Systems

JMeter is a 100% Java application and should run correctly without operating system dependency.

Why Is Jmeter Needed?

When we need Load, Stress, or Performance testing, we can use Jmeter. In this way, we can handle more than one need with a single tool.

Knowing the Performance, Load, Stress or Functional Testing will help us understand our needs.

Performance Testing:

Performance Testing allows us to test whether the applications work within the planned performance criteria. Each application should have some performance criteria. They reflect our approach for the quality of the product.

Load Testing:

It is the type of test that gives information about how much load the system can carry, while still working with maximum performance. It is done to understand how much load the system can tolerate. Thanks to the Load Testing, it is easier to determine the load balancing servers or the necessary hardware required by a system whose performance test has been completed.

Stress Testing:

The Stress Testing is a type of test, performed to observe how the system behaves under a load above the planned maximum capacity, and most importantly, at what point it breaks. The expected behavior is that the application remains stable in producing accurate results after the test is over.

Functional Testing:

It is the technique used to test the features and functionality of the system or software. Functional Testing looks for answers to questions such as “can the user do this” or “does this feature work”.

How Does JMeter Work?

JMeter simulates a group of users sending requests to a target server, and return statistics information of the target server through graphical diagrams.

JMeter Components

In this section, instead of discussing all JMeter components in detail, we will include the most important JMeter components that will allow us to get started quickly.

Thread Groups

Thread Groups is a collection of Threads. Each thread represents one user using the applications under test. Basically, each Thread simulates one real user request to the server.

- Set up a number of threads

- Set up a ramp-up period

- Number of times test execute

Elements of Thread Groups:

Ramp-up Time

Assume that the applications we offer in our test are wanted is to be tested for 1000 users. It is not very realistic to let all 1000 users in the system at the same time. Ramp-up Time determines how many seconds 1000 users will be included in the system by the test tool.

When the Ramp-up Time is given as 20 seconds for 1000 users, 50 users will have entered the system at the end of the first second.

Loop Count

Loop Count is the total number of entries of each test user into the system. For example; if we get 10K users into the system in 1 minute and let the finished user enter the system over and over again (10 times in total), we have provided 100k users.

You can access:

Right-click on the “Thread Group” > Add > Threads > Thread Group

Think Time

It takes some time for real users to use it with a mouse or keyboard between two test steps. This time can be simulated with Think Time, which is determined between steps for virtual users in load and performance tests.

You can access:

Right-click on the “Thread Group” > Add Think Times to children

Sampler

JMeter components are needed to interact with the system to be tested.

Below you can see the most used sampler types.

HTTP Request: This sampler lets you send an HTTP/HTTPS request to a web server. Consider the example below. JMeter sends an HTTP request to the Google website and retrieves HTML files or images from this website.

Access Log Sampler: This sampler allows you to read access logs and generate HTTP requests. The log could be image, Html, CSS…

FTP Request: Let’s imagine you want to performance test an FTP server. You can use an FTP request sampler in JMeter to do this task. This controller lets you send an FTP “download file” or “upload file” request to an FTP server.

JDBC Request: This sampler lets you execute Database Performance Testing. It sends a JDBC Request (an SQL query) to a database.

Mail Reader Sampler: The Mail Reader Sampler can read (and optionally delete) mail messages using POP3(S) or IMAP(S) protocols.

SMTP Sampler: If you want to test a mail server, you can use the SMTP sampler. This sampler is used to send email messages using the SMTP protocol.

Listeners

Shows the results of the test execution. They can show results in different formats, such as a tree, table, graph, or log file.

You can see the most used Listener types below. The type of Listener you will use depends on what you are testing and the purpose of the test.

View Result Tree

View Results in Table

Summary Report

Thanks to the Listeners, we can see the connection time to the server “Connect Time”, the loading time of the page “Load Time”, the size of the HTTP response “Size in bytes”.

Configuration Elements

Configuration elements can be used to set up defaults and variables for later use by samplers.

CSV Data Set Config

Suppose you want to test a website for 100 users signing-in with different credentials. You do not need to record the script 100 times! You can parameterize the script to enter different login credentials. This login information (e.g. username, password) could be stored in a text file. JMeter has an element that allows you to read different parameters from that text file. It is “CSV Data Set Config”, which is used to read lines from a file, and split them into variables.

Jmeter_example.csv file content:

Username1;password1

Username2;password2

The appearance of the CSV Data Set Config component is shown below.

1. Name of the CSV file (actually relative path relative to the location of the JMX file)

2. Variable names are given to be assigned to the columns in the input file.

3. Which character is the separator between the columns in the CSV file.

The most important point to be aware of is that the JMX file where the JMeter Test Plan is saved and the CSV file must be in the same folder, or the relative path must be given according to the JMX file when configuring the CSV file.

HTTP Cookie Manager

Let’s understand this with an example. You used your browser (Firefox, IE…Etc) to browse www.google.com. You log in with your user and password. Your username and password will be stored on your computer as cookies. Next time, when you visit www.google.com, you don’t need to do log in again because your browser will use your cookies as user data to log in.

HTTP Cookie Manager also has the same feature as a web browser. If you have an HTTP Request and the response contains a cookie, the Cookie Manager automatically stores that cookie and will use it for all future requests to that particular website.

HTTP Request Default

This element lets you set default values that your HTTP Request controllers use.

For example,

You are sending 100 HTTP requests to the server google.com

You would have to manually enter server name = google.com for all these 100 requests

Instead, you could add a single HTTP request defaults with the “Server Name or IP” field = google.com

No need to type 100 times!

Assertions

Assertion helps verify that your server under test returns the expected results.

We will see this subject in practice in sample test writing. You can see the assertion types below.

Report Formats and Results Interpretation

JMeter supports the dashboard report generation to get charts and statistics from a test plan.

Open your command prompt and make sure that you are inside the bin folder before running your Load Testing Script.

Run the test in non-GUI mode and save the report as CSV or jtl format.

Windows:

The syntax for running the Script in Command prompt:

Jmeter –n –t [path to test JMX file] –l [path to result in file]

-n: This specifies Jmeter is to run in non-GUI mode

-t: name of JMX file that contains the Test Plan

-l: name of JTL file to log sample results.

Example:

Jmeter –n –t C:\jmeter\gen-report.jmx –l C:\jmeter\gen-report.jtl

Use the following command to generate a report from an existing sample CSV/JTL result file as above.

Windows:

JMeter -g [path to result file] -o [path to report output folder]

-g: [path to CSV file] generate report dashboard only

-o: output folder where to generate the report dashboard after the load testing. The folder must not exist or must be empty

Example:

JMeter -g C:\jmeter\gen-report.jtl -o C:\jmeter\report

After running the script you can open the folder which you have mentioned while running the script, you can see the report has been generated.

Open the Index.html file and you can see your load testing consolidate report.

The reports will contain detailed information about the request, response, pass and failure error, complete graph information.

Best Practices for Jmeter Tests

Jmeter has some limitations especially when it is run in a distributed environment. Following these guidelines will assist in creating a real and continuous load:

1. Do not use the Console to create your load. You can use the console for debugging purposes or to run a small load from it to make sure the script is running correctly. The GUI consumes a lot of memory under heavy load, therefore the console server by itself can not sustain a heavy load.

2. The limit for one load generator is 300/400 threads (Number of Users). Your hardware capabilities, as well as the Test Plan design, will both impact the number of threads you can effectively run with JMeter. The number will also depend on how fast your server is (a faster server makes JMeter work harder since it returns a response quicker).

3. Disable the “View Result Tree” listener as it consumes a lot of memory and can result in the console freezing or JMeter running out of memory. Add listeners appropriately. Keep Result Tree disable when run for a large number of users.

4. Disable all JMeter graphs as they consume a lot of memory.

5. Do not forget to add timers as per your requirement.

6. Always use a relative path for the CSV file if used in the script.

7. Make sure how to use the cookie manager and cache manager along with your requirement.

Advantages and Disadvantages of using JMeter

Advantages:

Performance Testing is crucial to determine that the web applications under test will satisfy high load requirements. It can be used to analyze overall server performance under heavy load.



Apache JMeter testing tool offers the following benefit in Performance Testing:



- JMeter can be used to test the performance of both static resources such as JavaScript and HTML, as well as dynamic resources, such as JSP, Servlets, and AJAX.



- JMeter can discover the maximum number of concurrent users that your website can handle



- JMeter provides a variety of graphical analyses of performance reports.

Disadvantages:

Memory Consumption: JMeter can simulate heavy load and visualize the test report. This may consume lots of memory and can lead to out of memory under heavy load.



Web applications only: JMeter is a good tool for testing web applications but it is not a suitable tool for testing desktop applications.



Lack of support for JavaScript: JMeter is not a browser, so it cannot run JavaScript in web applications. It has limited support for processing JavaScript or Ajax, this may affect the accuracy of the simulation.

Thank you for your interest. Hope it was useful. If you want more information and do practice about load testing with JMeter, just wait for the next post which would be quite useful.