How To Test A Chatbot?
By: Kate Zalozna
Tags: Software Testing, Chatbot Testing, Chatbots, QA

Just like any other software product, chatbots require thorough testing. A bot can communicate with lots of customers simultaneously, but the value of this communication will depend on a number of tech aspects and overall usability. So to make sure the chatbot satisfies your business expectations, a project team will need to work on well-thought-out business logic, accurate implementation, and provide professional QA services.

Frequent Chatbot Issues

Chatbots allow optimizing the work of customer support. Automating frequent queries helps to reduce the time of response and contact human specialists only in difficult cases. However, there are issues that undermine these plans and expectations. The most common issues are:

Broken scripts.

Long pauses before answers.

No connection to other business channels.

Too many tasks for one bot to cover.

Lack of utterance and/or accuracy.

Lack of margin for error.

Poor navigation and/or conversation design.

Testing Checklist: Features and Cases

Every chatbot will require a custom strategy based on project requirements. Nevertheless, I decided to break down chatbot testing and share several examples — a type of testing you may need to run and test activities that fall under this category.

Functional Testing

The purpose of functional testing is to validate the software behavior against the requirements. In other words, QA engineers look up into the project documentation and check if a chatbot behaves as planned. Some of the features we check at this stage are:

Slot filling. A chatbot needs to know the basic lingo of the niche it operates in. These entities are programmed at the early stages, before developing conversation scripts.

A chatbot needs to know the basic lingo of the niche it operates in. These entities are programmed at the early stages, before developing conversation scripts. Validating inputs. If a chatbot accepts inputs like email addresses, phone numbers, and zip codes, it is critical for the system to recognize a correct data format.

If a chatbot accepts inputs like email addresses, phone numbers, and zip codes, it is critical for the system to recognize a correct data format. Error handling. It is necessary to investigate the boundaries of the programmed logic and make sure the bot has decent responses to the inputs it doesn’t understand.

Test case #1

Run tests using domain-specific lingo to find out if the bot is familiar with the main entities. For example, if we have a coffee shop chatbot that lets users place orders, it should understand entities that denote a type of coffee, sugar options, a number of sugars, a customer’s name, etc. In other words, inputs like “A cappuccino with one sugar and a no sugar cold brew for Ben” should be clear for the bot.

Test case #2

Feed the chatbot email addresses, phone numbers, and zip codes in different incorrect formats. It can be an email address without @ symbol or with spaces in it, an incorrect number of digits in a zip code or a mobile number, etc. Thus, you will learn how the system accepts invalid data and what responses it shows when receiving incorrect data.

Test case #3

Run exploratory tests to learn how a chatbot handles non-standard inputs, like casual conversations, meaningless sentences, obsolete words, offensive messages, and different kinds of errors in the test.

GUI Testing

Chatbots often show buttons with response options, provide links to a brand’s website or other resources, have cards with embedded information or other images. All these elements should be functional — links lead to the correct sources, images optimized and displayed correctly, all the chains of responses connected so that a conversation doesn’t stop abruptly.

Test case #4

Check the entire communication flow to find broken links, graphic elements, etc., and also detect gaps in logic.

Compatibility Testing

Make sure that a chatbot looks properly on different devices, especially if it pops up on a website instead of a messenger app.

Test case #5

Check how a chatbot window looks on different devices, browsers, and OS versions. Resize a browser window on the computer screen to make sure that the graphics appear undistorted and the layout isn’t broken. Texts should remain readable and not overflow out of the boundaries.

UX Testing

Conversation flow is one of the most obvious aspects of a chatbot to check. UX testing is based on both business requirements and users’ personal experience. Make sure that a chatbot generates correct and accurate responses.

Test case #6

Run different queries mimicking the behavior of a user who sees your chatbot for the first time. The following questions will help to evaluate usability:

Does the chatbot understand the questions?

Does it provide instant responses to those queries?

Is the information relevant, accurate, and useful to a user?

Does a user journey require more or fewer steps to finish?

Is it engaging for users? Do they tend to abandon a chatbot?

How can we make the conversation more engaging?

Performance Testing

People using chatbots expect to get accurate responses instantly. Therefore, chatbot performance testing is essential. Make sure it takes up to 2–3 seconds for a chatbot to retrieve the response and display it.

Test case #7

Check different queries that cover small and large dialogues and feature different types of content. It should take the same time for each response to appear on the screen, regardless of its place in a user journey and featured content.

Integration/API Testing

As a rule, a chatbot doesn’t function on its own — it becomes an integral part of the company’s digital ecosystem. That is, you’ll eventually need to connect it to the official website, a social network account, messenger, application, or another digital medium.

Test case #8

If you use bot building software, you are likely to find an API that allows connecting a chatbot to your system and run API tests. If you use a custom solution, it is essential to run integration tests to check how a bot and other system features behave when connected.

Localization Testing

Last but not least, run the check for chatbot’s literacy and multilingual features. Avoid errors in responses — spelling, grammatical, lexical, any errors. If a website is multilingual, it makes sense to create a chatbot that is multilingual as well. Keep in mind that localization encompasses other aspects as well — time zones, currencies, metric systems, and more.

Test case #9

Check the responses for spelling, lexical, and grammatical errors, punctuation, and cohesion. Make sure the formatting is correct, too. If the chatbot is multilingual, it should be able to access the default language settings or start the conversation flow by specifying these details.

To Sum Up

These are just a few of the QA activities that will bring you closer to releasing a smart and functional bot. You can learn more about chatbot testing tips in our blog. Just remember: while it is relatively simple to inspect and improve the communication flow on your own, testing the user interface, response speed, error handling, and other tech aspects is a task that will suit better for a professional software testing company.