Testing and the Future of NinjaChat
By: Dexter Fong
Tags: Chatbots, Machine Learning, Technology, Software Development, Ninja Van

A Brief Note on Testing our Integration

Building something entirely from scratch means we also need assurances that it would work as intended when we finally deploy. This is especially so for an application based on a conversation tree where we should strive to cover as many possible pathways as possible. As such, we wanted a good foundation that covers testing the application at different levels.

Unit Testing

One advantage of writing our intent service classes such that each matched intent carries an action value which executes a fixed handler method, is that writing units for them involves testing solely the handler method and expecting a given output in the form of a DialogflowQueryResponseBean . Here’s an example that tests for an error response when one of our internal services throw an exception:

@Test

public void givenNoOrders_whenHandleActionY_thenProcessAction() {

//Arranging the error from an internal api call

when(orderApi.searchForOrders(/* ... */))

.thenReturn(completedFuture(Optional.empty())); public void givenNoOrders_whenHandleActionY_thenProcessAction() {//Arranging the error from an internal api callwhen(orderApi.searchForOrders(/* ... */)).thenReturn(completedFuture(Optional.empty())); //Asserting on the response as an error bean

assertErrorBean(intentService.processMatchedIntent(

buildCustomerRequest(DEFAULT_SYSTEM_ID),

buildQueryResult(CUSTOMER_FETCH_ORDERS_FOR_ISSUES_SEARCH)),

DF_FETCH_ORDERS_FAILURE.getKey()));

}

*Note: buildQueryResult() mocks a matched intent result from Dialogflow with the given action CUSTOMER_FETCH_ORDERS_FOR_ISSUES_SEARCH . DF_FETCH_ORDERS_FAILURE refers to the error message key shown to the user.

Dialogflow Integration Testing

Naturally, once we’ve made sure that the individual handlers for each matched intent action behave as expected (i.e. returns the right response to the user, makes proper api calls, and parses responses from intermediate calls appropriately) via unit tests, we also use an inhouse framework that allows us to test the chatbot experience from a more integrated, user-relevant perspective.

What this means is we run integration tests that tests on the level of the main Dialogflow service class. This class is responsible for accepting a query string from a given user, converting it to a Dialogflow request bean, processing it and churning out a Dialogflow response bean, and finally returning the response as per what the user expects to see on any given platform.

Here’s a sample of one such test:

@Test

public void givenTrackOrder_whenUserChatsWithBot_thenProcessConversation() {

AbstractModule customModule = configureCustomerProfileModule(

new DialogflowCustomerIntegrationTestProfile()

.buildDefaultProfile()); public void givenTrackOrder_whenUserChatsWithBot_thenProcessConversation() {AbstractModule customModule = configureCustomerProfileModule(new DialogflowCustomerIntegrationTestProfile().buildDefaultProfile()); DialogflowConversationContainer con =

newConversation(customModule, CUSTOMER); con.assertThat(whenUserSays("hello there")

.thenBotRepliesWith("Thank you for using Ninja Van!"

+ "How can I help you today?")

.andShowsOptions("Track Orders",

"Reschedule Delivery", "Other Options")); con.assertThat(whenUserSays("i want to track my order")

.thenBotRepliesWith("Here is a list of your orders...")

.andShowsOptions("NVSGD123456",

"NVSGD135790", "NVSGD246810", "Return to Main Menu")); /* ... and so on */

}

This test covers a series of interaction starting from the user saying hello to our bot, to tracking their order and finally returning back to the main menu.

Stuff to note: