Software Testing and Machine Learning | by Scott Lindeman | Towards Data Science

Let’s face it, machine learning (ML) is becoming a standard part of many software systems. A trained model in your system may be surfacing predictions directly to users to help them make a human decision, or it may be making automatic decisions within the software system itself. Whether the ML in a system is developed in-house or is retrieved from a 3rd party’s pre-trained model API, if production software utilizing a trained model’s predictions is being used, it needs to be rigorously tested like you would any other aspect of your software. In this article, we will go over a rule-of-thumb framework on how to test your software when ML is involved, and identify what some common traps could be.
Author’s Note: Testing and change management in ML is a massive subject. This article is not intended to help evaluate how robust a model’s accuracy and performance is through testing, but to understand a model’s prediction interface and behavior. That said, you may find common model shortcomings and limitations through testing your software system in this manner.
Before we dig into how the introduction of ML changes systems, let’s quickly discuss the reasons for testing a software system. Testing helps developers ensure the behavior of a system is working as specified. Programs and software are constantly changing systems and if there are no automated tests that catch changes in behavior due to any number of reasons, systems are prone to errors, failures and bugs.
That said, what do we mean by testing? While there are many ways to unit test a piece of code, one common way to think about testing is through invariants. What universal truths about a function can we test?
def is_above_threshold(value):    return value > THRESHOLD
Here is a very basic example of a function we can test. Three test cases that immediately come to mind are checking a number below THRESHOLD, a number above THRESHOLD and the THRESHOLD value itself.
def test_over_threshold():    assert is_above_threshold(THRESHOLD + 1)def test_under_threshold():    assert not is_above_threshold(THRESHOLD — 1)def test_threshold_edge():    assert not is_above_threshold(THRESHOLD)
Great. Now because this is Python, we may also want a few tests that try a few different input types. Do we accept floats, ints, numpy floats, infinity? Those examples should be fairly straightforward. The only time we should have to change these tests is if the invariant of the function changes. Say we change > to >=. That will cause one or more of these tests to fail because the invariant we were testing against changed.
However, we know that this isn’t all the testing that is needed for a software system. Systems are more than just one or two Python functions calling each other; they often consist of large interconnected layers of functions, which talk to other software systems over a network protocol. To handle that level of complexity, we need integration testing.
Like unit testing, different developers have different definitions of what exactly integration testing entails. For the purposes of this example, we want to test that our code works with another component of our system. This component could be reading from a database, or calling a rest API and getting information back. Here’s a simple example using our is_above_threshold function from earlier:
def make_decision(user_input):    collected_value = component.get(user_input)    if is_above_threshold(collected_value):        return “Do it!”    else:        return “Don’t do it!”
What are the attributes of this function we want to test?
First and foremost, we want to make sure that we are using component.get’s interface correctly. We will need to provide enough different inputs to make_decision to exercise the various outputs of component.get, ensuring we haven’t missed any possible outcomes of its interface. Some examples of this:
Given the documentation, what values can be passed in as input? Just like our unit test example, try out edge cases to capture the behavior of non-normal situations.
Will component.get raise an error if given bad input? If so, do we handle that in this function or raise it to the caller of make_decision?
What if component.get sometimes returns a non-numeric value? In a more complex case, what if component.get returns a different data type depending on the input being passed in?
These are all valid tests we can perform because we are testing if our code is prepared for component.get’s interface, but not digging any deeper than we have to into the function’s implementation.
We also want to test that our code is working correctly. To do that, we can take a few shining examples of user_input and make sure we get the expected decision out at the end.
In most software systems, when we are using a library or component, we can expect that the component has gone through its own suite of invariant and integration testing. Therefore, we should only have to test the validity of the intersection.
What would cause these tests to change? If we updated the code in any way that altered the outcome for some of our shining examples of user_input, then we would have to update the tests to accommodate for the new behavior.
Now say we update the major version of the library component comes from and the interface has changed. The get function is no longer the method we want to use. We don’t want the behavior of our make_decision function to change. We can alter make_decision to accommodate the new API and, if we wrote the tests correctly, they should not have to change.
Now that we’ve gone through some testing basics, the real question is what is different when we add ML to our software system?
Right off the bat, let’s make it clear that many of the libraries used to build ML models are well tested. [1] However, using ML in your software system is usually not using those well tested library functions directly, but an artifact that was created from the library, your trained model.
When your code calls model.predict, you have assurances that all of the layers of methods and functions calling each other work at an invariant level, but you do not have any assurance that the library knows what the data you feed the model looks like. In our invariant testing example earlier, we spoke about testing for variable types like floats, ints, inf, etc. The same applies to testing a trained model, but the data types may be much more complex. For example, the number 3 is much simpler to test than a categorical feature that has 30 levels. So, does the three to eight tests we developed earlier for invariant testing hold up when we are talking about a much larger set of input data? Not quite.
A trained ML model is much more complex than our earlier example of comparing two numbers with the > sign. What invariants should we hold true when making predictions with a trained model artifact then? Let’s start with some basics:
The model’s predictions should be deterministic. That means when I pass in a single, compliant row of data for a prediction, I should get the same prediction back every time. Similarly, prediction consistency should hold true when making single row predictions and batch predictions. For example, the prediction for row 3 should be the same whether row 3 is alone or in batch with rows 1–10.Extending from a single compliant row, we should be able to reproduce the same error metric score on the same test data used to evaluate the model. Ignoring the fact whether the metric score is good or not, we want to be able to test that it doesn’t change.The model should make predictions under a certain amount of time. Some complex input data may cause a model to take longer to make a prediction than less complex inputs, but there should be an upper limit you can measure.
The model’s predictions should be deterministic. That means when I pass in a single, compliant row of data for a prediction, I should get the same prediction back every time. Similarly, prediction consistency should hold true when making single row predictions and batch predictions. For example, the prediction for row 3 should be the same whether row 3 is alone or in batch with rows 1–10.
Extending from a single compliant row, we should be able to reproduce the same error metric score on the same test data used to evaluate the model. Ignoring the fact whether the metric score is good or not, we want to be able to test that it doesn’t change.
The model should make predictions under a certain amount of time. Some complex input data may cause a model to take longer to make a prediction than less complex inputs, but there should be an upper limit you can measure.
Models make predictions, but they should be consistent about their predictions. Much like people, unless the model learns something new, it can only produce a result with currently known information. There is a time and place for online machine learning, but I will not be going in depth on this topic for this article.
In order to test consistency, we need our test cases to look at many more types of input data. Consider using the entire out-of-sample set of data that was used to evaluate the model. This is data that was not included when training the model, but has actual outcomes that you can compare against model predictions. To ensure the model is deterministic, we won’t be comparing the model’s predictions with actual outcomes, but with the original predictions this model made on the same set of data. If you make predictions on this data every time your tests run, whether as part of automation on merge requests or your integration pipelines, you will ensure that the model is producing consistent predictions.
Maybe you are worried your original out-of-sample data doesn’t cover enough of the potential inputs that could feed into your model. You can calculate the total number of rows needed to exhaustively test your model. Let’s look at a toy example:
Say your model was trained on five features with 1000 rows of data. Three of those features were categorical with 5, 10 and 30 levels respectively. The other two columns were numeric, with 500 unique values each. To calculate the total number of rows to exhaustively test the prediction behavior of your model on every combination of data, you would need to provide 5 * 10 * 30 * 500 * 500 = 375 million rows. This level of testing would tell you exactly how your model would behave in every possible situation, but only for the data the model has learned.
The exhaustive test above is impractical for the CI pipelines of a software system. If this model could make a single prediction in one tenth of a second, the test would still take more than ten thousand hours to complete. This is ignoring the hardware and infrastructure needed to load, predict and assert that large amount of data.
With the out-of-sample data and their outcomes, you have a good sample of data you can test your model on, as this was the criteria the model builders and evaluators used for evaluation purposes. It should be enough to catch invariant changes in the model. In this case, the tests should change to accommodate the invariant change. Do not forget to test “edge” cases such as null values and numeric data or categories that were not part of the training data for the model.
If a library or dependency upgrade in your system causes predictions to change, these tests will also fail. However as we mentioned earlier, generally library and dependency upgrades should not affect the external behavior of our system, so likely the model code will need to be altered in a way to ensure consistent predictions in this case. There are also many testing considerations around the storage format of trained models and their runtime environment. Some storage formats, like pickle, can cause runtime errors when trying to load the model into memory if the environment dependencies are updated. [2]
We’ve gone over some very basic invariants needed to test a software system that uses ML models. At the integration level, we luckily can keep some of the same concepts we had earlier. Let’s see what our make_decision function looks like if we replace component.get with model.predict:
def make_decision(user_input):    prediction = model.predict(user_input)    if is_above_threshold(prediction):        return “Do it!”    else:        return “Don’t do it!”
The model.predict call functions analogously to component.get, taking in data from a user and producing a value we need to evaluate. To satisfy integration testing, you will similarly need to test that you are covering the model.predict interface correctly, catching or raising errors when necessary. If we know that a null feature in one value will cause predict to raise an error, test that case and have your function handle it accordingly. Again provide a few shining examples of input into the system.
We’ve reviewed some basic concerns you need to take into consideration when adding ML to your system. This initial suite of tests is a reasonable baseline, but there is much more you could do. [3]
For example, monitoring your model after it has been deployed. Production software systems do require monitoring to ensure they are working as intended. While our tests assert that the model’s behavior doesn’t change, we cannot test how the real world inputs to our model are changing and causing poor predictions to be made. This is an important step in model change management.
Testing the prediction interface and behavior of a model will ensure developers understand a model’s behavior and keep systems bug-resistant. ML models will be used more, so common best practices of how to test them will be critical for future software development.
[1] Not all libraries are perfect. You’ll eventually encounter bugs and find the limitations of a library’s interface. In these cases it is a good opportunity to find the repository, look through issues to see if others have found the bug, offer a minimal reproducible case and possibly suggest a fix.
[2] See documentation on pickling
[3] D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Franc ̧ois Crespo, Dan Dennison, Hidden Technical Debt in Machine Learning Systems (2014), NIPS’15