Package: pytest-visual
pytest-visualpytest-visualintroduces the concept of visual testing to ML (with automated change detection!).Problem statementMost ML code is poorly tested. This is at least partially because a lot of ML code is very hard to test meaningfully.For example, we can easily test whether an image data augmentation pipeline produces an image with the correct shape, but does the image also look realistic? And does augmentation produce enough variation? These tests might succeed, but that doesn't mean that your code works.Solution - visual testing for MLpytest-visualaims to streamline the organization of visualizations in ML code, enhancing debugging, verification, and code clarity. It automatically detects changes in visualization outputs, presents them in your browser, and prompts for approval. Accepted changes are memorized and not prompted for again. As a result, your visualizationswill actually catch bugs, as changes won't go unnoticed.are always up to date, as they're run on everypytest-visualinvocation.won't clutter your production code, as they're separated into test cases.ExampleIn this example, we demonstrate how to create a basic visual test case. We first create a Python file with a prefixtest_*(eg.test_visualization.py), and add this function:# `pytest-visual` accepts `plotly` figuresimportplotly.expressaspx# `test_` prefix makes this a pytest test# `visual` argument makes this a visual testdeftest_show_square(visual):# Plot the test datax,y=[0,1,2,3,4,5],[0,1,4,9,16,25]fig=px.scatter(x=x,y=y)# You can have multiple print() and show() calls in a single case.visualize.print("Square plot")visualize.show(fig)Then runpytest --visualize, and open the url displayed in the terminal window (most likelyhttp://127.0.0.1:8050). If the visualization looks OK, click "Accept", and pytest should complete with success.Here's a sample output fromexamples/test_data_augmentation.py:SeeInstallationfor instructions on how to run these examples.Installationpytest-visualcan be installed with the following command:pipinstallpytest-visual--upgradeIn order to run examples in this repository, you also need development dependencies. You can install them withpipinstall-rrequirements-dev.txtThen go to the root directory of this repository and run visualizations withpytest--visualand open your browser at the printed url. Do note that development dependencies are quite heavy (for example Pytorch CPU), but that's just the nature of most DL frameworks.CLI usageWithpytest-visualplugin, the following options are added topytest:CommandDescriptionpytest --visualRun pytest with visual tests, and prompt user for manual review on all detected changed.pytest --visual-yes-allAccept everything without prompting. Useful when you clone a repository or check out a new branch.pytest --visual-reset-allMark all cases as declined.FAQCan I mix unit testing with visual testing?Yes. And this is exactly what you should be doing! Just omit thevisualfixture, and it's a normalpytesttest again.Which parts of a typical ML codebase can be visually tested?Visual testing can be used to verify the correctness ofdata loading and augmentationsynthetic data generationNN model struturesloss functionslearning rate schedulesdetection algorithmsvarious filtering algorithmsetc.My visual tests are flaky. Why?This is typically because your output is undeterministic. For example, plotting a loss over a short training run is probably not a good idea, as it will likely be different for each run. If you need it as a test, consider a normal unit test with some threshold for expected accuracy. Tiny floating point errors could lead to inconsistent output.That said, we're planning to makepytest-visualrobust to small fluctuations in output.Why even bother testing ML code?If you've implemented complex data processing logic, custom neural network architectures, advanced post processing steps, or tried to reproduce a paper, you know that bugs can take a very large part of the development effort. Reduced accuracy is often the only symptom of a bug. When conducting an ML experiment, it's typical to waste a lot of time ondebugging when your ML code contains a bug,debugging when your ML code doesn't contain a bug, but you don't know this,revisiting an idea, because you're not sure whether your experiment was correct,dropping a good idea because your experiment contained a bug, but you don't know this.Proper testing, including visual testing, can avoid a large portion of these headaches.Isn't automation the whole point of unit testing automation? Now we have a potentially lazy/irresponsible human in the loop, who can just blindly "accept" a test.Full automation is a of course a great, but even the tests that are automatically run are written by people. Test quality can vary a lot, and tests written by a lazy/irresponsible person probably won't catch many bugs. Just having unit tests won't magically fix your code, but they are a very efficient tool to enableresponsibledevelopers write reliable code.ContributingYour help is greatly needed and appreciated! Given that the library is still in alpha, there will likely be a lot of changes. Thus the main focus is to get the core features right and fix bugs.We encourage you to contribute by submitting PRs to improve the library. Usually, the PR should link to an existing issue, where the solution idea has been proposed and discussed. In case of simple fixes, where you can clearly see the problem, you can also directly submit a PR.To get started with development, install all the required dependencies for development:pipinstall-rrequirements.txt&&pipinstall-rrequirements-dev.txt&&pipinstall-e.The entry point for the plugin isvisual/interface.py, you should start exploring from that file. The code should be reasonably documented, and is quite short.Before submitting a PR, pleaseRun non-visual tests withpytestLint your code withpre-commit run -a, and ensure no errors persistVerify thatpytest-visualstill works end-to-end withpytest --visual-reset-all && pytest --visual, and try accepting and declining changes.Also, please verify that your function inputs and outputs are type-annotated unless there's a good reason not to do so.