Package: detest
DETestA Differential Equation Testing SuiteAlejandro Francisco QueirugaLawrence Berkeley National Lab2018This repository contains a set of testing problems with known analytical solutions or reference solutions from a "trusted" code: oracles.
It can generate a testing framework based on Python's unittest test will automatically compare your code to these oracles, making sure on a variety of problems that your code are exactly correct, an approximation that converges at the expected rate, or at least gives you the same answer you got yesterday.These tests were originally written to support various codes in the Earth and Environmental Sciences division at Lawrence Berkeley National Lab, and have been merged into this unifying framework.
The impetus was when I realized I kept copying and pasting the same Python files implementing analytical solutions, and hapharzardly implementing the same convergence testing.
When you write an incredibly complicated test yourself and it does not agree with your code, you don't know if the problem is with the test or your code.
This makes the tests themselves problematic: I and collaborators have had plenty of trouble deriving, researching, and even typing up the analytical solutions themselves.
Having a centralized testing framework not only makes iteasier to follow best practices, it alsovalidates the tests themselvesby cross referencing with multiple codes:Installationdetest is compatible with both Python 2 and 3. Numpy is required, and
some of the oracles make use of Scipy and Sympy. Install it from PyPI,pip3installdetestor clone it and use setuptools,python3setup.pyinstall--prefix=/path/to/install/toand then import it withimport detest.IntroductionTesting numerical scientific and engineering codes has a further challenge than normal software.
The results are inexact and the expected behavior can be unknown.
When we see something wrong with out codes, we don't know if it's a problem with:a bug in the code?a failure of the numerical model?a fundamental problem with the underlying theory?or incorrect expectations?We address bullet point #4 by providing a trusted set of analytical solutions to be oracles in this library.
The oracles (their theory, mathematical solution, and implementation) are verified by being in agreement with multiple codes!
We can use these oracles and an understanding of the behavior we expect to see from our codes to help us diagnose issues #1-#3.Unit Tests - make sure the code worksKnown Analytical Oracles - tests with known solutions. The code could either get these solutions exactly correct, or only approximates the solution approach at an expected convergence rate.Self-Similarity Convergence - Problems without known solutions, but the testee should behave consistently with itself.Reference Tests - tests with no known solutions, but we compare the codes to an over-discretized trusted code, or experimental data.The philosophy I have evolved has the following phases:Unit TestsThese are software tests.
These tests are for things on the order of "Does the code read an input file correctly?"
This repository doesn't deal with them: they're unique to the codebase.Oracle-based Exact Precision TestsThe methods we use to solve problems should be able to get certain solutionsExactly Correct.These can be done in the same unit testing framework. They should be cheap.
This library will generate a unittest object for requested exact precision tests.Oracle-based Convergence TestsThese problems should also have the property that they are Lipschitz continuous; i.e. the numerical problem should converge smoothly.These require more computational effort to run.
There is some advice for designing simple problems that the testee should be able to execute extremely quickly.This library will generate a unittest object for requested exact precision tests.for an expected rate.Self-Similarity Convergence TestsWe assume self similarity.where $H^*\ll h$.Reference TestsWe can construct a reference solution by saving the solutions of a "golden code" in the database, and then comparing future codes to it.Regression TestsRunning all of the above tests can be costly.
Most of our everyday programming shouldn't effect the results for most of the codes capabilities.
After a set of changes, we should assertfor every problem we don't think we changed.
These can be very fast and short, and randomly generated every day.DETest will eventually have a class that runs these tests and maintains this history database.What makes a good test?What happens when one of these tests isn't Lipschitz continuous? What if the physics itself has bifurcation points?
How do we verify a code that is supposed to be solving a nonsmooth problem?
These are hard problems to solve.
A strategy is to examine ergodic properties, such as the center point of an attractor, the total amount of gas in the reservoir, the granular temperature of the system, etc.
These have to be coupled by making your code solve the simpler problems, too.
Currently, we have no such tests in this package.The TestsRight now, the focus is on problems in subsurface flow and
mechanics. The complete list of oracles isConstant strain modes in elasticityConstant flux flowTerzaghi's consolidationde Leeuw's consolidationThin crack in tension / under pressurePoisson problemRadial production in a poroelastic systemNo "gold standard" datasets for reference problems have been included yet.The problems in the library each have their own module.
Each module has a description, default parameter dictionary, and at least one class that implements the solution.
The class is meant to behave like a closure, where it is initialized with a parameter set and is callable.
Each closure evaluates the analytical solution at a point that is some combination of $(x,y,z,t)$ for the given parameters.
The module will have a default set of parameters.The units for default properties are all in SI.
This strictly doesn't matter, but for some codes the properties are not inputs, but instead intrinsic to the theoretical formulation.Using the Test SuiteWrapping the codeThe developer needs to wrap their code into a Python script with one function call for each oracle.
The routine for your code takes two arguments: a dictionary of parameters, and an 'h' argument for discretization scale.
The parameters a specific to the oracle problem (e.g. domain size, conductivity, etc.); read the description of the oracle.
To check mesh refinement, 'h' will be a grid spacing or a timestep size. (I sometimes make it control both.)
Detest will automatically call the script with a set of parameters with many different 'h's.
The output is a dictionary that matches the field names of the oracle, plus an additional field 'points'.For a command line code, this will look something like this:defmyScript(parameters,h):make_tough_input(h,parameters)sp.call(['TH','millstone_input.py',str(h)])x,U,P=process_results_for_testing()return{'U':U,'P':P,'points':x}where the developer is responsible for autogenerating their input files.Making an automated suiteDetest will autogenerate a unittest suite.
A snippet this short will populate the unittest framework:suite=[detest.ExactTest(detest.oracles.Uniaxial,myUniaxial),detest.ExactTest(detest.oracles.Shear,myShear),detest.ConvergenceTest(detest.oracles.Terzaghi,myTerzaghi,1),]MyTestSuite=detest.make_suite(suite)The file can then be executed with the unittest module,python-munittestMyTestSuite.pyThe power of this architecture is that list can be generated with a loop.
An example of this is inafqsrungekutta,
wherein a seperate ConvergenceTest for every each tableau is made.Just using the oraclesDetest is also a library of analytical solutions.
One can just import the solutions if that's useful,importdetestf=detest.oracles.Terzaghi({'k':2.0e-12})displacement=f([[5.0,10.0]])['U']This is particularly useful for applying truncated boundary conditions for some of the convergence tests.Computation Expense ConcernsRunning numerical tests is expensive in terms of computing time, which is also a dollar-cost.
There are different strategies to minimize the cost and enable real-time continuous integration:Only test randomly with frequency, and save the rigorous-churn through tests for weekly tests.Use a scheduling environment to run tests in parallel on commodity resources.Schedule them for low-priority queues at a low-cost off-hours.LicenseCopyright (C) Alejandro Francisco Queiruga, 2015-2018Lawrence Berkeley National LabDETest is released under version 3 of the GNU Lesser General Public License, as per LICENSE.txt.AcknowledgementsThis library was developed to support various projects in the Earth and Environmental Sciences division at Lawrence Berkeley National Lab.