ufz/ogs — pull request #400 by bilke (2014-04-15 09:14:50+00:00)
Title: Executable testing and test data integration

# Test Wrapper Introductiion

As proposed by Dima an executable test can be run with several wrappers, e.g. with `valgrind` or `memcheck`. Each wrapper run can then be validated, e.g. with file-`diff`s or -`grep`s. This can be configured in CMake:

```cmake
INCLUDE(test/AddTest)
AddTest(SimpleGroundWaterFlow ${ExternalData_SOURCE_ROOT}/GroundwaterFlow_Quad4/288Cells case TIME DIFF)
AddTest(SimpleGroundWaterFlow ${ExternalData_SOURCE_ROOT}/GroundwaterFlow_Quad4/288Cells case MEMCHECK)
AddTest(SimpleGroundWaterFlow ${ExternalData_SOURCE_ROOT}/GroundwaterFlow_Quad4/288Cells case CALLGRIND)
```

Where arguments are:

1. executable
2. test data directory
3. test case name
4. wrapper name
5. checker (validator) name, optional

Tests are then run with

`make ctest`

If the checker has some errors they are displayed.

The functionality is very flexible and more wrappers and checker can be added later on. e.g. for running some statistics on output files and comparing them with statistics from reference files.

The testing functionality has not been fully adapted to the test data integration because of a missing OGS simulator executable.

# Test Data Introduction

For managing test data files Dima and I evaluated some options but in my the functionality provided by CMake (in the module [ExternalData](http://www.cmake.org/cmake/help/v3.0/module/ExternalData.html)) is really nice:

- In the source tree there placeholder files representing the test data (see 3a92c8d). The files have the same name but an additional `.md5` at the end. The content of a file consists of a MD5-hash of the original file.
- The files can then be referenced in CMake when tests are added (see `scripts/cmake/test/Data.cmake` in ab8f000, a real example will be added once the OGS simulator executable is in the official repository)
- The original data files are automatically retrieved by CMake when tests are run
- The files are renamed based on their MD5 hash
- The files are automatically downloaded from a web server (at the moment this is `http://www.opengeosys.org/images/dev/MD5/`)
- Retrieved files are stored in `ogs-source-dir/../ogs6-data/MD5/`

This approach has some really nice benefits:

- Large files e.g. a meshes which is used by several tests exists in the source tree several times but gets downloaded only once and is also stored locally only once (in the build tree severak symlinks get created which point to the actual file)
- The same is true for files which differ in their filename but not in their content
- Everything is nicely integrated into CMake
- Adding new test files is relatively easy (see below)
- Test data can be shared between several build configurations and source trees

## Adding new test files

- Just put the new files into the source tree under `Tests/Data/`
- Reference these files in some tests (in CMake)
- Run CMake
- This automatically generates the `*.md5` reference files
- Build the `move-data`-target, e.g. `make move-data`
- This moves the original files into your `ogs6-data`-directory and renames them based on their hash
- Upload these files to the webserver (how to do that has to be discussed)
- Add the new (`*.md5` reference) files in `Tests/Data/` to the git repository

----

Once merged I will move this documentation over to http://docs.opengeosys.org/docs/devguide/testing/test-data.