Testing Modules in Chainer

# Introduction

Thanks to many contributors, [Chainer](http://chainer.org) has been developed greatly. But as we review the PRs, we noticed there are limited guidelines on how to test the implementation. So those who tries to send PR must make test cases by imitating the existing test cases or read the code directly. In this article, I will briefly review the testing modules in Chainer and CuPy. As Chainer's testing modules are subset of CuPy's ones, I will focus on CuPy.

CuPy's (resp. Chainer's) testing tools are located in `cupy.testing` (resp. `chainer.testing`) and consists of the following modules (* indicates that Chainer also has this module in `chainer.testing`) :

- array
- attr (*)
- condition (*)
- helper
- hypothesis
- parameterized (*)

Note that this article is based on v1.5.0.3

# array module

`cupy.testing.array` module implements NumPy-like assertion functions. These functions accept both NumPy and CuPy ndarrays. The implemented assertions are as follows:

* assert_arrays_almost_equal_nulp
* assert_array_max_ulp
* assert_array_equal
* assert_array_list_equal
* assert_array_less

These assertions are used in **NumPy-CuPy consistency check decorators**  explained in `helper` module. As we use this decorators more often, we do not expect testers use these assertions directly (of course we can  choose to use them).

# attr module

`cupy.testing.attr` contains several decorators that enable/disable test cases and test fixtures[^1].

* `@attr.gpu` specifies that the test uses single GPU.
* `@attr.multi_gpu(N)` specifies this test requires N GPUs. `@attr.gpu` is equivalent to `@attr.multi_gpu(1)`
* `@attr.cudnn` specifies this test uses cuDNN module.

For more details, please see the [testing guideline](http://docs.chainer.org/en/stable/contribution.html#testing-guidelines).

[^1]: In this article, **test case** denotes the subclass of `unittest.TestCase`, while **test fixture** the method of the test case.

# hypothesis module

`cupy.testing.hypothesis` module contains hypothesis testing tools, to test statistical behaviors. For now, it has simple goodness-of-fit test with Peason's Chi-squared test only. We use it for testing random generator of ints like `cupy.randint` or `cupy.random_integers`.

# parameterized module

`cupy.testing.parameterized` module offers the standard way of parameterized tests. Basic usage is as follows.

```py:usage_of_parameterized_module.py
@testing.parameterize(
    {'height': 150, 'weight': 45},
    {'height': 180, 'weight': 80})
class BMITest(unittest.TestCase):
    def test_bmi(self):
        self.assertLessEqual(
             calculate_bmi(self.height, self.weight), 25.0)
```

This test calculates BMI(Body Mass Index) based on the height and weight and checks if it is less than the threshold. `height` and `weight` are parameters in this test. We can access them as attributes of the test case.

`parameterized` decorator automatically generates the test case for each set of parameters. The naming convention of generated test cases is `<original class name>_param_<n>` where `<n>` is the index number of the set parameters. In this example, `BMITest_param_0` and `BMITest_param_1` are generated. Note that original test case (`BMITest` in the example) is not executed.

We have an utility that makes the product set of parameter set. For example, `testing.product({‘a': [1, 2]}, {‘b’: [3, 4]})` is equivalent to `[{‘a’: 1, ‘b’: 3}, {‘a’: 1, ‘b’: 4}, {‘a’: 2, ‘b’: 3}, {‘a’: 2, ‘b’: 4}]`

# condition module

Decorators in `cupy.testing.condition` module customize the condition test fixtures are regarded as "success". For now, we have decorators for running test fixtures multiple times.

* `@attr.retry(N)` tries the decorated fixture N times and considers success if **at least one** of the trial is successful.
* `@attr.repeat(N)` tries the decorated fixture N times and considers success if **all** trials are successful.

Decorators abort the trials if we can judge the final result before we do execute remaining trials. If the decorated test fixture is considered failed, it shows the number of failed and successful trials and error message of first failed trial.

# helper module

`cupy.testing.helper` consists of some utility decorators for test cases and fixtures. Currently there are two types of decorators, namely, **NumPy-CuPy consistency check** and **parameterized dtype test**[^2]

[^2]: These names are not official.

### NumPy-CuPy consistency check

`cupy.ndarray` is designed so that most of its API (method name and arguments) is identical to corresponding ones in `numpy.ndarray`.  NumPy-CuPy consistency check decorators offer easy way to check the consistency of these APIs.

Take `testing.numpy_cupy_allclose` decorator for example. The typical usage is as follows (we modified slightly from the original code):

```py:usage_of_numpy_cupy_consistency_decorator.py
@testing.gpu
class TestFoo(unittest.TestCase):

    @testing.numpy_cupy_allclose()
    def test_mean_all(self, xp):
        a = testing.shaped_arange((2, 3), xp)
        return a.mean()
```

This test fixture checks `numpy.mean()` and `cupy.mean()` should be the same result. `xp` is an additional argument inserted by the decorator. It takes either `numpy` or `cupy`. Decorated function is required to return the same value[^3] even if `xp` is `numpy` or `cupy`. We can change the argument name from `xp` by `name` argument.

[^3]: Of course, the type of array module(`numpy`/`cupy`) is different the decorator checks `cupy.asnumpy(cupy_result)` is equivalent to `numpy`'s result.

### Parameterized dtype test

This kind of decorators makes decorated test fixture parameterized with respect to dtype. Of course, we can do the parameterized test with respect to dtype with `@testing.parameterized` decorator. But parameterized dtype test decorators offer easier way. Let's look at the example in CuPy test code.

```py:tests/cupy_tests/io_tests/test_npz.py
@testing.gpu
class TestNpz(unittest.TestCase):
...
    @testing.for_all_dtypes()
    def test_pickle(self, dtype):
        a = testing.shaped_arange((2, 3, 4), dtype=dtype)
        s = six.moves.cPickle.dumps(a)
        b = six.moves.cPickle.loads(s)
        testing.assert_array_equal(a, b)
```

This test fixture checks if `cPickle` successfully reconstructs `cupy.ndarray` for various dtypes. `dtype` is an argument inserted by the decorator as with the NumPy-CuPy consistency check decorator's case. We can change the argument name by `name` argument of the decorator.

Here is the correspondence table of decorators and dtypes to be checked.
 
||bool|float[16, 32, 64]|int[8, 16, 32, 64]|uint[8, 16, 32, 64]|
|-----|:-----:|:-----:|:-----:|:-----:|:-----:|
|`for_all_dtypes`|○|○|○|○|
|`for_float_dtypes`||○|||
|`for_signed_dtypes`|||○||
|`for_unsigned_dtypes`||||○|
|`for_int_dtypes`|○||○|○|

`numpy.bool_` and `numpy.float16` are optional. If `no_float16` (resp. `no_bool`) option set `True`, `numpy.float16` (resp. `numpy.bool_` ) is disabled.

##### combinatorial dtype test

Some test fixtures require parameterization with respect to the **product** of dtypes. Decorators named as `for_***_dtypes_combination`  offer this functionality.

```py:usage_of_combinatorial_dtype_test.py
@testing.gpu
class TestFoo(unittest.TestCase):
    @testing.for_all_dtypes_combination(dtypes=['a_type', 'b_type'], full=True)
    def test_foo(self, dtype):
         a = cupy.arange(10, dtype=a_type)
         b = cupy.arange(20, dtype=b_type)
         # (some assertions with a and b)
```

Let N be the number of dtypes and M be the number of values each dtype can take. This decorator exexutes N**M tests if `full` option is set `True`. In some case, this can be costly, so we only check selected M tests only if `full` is `False`. If `full` argument is not specified, the default behavior depends on the environment variable `CUPY_TEST_FULL_COMBINATION`.

We can (and in most case do) use both of NumPy-CuPy consistency check decorator and Parameterized dtype test decorator simultaneously. Here is the example of unittests for `cupy.mean`

```py:tests/cupy_tests/statics_tests/test_meanvar.py
    @testing.for_all_dtypes()
    @testing.numpy_cupy_allclose()
    def test_mean_all(self, xp, dtype):
        a = testing.shaped_arange((2, 3), xp, dtype)
        return a.mean()
```
Note that for implementation reason, Parameterized dtype decorators must be preceded by NumPy-CuPy consistency check decorators.

# Conclusion

As we have seen in this article, Chainer and CuPy have various kind of utility that make the test cases easier. We briefly review the function and basic usage of them. I hope this article helps to promote the user to contribute to Chainer.

After we finish writing the article, I noticed that I forget to write about `chainer.gradient_check`, which is one of the most important testing tool in Chainer. I will write it in another article (or I hope someone write about it).

This article resolves issue [#714](https://github.com/pfnet/chainer/issues/714) :)