Repository: sdv-dev/RDT (language: python)

Issue:
Create `validate_transformer_integration` function
A function should be implemented to automatically validate the Integration Tests of any new Transformer.

## Function Name and Module

The function should be implemented inside `tests/contributing.py` and should be called `validate_transformer_integration`

## Inputs

The function should accept a single input:

* `transformer (class or str)`: Transformer class or full Python name of the class (e.g. `DatetimeTransformer` or `"rdt.transformers.time.DatetimeTransformer"`)

## Outputs

The function should return a single `bool` value indicating whether the integration tests passed or not.

## Behavior

This function runs the integration test functions on the Transformer. The integration tests are automated (the user does not write them explicitly) and simply generate data using all the Dataset Generators that have been created for the Transformer data type and then run the Transformer on them in two ways:
* Directly, transforming and reverse transforming every dataset and validating the output types. If the Transformer has been flagged as deterministic, this also validates that it actually is.
* Using the HyperTransformer, going from the Transformer Data Type to Numerical and back.


### List of Checks

Check | Correct if
-- | --
Transformer defines data type | The Transformer correctly defines the data type that it supports
Dataset Generators exist | At least one Dataset Generator exists for the Transformer data type
Transform output types | The Transformer can transform data and produces outputs of the indicated data types
Reverse output type | The Transformer can reverse transform the data it produces, going back to the original data type
Composition is Identity | (only if the Transformer indicates so) Transforming data and reversing it recovers the original data
HyperTransformer can transform | The HyperTransformer is able to use the Transformer and produce float values
HyperTransformer can reverse transform | The HyperTransformer is able to reverse the data that has previously transformer and restore the original data type

### Prints to console

The function prints the following information in the console:

* The Transformer that is being validated
* Whether the Integration tests were successful or not
* Which checks from the Integration tests were correct (list above)

## Usage Example

![image4](https://user-images.githubusercontent.com/6129283/134695409-dccd180f-f5c7-494b-a9c3-b6d35bc5f7ec.png)


Discussion / hints:

Fix patch:
diff --git a/setup.cfg b/setup.cfg
index 087c87384..0290c9b69 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -35,8 +35,9 @@ universal = 1
 max-line-length = 99
 exclude = docs, .tox, .git, __pycache__, .ipynb_checkpoints, rdt/transformers/__init__.py
 inline-quotes = single
-extend-ignore = D107, SFS3
+extend-ignore = D107, SFS3, PD005
 per-file-ignores =
+    tests/contributing.py:T001
     tests/performance/profiling.py:T001
     tests/performance/test_performance.py:T001
 
diff --git a/setup.py b/setup.py
index ab889469d..61e340401 100644
--- a/setup.py
+++ b/setup.py
@@ -19,6 +19,7 @@
     'pandas>=1.1,<1.1.5',
     'scipy>=1.4.1,<2',
     'psutil>=5.7,<6',
+    'tabulate>=0.8.9,<1',
 ]
 
 copulas_requires = [


Test patch:
diff --git a/tests/contributing.py b/tests/contributing.py
new file mode 100644
index 000000000..1e4c6ad8b
--- /dev/null
+++ b/tests/contributing.py
@@ -0,0 +1,123 @@
+"""Validation methods for contributing to RDT."""
+
+
+import importlib
+import traceback
+
+import pandas as pd
+from tabulate import tabulate
+
+from tests.integration.test_transformers import validate_transformer
+
+# Mapping of validation method to (check name, check description).
+CHECK_DETAILS = {
+    '_validate_dataset_generators': (
+        'Dataset Generators',
+        'At least one Dataset Generator exists for the Transformer data type.',
+    ),
+    '_validate_transformed_data': (
+        'Output Types',
+        'The Transformer can transform data and produce output(s) of the indicated data type(s).',
+    ),
+    '_validate_reverse_transformed_data': (
+        'Reverse Transform',
+        (
+            'The Transformer can reverse transform the data it produces, going back to the ',
+            'original data type.',
+        ),
+    ),
+    '_validate_composition': (
+        'Composition is Identity',
+        (
+            'Transforming data and reversing it recovers the original data, if composition is ',
+            'identity is specified.',
+        ),
+    ),
+    '_validate_hypertransformer_transformed_data': (
+        'Hypertransformer can transform',
+        'The HyperTransformer is able to use the Transformer and produce float values.',
+    ),
+    '_validate_hypertransformer_reverse_transformed_data': (
+        'Hypertransformer can reverse transform',
+        (
+            'The HyperTransformer is able to reverse the data that it has previously transformed ',
+            'and restore the original data type.',
+        ),
+    ),
+}
+
+
+def get_class(class_name):
+    """Get the specified class.
+
+    Args:
+        class (str):
+            Full name of class to import.
+    """
+    obj = None
+    if isinstance(class_name, str):
+        package, name = class_name.rsplit('.', 1)
+        obj = getattr(importlib.import_module(package), name)
+
+    return obj
+
+
+def validate_transformer_integration(transformer):
+    """Validate the integration tests of a transformer.
+
+    This function runs the automated integration test functions on the Transformer.
+    It will print to console a summary of the integration tests, along with which
+    checks have passed or failed.
+
+    Args:
+        transformer (string or rdt.transformers.BaseTransformer):
+            The transformer to validate.
+    Output:
+        bool:
+            Whether or not the transformer passes all integration checks.
+    """
+    if isinstance(transformer, str):
+        transformer = get_class(transformer)
+
+    print(f'Validating Integration Tests for transformer {transformer.__name__}\n')
+
+    steps = []
+    validation_error = None
+    error_trace = None
+
+    try:
+        validate_transformer(transformer, steps=steps)
+    except Exception as error:
+        error_trace = ''.join(traceback.TracebackException.from_exception(error).format())
+
+        for check in CHECK_DETAILS:
+            if check in error_trace:
+                validation_error = str(error)
+
+    if validation_error is None and error_trace is None:
+        print('SUCCESS: The integration tests were successful.\n')
+    elif validation_error:
+        print('ERROR: One or more integration tests were NOT successful.\n')
+    elif error_trace:
+        print('ERROR: Transformer errored out with the following error:\n')
+        print(error_trace)
+
+    result_summaries = []
+    seen_checks = set()
+    failed_step = None if validation_error is None else steps[-1]
+    for step in steps:
+        check, details = CHECK_DETAILS[step]
+        if check in seen_checks:
+            continue
+
+        seen_checks.add(check)
+
+        if failed_step and step == failed_step:
+            result_summaries.append([check, 'No', validation_error])
+        else:
+            result_summaries.append([check, 'Yes', details])
+
+    summary = pd.DataFrame(result_summaries, columns=['Check', 'Correct', 'Details'])
+    print(tabulate(summary, headers='keys', showindex=False))
+
+    return validation_error is None and error_trace is None
diff --git a/tests/datasets/base.py b/tests/datasets/base.py
index 3aa90280c..e3a0c5a21 100644
--- a/tests/datasets/base.py
+++ b/tests/datasets/base.py
@@ -40,6 +40,7 @@ def get_subclasses(cls):
         return subclasses
 
     @staticmethod
+    @abstractmethod
     def get_performance_thresholds():
         """Return the expected threseholds."""
         raise NotImplementedError()
diff --git a/tests/datasets/boolean.py b/tests/datasets/boolean.py
index ef398d5e2..8783cb8f6 100644
--- a/tests/datasets/boolean.py
+++ b/tests/datasets/boolean.py
@@ -1,5 +1,7 @@
 """Dataset Generators for boolean transformers."""
 
+from abc import ABC
+
 import numpy as np
 
 from tests.datasets.base import BaseDatasetGenerator
@@ -8,7 +10,7 @@
 MIN_PERCENT = 20  # the minimum percentage of true or false is 20%
 
 
-class BooleanGenerator(BaseDatasetGenerator):
+class BooleanGenerator(BaseDatasetGenerator, ABC):
     """Base class for generators that generate boolean data."""
 
     DATA_TYPE = 'boolean'
diff --git a/tests/datasets/categorical.py b/tests/datasets/categorical.py
index c1ca5a615..434218b9c 100644
--- a/tests/datasets/categorical.py
+++ b/tests/datasets/categorical.py
@@ -1,5 +1,7 @@
 """Dataset Generators for categorical transformers."""
 
+from abc import ABC
+
 import numpy as np
 
 from tests.datasets.base import BaseDatasetGenerator
@@ -7,7 +9,7 @@
 from tests.datasets.utils import add_nans
 
 
-class CategoricalGenerator(BaseDatasetGenerator):
+class CategoricalGenerator(BaseDatasetGenerator, ABC):
     """Base class for generators that generate catgorical data."""
 
     DATA_TYPE = 'categorical'
diff --git a/tests/datasets/datetime.py b/tests/datasets/datetime.py
index ba254afb6..4b7a05235 100644
--- a/tests/datasets/datetime.py
+++ b/tests/datasets/datetime.py
@@ -1,6 +1,7 @@
 """Dataset Generators for datetime transformers."""
 
 import datetime
+from abc import ABC
 
 import numpy as np
 import pandas as pd
@@ -9,7 +10,7 @@
 from tests.datasets.utils import add_nans
 
 
-class DatetimeGenerator(BaseDatasetGenerator):
+class DatetimeGenerator(BaseDatasetGenerator, ABC):
     """Base class for generators that generate datatime data."""
 
     DATA_TYPE = 'datetime'
diff --git a/tests/datasets/numerical.py b/tests/datasets/numerical.py
index 73457e412..27a58b583 100644
--- a/tests/datasets/numerical.py
+++ b/tests/datasets/numerical.py
@@ -1,12 +1,14 @@
 """Dataset Generators for numerical transformers."""
 
+from abc import ABC
+
 import numpy as np
 
 from tests.datasets.base import BaseDatasetGenerator
 from tests.datasets.utils import add_nans
 
 
-class NumericalGenerator(BaseDatasetGenerator):
+class NumericalGenerator(BaseDatasetGenerator, ABC):
     """Base class for generators that create numerical data."""
 
     DATA_TYPE = 'numerical'
diff --git a/tests/integration/__init__.py b/tests/integration/__init__.py
index 2eaf4bddb..e34473071 100644
--- a/tests/integration/__init__.py
+++ b/tests/integration/__init__.py
@@ -1,1 +1,8 @@
 """RDT integration testing package."""
+
+
+from tests.integration.test_transformers import validate_transformer
+
+__all__ = [
+    'validate_transformer',
+]
diff --git a/tests/integration/test_transformers.py b/tests/integration/test_transformers.py
index dd9d2c79b..59f6f6bf9 100644
--- a/tests/integration/test_transformers.py
+++ b/tests/integration/test_transformers.py
@@ -13,6 +13,7 @@
 
 PRIMARY_DATA_TYPES = ['boolean', 'categorical', 'datetime', 'numerical']
 
+
 # Mapping of rdt data type to dtype
 DATA_TYPE_TO_DTYPES = {
     'boolean': ['b', 'O'],
@@ -24,6 +25,23 @@
 }
 
 
+def _validate_helper(validator_function, args, steps):
+    """Wrap around validation functions to either return a boolean or assert.
+
+    Args:
+        validator_function(function):
+            The function to validate.
+        args (list):
+            The args to pass into the function.
+        steps (list):
+            List of steps that the validation has completed.
+    """
+    if steps is not None:
+        steps.append(validator_function.__name__)
+
+    validator_function(*args)
+
+
 def _is_valid_transformer(transformer_name):
     """Determine if transformer should be tested or not."""
     return transformer_name != 'IdentityTransformer' and 'Dummy' not in transformer_name
@@ -63,9 +81,9 @@ def _find_dataset_generators(data_type, generators):
     return generators.get(data_type, [])
 
 
-def _validate_input_type(input_type):
-    """Check that the transformer input type is not null."""
-    assert input_type is not None
+def _validate_dataset_generators(dataset_generators):
+    """Check that the number of dataset generators is greater than zero."""
+    assert len(dataset_generators) > 0, 'There are no associated dataset generators.'
 
 
 def _validate_transformed_data(transformer, transformed_data):
@@ -74,8 +92,10 @@ def _validate_transformed_data(transformer, transformed_data):
     transformed_dtypes = transformed_data.dtypes
 
     for column, expected_data_type in expected_data_types.items():
-        assert column in transformed_data
-        assert transformed_dtypes[column].kind in DATA_TYPE_TO_DTYPES[expected_data_type]
+        message = f'Column {column} is expected but not found in transformed data.'
+        assert column in transformed_data, message
+        message = f'Column {column} is not the expected data type {expected_data_type}'
+        assert transformed_dtypes[column].kind in DATA_TYPE_TO_DTYPES[expected_data_type], message
 
 
 def _validate_reverse_transformed_data(transformer, reversed_data, input_dtype):
@@ -84,7 +104,8 @@ def _validate_reverse_transformed_data(transformer, reversed_data, input_dtype):
     Expect that the dtype is equal to the dtype of the input data.
     """
     expected_data_type = transformer.get_input_type()
-    assert reversed_data.dtypes[TEST_COL].kind in DATA_TYPE_TO_DTYPES[expected_data_type]
+    message = f'Reverse transformed data is not the expected data type {expected_data_type}'
+    assert reversed_data.dtypes[TEST_COL].kind in DATA_TYPE_TO_DTYPES[expected_data_type], message
 
 
 def _validate_composition(transformer, reversed_data, input_data):
@@ -93,9 +114,6 @@ def _validate_composition(transformer, reversed_data, input_data):
     This is only applicable if the transformer has the composition
     identity property.
     """
-    if not transformer.is_composition_identity():
-        return
-
     if isinstance(reversed_data, pd.DataFrame):
         reversed_data = reversed_data[TEST_COL]
     elif isinstance(reversed_data, np.ndarray):
@@ -110,10 +128,11 @@ def _validate_composition(transformer, reversed_data, input_data):
         check_dtype=False,
         check_exact=False,
         rtol=1e-03,
+        obj='Reverse transformed data and input data',
     )
 
 
-def _test_transformer_with_dataset(transformer_class, input_data):
+def _test_transformer_with_dataset(transformer_class, input_data, steps):
     """Test the given transformer with the given input data.
 
     This method verifies the transformed data's dtype, the reverse
@@ -124,6 +143,8 @@ def _test_transformer_with_dataset(transformer_class, input_data):
             The transformer class to test.
         input_data (pandas.Series):
             The data to test on.
+        steps (list):
+            List of steps that the validation has completed.
     """
     transformer = transformer_class()
     # Fit
@@ -131,30 +152,44 @@ def _test_transformer_with_dataset(transformer_class, input_data):
 
     # Transform
     transformed = transformer.transform(input_data)
-    _validate_transformed_data(transformer, transformed)
+    _validate_helper(
+        _validate_transformed_data,
+        [transformer, transformed],
+        steps,
+    )
 
     # Reverse transform
     out = transformer.reverse_transform(transformed)
-    _validate_reverse_transformed_data(transformer, out, input_data.dtypes[TEST_COL])
+    _validate_helper(
+        _validate_reverse_transformed_data,
+        [transformer, out, input_data.dtypes[TEST_COL]],
+        steps,
+    )
 
-    _validate_composition(transformer, out, input_data)
+    if transformer.is_composition_identity():
+        _validate_helper(
+            _validate_composition,
+            [transformer, out, input_data],
+            steps,
+        )
 
 
 def _validate_hypertransformer_transformed_data(transformed_data):
     """Check that the transformed data is not null and of type float."""
-    assert transformed_data.notna().all(axis=None)
+    assert transformed_data.notna().all(axis=None), 'Transformed data has nulls.'
 
     for dtype in transformed_data.dtypes:
-        assert dtype.kind in DATA_TYPE_TO_DTYPES['numerical']
+        assert dtype.kind in DATA_TYPE_TO_DTYPES['numerical'], 'Transformed data is not numerical.'
 
 
-def _validate_hypertransformer_reversed_transformed_data(transformer, reversed_data):
+def _validate_hypertransformer_reverse_transformed_data(transformer, reversed_data):
     """Check that the reverse transformed data has the same dtype as the input."""
     expected_data_type = transformer().get_input_type()
-    assert reversed_data.dtype.kind in DATA_TYPE_TO_DTYPES[expected_data_type]
+    message = f'Reversed transformed data is not the expected data type {expected_data_type}'
+    assert reversed_data.dtype.kind in DATA_TYPE_TO_DTYPES[expected_data_type], message
 
 
-def _test_transformer_with_hypertransformer(transformer_class, input_data):
+def _test_transformer_with_hypertransformer(transformer_class, input_data, steps):
     """Test the given transformer in the hypertransformer.
 
     Run the provided transformer using the hypertransformer using the provided
@@ -166,6 +201,8 @@ def _test_transformer_with_hypertransformer(transformer_class, input_data):
             The transformer class to test.
         input_data (pandas.Series):
             The data to test on.
+        steps (list):
+            List of steps that the validation has completed.
     """
     hypertransformer = HyperTransformer(field_transformers={
         TEST_COL: transformer_class.__name__,
@@ -173,10 +210,46 @@ def _test_transformer_with_hypertransformer(transformer_class, input_data):
     hypertransformer.fit(input_data)
 
     transformed = hypertransformer.transform(input_data)
-    _validate_hypertransformer_transformed_data(transformed)
+    _validate_helper(
+        _validate_hypertransformer_transformed_data,
+        [transformed],
+        steps
+    )
 
     out = hypertransformer.reverse_transform(transformed)
-    _validate_hypertransformer_reversed_transformed_data(transformer_class, out[TEST_COL])
+    _validate_helper(
+        _validate_hypertransformer_reverse_transformed_data,
+        [transformer_class, out[TEST_COL]],
+        steps,
+    )
+
+
+def validate_transformer(transformer, steps=None, subtests=None):
+    """Validate that the transformer passes all integration checks.
+
+    Args:
+        transformer (rdt.transformer.BaseTransformer):
+            The transformer to validate.
+        steps (list):
+            List of steps that the validation has completed.
+        subtests:
+            Whether or not to test with subtests.
+    """
+    input_data_type = transformer.get_input_type()
+
+    dataset_generators = _find_dataset_generators(input_data_type, generators)
+    _validate_helper(_validate_dataset_generators, [dataset_generators], steps)
+
+    for dg in dataset_generators:
+        data = pd.DataFrame({TEST_COL: dg.generate(DATA_SIZE)})
+
+        if subtests:
+            with subtests.test(msg='test_transformer_with_dataset', generator=dg):
+                _test_transformer_with_dataset(transformer, data, steps)
+                _test_transformer_with_hypertransformer(transformer, data, steps)
+        else:
+            _test_transformer_with_dataset(transformer, data, steps)
+            _test_transformer_with_hypertransformer(transformer, data, steps)
 
 
 transformers = _get_all_transformers()
@@ -194,17 +267,4 @@ def test_transformer(subtests, transformer):
         transformer (rdt.transformers.BaseTransformer):
             The transformer to test.
     """
-    input_data_type = transformer.get_input_type()
-
-    dataset_generators = _find_dataset_generators(input_data_type, generators)
-    assert len(dataset_generators) > 0
-
-    for dg in dataset_generators:
-        with subtests.test(msg='test_transformer_with_dataset', generator=dg):
-            try:
-                data = pd.DataFrame({TEST_COL: dg.generate(DATA_SIZE)})
-            except NotImplementedError:
-                continue
-
-            _test_transformer_with_dataset(transformer, data)
-            _test_transformer_with_hypertransformer(transformer, data)
+    validate_transformer(transformer, subtests=subtests)