Repository: sdv-dev/RDT (language: python)

Issue:
Addons implementation
### Expected behavior

We want to create different families of transformers and those to be optionally installed. 

### Aproach

Using [Namespace Packaging](https://packaging.python.org/guides/packaging-namespace-packages/) we will be able to achieve multiple packages within the `RDT` project and those to be installed afterwards as addons and only be installed if the user actually wants them.

#### Structure

The `addons` folder for the multiple families should be created within the `rdt/transformers/` without a `__init__.py`  file inside it. This folder will contain multiple folders representing a family which will become a `package`. 

```
rdt
├── hyper_transformer.py
├── __init__.py
└── transformers
    ├── addons
    │   ├── digital
    │   ├── geo
    │   ├── time
    │   └── web
    ├── base.py
    ├── boolean.py
    ├── categorical.py
    ├── __init__.py
    ├── null.py
    └── numerical.py
```

Each family folder will contain a `__init__.py` file, a `config.json` file and the python modules that contain code for this addon.

```
addons/time/
├── config.json
├── datetime.py
└── __init__.py
```

#### Config file (JSON)

The config file for each addon has the following structure:
* `name`: The name of the package / library. This is usually `rdt_<family_name>`. **Required**.
* `transformers`: A list of transformers that can be imported from this family. **Required**
* `requirements`: A list of requirements for that package. Only required if there are any requirements needed to be installed.

```
{
    "name": "rdt_time",
    "requirements": [
        "maya"
    ]
    "transformers": [
        "rdt.transformers.addons.time.datetime.DatetimeTransformer",
        "rdt.transformers.addons.time.datetime.SplitDatetimeTransformer",
        "rdt.transformers.addons.time.dummy.DummyTimeTransformer"
    ]
}
```

#### Changes to the current code

##### `setup.py`
The `setup.py` file has to be adapted to:
* Read all the `yaml` dependencies so those can be installed for testing environment.
* Use `find_namespace_package` instead of `find_packages`, and specify what to be found (change the `rdt.*` for `rdt.transformers` ).
* Add `yaml` dependency.

##### `rdt/transformers/__init__.py`
The `__init__.py`  must dynamically detect and import the installed `addons`. This is done by reading if found any `config.yaml` file on the `addons/` folders.

The best aproach for this is by using `importlib`.

#### Multiple Package Deployment

A dynamic way of deploying the packages to PyPI by using the `config.json` file for each family must be created for easy maintanance and release process.


Discussion / hints:

Fix patch:
diff --git a/Makefile b/Makefile
index 2bf86285d..b4dd67619 100644
--- a/Makefile
+++ b/Makefile
@@ -146,6 +146,7 @@ serve-docs: view-docs ## compile the docs watching for changes
 dist: clean ## builds source and wheel package
 	python setup.py sdist
 	python setup.py bdist_wheel
+	python rdt/transformers/addons/addons_setup.py
 	ls -l dist
 
 .PHONY: publish-confirm
diff --git a/rdt/transformers/__init__.py b/rdt/transformers/__init__.py
index 7327e0dbe..a1e27330e 100644
--- a/rdt/transformers/__init__.py
+++ b/rdt/transformers/__init__.py
@@ -1,7 +1,11 @@
 """Transformers module."""
 
+import importlib
+import json
+import os
 from collections import defaultdict
 from functools import lru_cache
+from glob import glob
 
 from rdt.transformers.base import BaseTransformer
 from rdt.transformers.boolean import BooleanTransformer
@@ -11,6 +15,28 @@
 from rdt.transformers.null import NullTransformer
 from rdt.transformers.numerical import GaussianCopulaTransformer, NumericalTransformer
 
+
+def _load_addons():
+    addons = []
+    addons_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'addons')
+    for addon_json_path in glob(f'{addons_path}/*/*.json'):
+        with open(addon_json_path, 'r', encoding='utf-8') as addon_json_file:
+            transformers = json.load(addon_json_file).get('transformers', [])
+            for transformer in transformers:
+                transformer = transformer.split('.')
+                module = '.'.join(transformer[:-1])
+                module = importlib.import_module(module)
+
+                transformer = transformer[-1]
+                transformer_class = getattr(module, transformer)
+                globals()[transformer] = transformer_class
+                addons.append(transformer)
+
+    return addons
+
+
+ADDONS = _load_addons()
+
 __all__ = [
     'BaseTransformer',
     'BooleanTransformer',
@@ -21,13 +47,13 @@
     'NullTransformer',
     'OneHotEncodingTransformer',
     'LabelEncodingTransformer',
-]
-
+] + ADDONS
 
 TRANSFORMERS = {
     transformer.__name__: transformer
     for transformer in BaseTransformer.get_subclasses()
 }
+
 DEFAULT_TRANSFORMERS = {
     'numerical': NumericalTransformer,
     'integer': NumericalTransformer(dtype=int),
diff --git a/rdt/transformers/addons/README.md b/rdt/transformers/addons/README.md
new file mode 100644
index 000000000..6630cbd85
--- /dev/null
+++ b/rdt/transformers/addons/README.md
@@ -0,0 +1,31 @@
+<p align="left">
+  <a href="https://dai.lids.mit.edu">
+    <img width=15% src="https://dai.lids.mit.edu/wp-content/uploads/2018/06/Logo_DAI_highres.png" alt="DAI-Lab" />
+  </a>
+  <i>An Open Source Project from the <a href="https://dai.lids.mit.edu">Data to AI Lab, at MIT</a></i>
+</p>
+
+[![Development Status](https://img.shields.io/badge/Development%20Status-2%20--%20Pre--Alpha-yellow)](https://pypi.org/search/?c=Development+Status+%3A%3A+2+-+Pre-Alpha)
+[![PyPi Shield](https://img.shields.io/pypi/v/RDT.svg)](https://pypi.python.org/pypi/RDT)
+[![Unit Tests](https://github.com/sdv-dev/RDT/actions/workflows/unit.yml/badge.svg)](https://github.com/sdv-dev/RDT/actions/workflows/unit.yml)
+[![Downloads](https://pepy.tech/badge/rdt)](https://pepy.tech/project/rdt)
+[![Coverage Status](https://codecov.io/gh/sdv-dev/RDT/branch/master/graph/badge.svg)](https://codecov.io/gh/sdv-dev/RDT)
+
+<img align="center" width=40% src="docs/images/rdt-logo.png">
+
+* Website: https://sdv.dev
+* Documentation: https://sdv.dev/SDV
+* Repository: https://github.com/sdv-dev/RDT
+* License: [MIT](https://github.com/sdv-dev/RDT/blob/master/LICENSE)
+* Development Status: [Pre-Alpha](https://pypi.org/search/?c=Development+Status+%3A%3A+2+-+Pre-Alpha)
+
+# Overview
+
+**RDT** is a Python library used to transform data for data science libraries and preserve
+the transformations in order to revert them as needed.
+
+# Addons
+
+**RDT** addons are families of transformers that are optionally installed to provide
+a wider range of transformers that can be optionally installed to the main package.
+
diff --git a/rdt/transformers/addons/addons_setup.py b/rdt/transformers/addons/addons_setup.py
new file mode 100644
index 000000000..37a435fb6
--- /dev/null
+++ b/rdt/transformers/addons/addons_setup.py
@@ -0,0 +1,110 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""The setup script for the addons packages."""
+
+import json
+import os
+import shutil
+import sys
+from copy import deepcopy
+from glob import glob
+from tempfile import TemporaryDirectory
+
+from setuptools import find_namespace_packages, setup
+
+import rdt
+
+with open('README.md', encoding='utf-8') as readme_file:
+    README = readme_file.read()
+
+RDT_VERSION = rdt.__version__
+ADDONS_PATH = os.path.dirname(os.path.realpath(__file__))
+
+
+def _build_setup(addon_json):
+
+    with open(addon_json, 'r', encoding='utf-8') as f:
+        addon = json.load(f)
+
+    addon_name = addon.get('name')
+    addon_module = addon.get('transformers')[0].split('.')
+    addon_module = '.'.join(addon_module[:-2])
+
+    install_requires = [f'rdt>={RDT_VERSION}']
+    install_requires.extend(addon.get('requirements', []))
+
+    # this does something
+    setup(
+        author='MIT Data To AI Lab',
+        author_email='dailabmit@gmail.com',
+        classifiers=[
+            'Development Status :: 2 - Pre-Alpha',
+            'Intended Audience :: Developers',
+            'License :: OSI Approved :: MIT License',
+            'Natural Language :: English',
+            'Programming Language :: Python :: 3',
+            'Programming Language :: Python :: 3.6',
+            'Programming Language :: Python :: 3.7',
+            'Programming Language :: Python :: 3.8',
+        ],
+        description='Reversible Data Transforms',
+        include_package_data=False,
+        install_requires=install_requires,
+        keywords=['rdt', addon_name],
+        license='MIT license',
+        long_description=README,
+        long_description_content_type='text/markdown',
+        name=addon_name,
+        packages=find_namespace_packages(include=[addon_module]),
+        python_requires='>=3.6,<3.9',
+        url='https://github.com/sdv-dev/RDT',
+        version=RDT_VERSION,
+        zip_safe=False,
+    )
+
+
+def _run():
+
+    path = sys.argv[0]
+    base_path = os.path.realpath(path).replace(path, '')
+
+    # clear build if exists
+    build_path = os.path.join(base_path, 'build')
+    if os.path.exists(build_path):
+        shutil.rmtree(build_path)
+
+    families = deepcopy(sys.argv[1:])
+    all_families = [
+        family
+        for family in os.listdir('.')
+        if os.path.isdir(family)
+    ]
+
+    families = list(set(families).intersection(set(all_families)))
+    for addon in glob(f'{ADDONS_PATH}/*/*.json'):
+        with TemporaryDirectory() as temp_dir:
+            build_command = [
+                path, 'bdist_wheel', '--dist-dir', 'dist', '--bdist-dir', temp_dir,
+                'sdist', '--dist-dir', 'dist', 'egg_info', '--egg-base', temp_dir
+            ]
+
+            base_name = os.path.basename(os.path.dirname(addon))
+            sys.argv = deepcopy(build_command)
+
+            if not families:
+                _build_setup(addon)
+
+            else:
+                if os.path.basename(os.path.dirname(addon)) in families:
+                    _build_setup(addon)
+
+            remove_addon_build = os.path.join(
+                base_path, 'build', 'lib', 'rdt', 'transformers', 'addons', base_name
+            )
+
+            # delete only the processed addon folder
+            shutil.rmtree(remove_addon_build)
+
+
+_run()
diff --git a/rdt/transformers/addons/identity/__init__.py b/rdt/transformers/addons/identity/__init__.py
new file mode 100644
index 000000000..9f0173892
--- /dev/null
+++ b/rdt/transformers/addons/identity/__init__.py
@@ -0,0 +1,1 @@
+"""Identity addons module."""
diff --git a/rdt/transformers/addons/identity/config.json b/rdt/transformers/addons/identity/config.json
new file mode 100644
index 000000000..f3ad1e27e
--- /dev/null
+++ b/rdt/transformers/addons/identity/config.json
@@ -0,0 +1,6 @@
+{
+    "name": "rdt_identity",
+    "transformers": [
+        "rdt.transformers.addons.identity.identity.IdentityTransformer"
+    ]
+}
diff --git a/rdt/transformers/addons/identity/identity.py b/rdt/transformers/addons/identity/identity.py
new file mode 100644
index 000000000..aaf2a87af
--- /dev/null
+++ b/rdt/transformers/addons/identity/identity.py
@@ -0,0 +1,47 @@
+"""IdentityTransformer module."""
+
+from rdt.transformers.base import BaseTransformer
+
+
+class IdentityTransformer(BaseTransformer):
+    """Identity transformer that produces the same data.
+
+    This transformer is intended for testing purposes only. The transform and reverse transform
+    of this data is equal to the input.
+    """
+
+    def _fit(self, data):
+        """Fit the transformer to the data.
+
+        Args:
+            data (pandas.Series or numpy.ndarray):
+                Data to fit the transformer to.
+        """
+        self.OUTPUT_TYPES = {
+            column: None
+            for column in self.columns
+        }
+
+    def _transform(self, data):
+        """Return the same input data.
+
+        Args:
+            data (pandas.Series or numpy.ndarray):
+                Data to transform.
+
+        Returns:
+            pandas.DataFrame or pandas.Series
+        """
+        return data
+
+    def _reverse_transform(self, data):
+        """Return the same input data.
+
+        Args:
+            data (pandas.Series or numpy.ndarray):
+                Data to revert.
+
+        Returns:
+            pandas.DataFrame or pandas.Series
+        """
+        return data
diff --git a/setup.cfg b/setup.cfg
index d2bdf60a1..57389136b 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -33,7 +33,7 @@ universal = 1
 
 [flake8]
 max-line-length = 99
-exclude = docs, .tox, .git, __pycache__, .ipynb_checkpoints
+exclude = docs, .tox, .git, __pycache__, .ipynb_checkpoints, rdt/transformers/__init__.py
 extend-ignore = D107, SFS2, SFS3
 
 [isort]
diff --git a/setup.py b/setup.py
index 9bb8673f7..5595e5577 100644
--- a/setup.py
+++ b/setup.py
@@ -3,7 +3,10 @@
 
 """The setup script."""
 
-from setuptools import setup, find_packages
+import json
+import os
+from glob import glob
+from setuptools import setup, find_namespace_packages
 
 with open('README.md', encoding='utf-8') as readme_file:
     readme = readme_file.read()
@@ -33,6 +36,14 @@
     'rundoc>=0.4.3,<0.5',
 ]
 
+addons_require = []
+
+for addon_json in glob('rdt/transformers/addons/*/*.json'):
+    with open(addon_json, 'r', encoding='utf-8') as addon_json_file:
+        requirements = json.load(addon_json_file).get('requirements')
+        if requirements:
+            addons_require.extend(requirements)
+
 development_requires = [
     # general
     'bumpversion>=0.5.3,<0.6',
@@ -85,8 +96,8 @@
     description='Reversible Data Transforms',
     extras_require={
         'copulas': copulas_requires,
-        'test': tests_require + copulas_requires,
-        'dev': development_requires + tests_require + copulas_requires,
+        'test': tests_require + copulas_requires + addons_require,
+        'dev': development_requires + tests_require + copulas_requires + addons_require,
     },
     include_package_data=True,
     install_requires=install_requires,
@@ -95,7 +106,7 @@
     long_description=readme + '\n\n' + history,
     long_description_content_type='text/markdown',
     name='rdt',
-    packages=find_packages(include=['rdt', 'rdt.*']),
+    packages=find_namespace_packages(include=['rdt', 'rdt.transformers']),
     python_requires='>=3.6,<3.9',
     setup_requires=setup_requires,
     test_suite='tests',


Test patch:
diff --git a/tests/unit/transformers/addons/__init__.py b/tests/unit/transformers/addons/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/unit/transformers/addons/identity/__init__.py b/tests/unit/transformers/addons/identity/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/unit/transformers/addons/identity/test_indentity.py b/tests/unit/transformers/addons/identity/test_indentity.py
new file mode 100644
index 000000000..03d5d7d1e
--- /dev/null
+++ b/tests/unit/transformers/addons/identity/test_indentity.py
@@ -0,0 +1,80 @@
+from unittest import TestCase
+from unittest.mock import Mock
+
+import numpy as np
+
+from rdt.transformers import IdentityTransformer
+
+
+class TestIdentityTransformer(TestCase):
+
+    def test__fit(self):
+        """Test ``IdentityTransformer._fit`` function.
+
+        The ``_fit`` function is expected to learn the columns ``INPUT_TYPE`` and
+        ``OUTPUT_COLUMNS`` from the ``self.columns``.
+
+        Setup:
+            - mock self.columns to be a list with values.
+        Input:
+            - ``data``, a ``numpy.ndarray`` with numerical values.
+        Output:
+            - n/a
+        """
+        # setup
+        instance = Mock()
+        instance.columns = ['a', 'b', 'c']
+
+        # run
+        data = np.array([0.5, 0.6, 0.7])
+        IdentityTransformer._fit(instance, data)
+
+        # assert
+        expected_output_types = {
+            'a': None,
+            'b': None,
+            'c': None,
+        }
+
+        assert instance.OUTPUT_TYPES == expected_output_types
+
+    def test__transform(self):
+        """Test ``IdentityTransformer._transform`` function.
+
+        The ``_transform`` function is expected to return the input object without modifying it.
+
+        Input:
+            - An `object` instance.
+        Output:
+            - Same ``object`` as input, unmodified.
+        """
+        # setup
+        instance = Mock()
+        input_object = object()
+
+        # run
+        output_object = IdentityTransformer._transform(instance, input_object)
+
+        # assert
+        assert input_object == output_object
+
+    def test__reverse_transform(self):
+        """Test ``IdentityTransformer._reverse_transform`` function.
+
+        The ``_reverse_transform`` function is expected to return the input object without
+        modifying it.
+
+        Input:
+            - An `object` instance.
+        Output:
+            - Same ``object`` as input, unmodified.
+        """
+        # setup
+        instance = Mock()
+        input_object = object()
+
+        # run
+        output_object = IdentityTransformer._transform(instance, input_object)
+
+        # assert
+        assert input_object == output_object