Repository: sdv-dev/RDT (language: python)

Issue:
RDT contributing guide
### Problem Description

There should be a guide on how to contribute to RDT.

### Expected behavior

Write a step-by-step contributing guide that can be used to add new Transformers to RDT.

It can be added in the `CONTRIBUTING.rst` document within RDT, with links to the SDV Developer Guides
It should cover all the steps that a contributor needs to take, including the execution of all the transformer validation functions.



Discussion / hints:

Fix patch:
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index bd70379a9..c6659497f 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -7,373 +7,541 @@ Contributing
 Contributions are welcome, and they are greatly appreciated! Every little bit
 helps, and credit will always be given.
 
-You can contribute in many ways:
+.. contents:: Table of contents
+   :local:
+   :depth: 3
 
-Types of Contributions
-----------------------
+Contributing to RDT
+-------------------
 
-Report Bugs
-~~~~~~~~~~~
+Setting Up a Local Development Environment
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Report bugs at the `GitHub Issues page`_.
+1. Fork the `Reversible Data Transforms` repo on GitHub.
+2. Clone your fork locally::
+
+    $ git clone git@github.com:your_name_here/RDT.git
+
+3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed,
+   this is how you set up your fork for local development::
+
+    $ mkvirtualenv RDT
+    $ cd RDT/
+    $ make install-develop
+
+4. Claim or file an issue on GitHub. If there is already an issue on GitHub for the
+   contribution you wish to make, claim it. If not, please file an issue and then claim
+   it before creating a branch.
+
+5. Create a branch for local development::
+
+    $ git checkout -b issue-[issue-number]-description-of-your-bugfix-or-feature
+
+   The naming scheme for your branch should have a prefix of the format ``issue-X``
+   where X is the associated issue's number (eg. ``issue-123-fix-foo-bug``). If you
+   are not developing on your own fork, further prefix the branch with your GitHub
+   username, like ``githubusername/gh-123-fix-foo-bug``.
+
+   Now you can make your changes locally.
+
+6. Commit your changes and push your branch to GitHub::
+
+    $ git add .
+    $ git commit -m "Your detailed description of your changes."
+    $ git push origin name-of-your-branch
+
+Code Style
+~~~~~~~~~~
+
+RDT follows certain coding style guidelines. Any change made should conform to these
+guidelines. RDT using the following third party libraries to check the code style.
+
+* flake8::
+
+    $ flake8 rdt
+    $ flake8 tests --ignore=D
+
+* isort::
+
+    $ isort -c --recursive rdt tests
+
+* pylint::
+
+    $ pylint rdt tests/performance --rcfile=setup.cfg
+
+* pydocstyle::
+
+    $ pydocstyle rdt
+    $ pydocstyle tests
+
+To run all of the code style checks in RDT, use the following command::
+
+    $ make lint
+
+or if you are developing on Windows you can use::
+
+    $ invoke lint
+
+Unit Test Guidelines
+~~~~~~~~~~~~~~~~~~~~
 
-If you are reporting a bug, please include:
+There should be unit tests created specifically for any changes you add.
+The unit tests are expected to cover 100% of your contribution's code based on the
+coverage report. All the Unit Tests should comply with the following requirements:
 
-* Your operating system name and version.
-* Any details about your local setup that might be helpful in troubleshooting.
-* Detailed steps to reproduce the bug.
+1. Unit Tests should be based only in unittest and pytest modules.
+
+2. The tests that cover a module called ``rdt/path/to/a_module.py`` should be implemented
+   in a separated module called ``tests/unit/path/to/test_a_module.py.`` Note that the module
+   name has the ``test_`` prefix and is located in a path similar to the one of the tested
+   module, just inside the tests folder.
+
+3. Each method of the tested module should have at least one associated test method, and
+   each test method should cover only **one** use case or scenario.
+
+4. Test case methods should start with the ``test_`` prefix and have descriptive names
+   that indicate which scenario they cover.
+   Names such as ``test_some_method_input_none``, ``test_some_method_value_error`` or
+   ``test_some_method_timeout`` are good, but names like ``test_some_method_1``,
+   ``some_method`` or ``test_error`` are not.
 
-Fix Bugs
-~~~~~~~~
+5. Each test should validate only what the code of the method being tested does, and not
+   cover the behavior of any third party package or tool being used, which is assumed to
+   work properly as far as it is being passed the right values.
+
+6. Any third party tool that may have any kind of random behavior, such as some Machine
+   Learning models, databases or Web APIs, will be mocked using the ``mock`` library, and
+   the only thing that will be tested is that our code passes the right values to them.
+
+7. Unit tests should not use anything from outside the test and the code being tested. This
+   includes not reading or writing to any file system or database, which will be properly
+   mocked.
 
-Look through the GitHub issues for bugs. Anything tagged with "bug" and "help
-wanted" is open to whoever wants to implement it.
+To run the test suite in RDT locally, use the following command::
 
-Implement Features
-~~~~~~~~~~~~~~~~~~
+    $ make test
 
-Look through the GitHub issues for features. Anything tagged with "enhancement"
-and "help wanted" is open to whoever wants to implement it.
+or if you are developing on Windows, use::
 
-Write Documentation
-~~~~~~~~~~~~~~~~~~~
+    $ invoke test
 
-Reversible Data Transforms could always use more documentation, whether as part of the
-official Reversible Data Transforms docs, in docstrings, or even on the web in blog posts,
-articles, and such.
+.. _Pull Request Guidelines:
 
-Submit Feedback
-~~~~~~~~~~~~~~~
+Pull Request Guidelines
+~~~~~~~~~~~~~~~~~~~~~~~
 
-The best way to send feedback is to file an issue at the `GitHub Issues page`_.
+Before you submit a pull request, check that it meets these guidelines:
 
-If you are proposing a feature:
+1. It resolves an open GitHub Issue and contains its reference in the title or
+   the comment. If there is no associated issue, feel free to create one.
+2. Whenever possible, it resolves only **one** issue. If your PR resolves more than
+   one issue, try to split it in more than one pull request.
+3. The pull request should include unit tests that cover all the changed code.
+4. The pull request should work for all the supported Python versions. Check the `Github actions
+   page`_ and make sure that all the checks pass.
 
-* Explain in detail how it would work.
-* Keep the scope as narrow as possible, to make it easier to implement.
-* Remember that this is a volunteer-driven project, and that contributions
-  are welcome :)
+Contributing a New Transformer
+------------------------------
 
-Adding a New Transformer
-------------------------
+In addition to the guidelines mentioned above, there are extra steps that need to be taken
+when adding a new ``Transformer`` class. They are described in detail in this section.
 
-Create Transformer Class
+Transformer Requirements
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-All new Transformer classes should inherit from `BaseTransformer` or one of its child classes.
+When contributing a new transformer, the most obvious requirement is creating the new Transformer
+class. The class should inherit from `BaseTransformer` or one of its child classes.
 There are only three required methods for a transformer:
 
-1. ``fit``: Used to store and learn any values from the input data that might be useful
-   for the transformer.
-2. ``transform``: Used to transform the input data into completely numeric data. This method
-   should not modify the internal state of the Transformer instance.
-3. ``reverse_transform``: Used to convert data that is completely numeric back into the
-   format of the fitted data. This method should not modify the internal state of the
-   Transformer instance.
+1. ``_fit(data: pd.DataFrame)``: Used to store and learn any values from the input data that
+   might be useful for the transformer.
+2. ``_transform(data: pd.DataFrame)``: Used to transform the input data into completely numeric
+   data. This method should not modify the internal state of the Transformer instance.
+3. ``_reverse_transform(data: pd.DataFrame)``: Used to convert data that is completely numeric
+   back into the format of the fitted data. This method should not modify the internal state of
+   the Transformer instance.
 
 Each transformer class should be placed inside the ``rdt/transformers`` folder, in a module
 file named after the data type that the transformer operates on. For example, if you are
 writing a transformer that works with ``categorical`` data, your transformer should be placed
 inside the ``rdt/transformers/categorical.py`` module.
 
-Common Performance Pitfalls
-"""""""""""""""""""""""""""
-It is important to try to keep the performance of these transformers as efficient as possible.
-Below are some tips and common pitfalls to avoid when developing your transformer, so as to
-optimize performance.
+For more detailed guide on writing transformers, refer to the `Development Guide`_.
 
-1. Avoid duplicate operations. If you need to do some change to an array/series, try to only
-   do it once and reuse that variable later.
-2. Try to use vectorized operations when possible.
-3. When working with Pandas Series, a lot of the operations are able to handle nulls. If you
-   need to round, get the max or get the min of a series, there is no need to filter out nulls
-   before doing that calculation.
-4. ``pd.to_numeric`` is preferred over ``as_type``.
-5. ``pd.to_numeric`` also replaces all None values with NaNs that can be operated on since
-   ``np.nan`` is a float type.
-6. If you are working with a series that has booleans and null values, there is a
-   `nullable boolean type`_ that can be leveraged to avoid having to filter out null values.
+On top of adding the new class, unit tests must be written to cover all of the methods the new
+class uses. In some cases, integration tests may also be required. More details on this can be
+found below.
 
-Create Performance Notebook
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+If the transformer add a previously unsupported `Data Type` to RDT, then more steps will need
+to be taken for the quality and performance tests. A new `DatasetGenerator` class may need to
+be created for the `Data Type`. You may also need to find a real world dataset containing this
+`Data Type` and request for it to be added. More details for these steps can be found below in
+the `Transformer Performance`_ and `Transformer Quality`_ sections respectively.
 
-Once your transformer is complete, please follow the format in this sample `Colab Notebook`_
-that runs common performance metrics on the implemented methods and create one for your
-transformer. This can help you find inefficiencies in the transformer and may give you ideas
-for improvements.
+Transformer Validations
+~~~~~~~~~~~~~~~~~~~~~~~
 
-Add Unit and Integration Tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. _Code Style:
 
-There should be unit tests and integration tests created specifically for the new transformer
-you add. Unit tests should cover specific cases for each of the following methods: ``__init__``,
-``fit``, ``transform`` and ``reverse_transform``. They can be added under
-``tests/unit/transformers/{transformer_module}``.
+Code Style
+""""""""""
 
-The integration tests should test the whole workflow of going from input data, to fitting, to
-transforming and finally reverse transforming the data. The tests should make sure the reversed
-data is in the same format or exactly identical to the input data. Integration tests can be
-added under ``tests/unit/transformers/{transformer_module}``.
+The code added for the new transformer must abide by the code style used in RDT. In addition,
+there are custom code style requirements that must also be met. These mostly have to do with
+class and method naming conventions. For example, all transformer classes must ened in
+``Transformer``. They also have to inherit from the ``rdt.transformers.BaseTransformer`` class.
 
-Adding Performance Tests
-~~~~~~~~~~~~~~~~~~~~~~~~
+Validating Code Style
+*********************
 
-Once the new ``transformer`` is complete and has been analyzed for performance, performance tests
-should be added for it. If it is a completely new ``transformer``, add a folder with the same name
-as it to this directory: ``/tests/performance/test_cases``. Otherwise, add the test cases to the
-appropriate existing folder.
-
-The naming convention for the test case files is as follows:
-
-``{description of arguments}_{dataset_generator_name}_{fit_size}_{transform_size}.json``
-
-For example, if we use the default arguments with the ``ConstantIntegerGenerator``, and generate
-1000 rows for ``fit`` as well as ``transform``, then we would have the name
-``default_ConstantIntegerGenerator_1000_1000.json``.
-
-Each test case configuration file has the following format::
-
-   {
-      "dataset": "tests.performance.datasets.UniqueCategories",
-      "transformer": "rdt.transformers.categorical.CategoricalTransformer,
-      "kwargs": {},
-      "fit_rows": 1000,
-      "transform_rows": 10000,
-      "expected": {
-         "fit": {
-               "time": 0.3,
-               "memory": 400
-         },
-         "transform": {
-               "time": 0.3,
-               "memory": 400
-         },
-         "reverse_transform": {
-               "time": 0.3,
-               "memory": 400
-         }
-      }
-   }
-
-The configuration should specify the full Python path of the transformer, the keyword arguments
-that need to be passed to the transformer when creating its instance, the full Python path of
-the dataset generator (explained in more detail below), the number of rows to generate for both
-``fit`` and ``transform`` and the max allowable time and memory for each method.
-
-There is a function called ``make_test_case_configs`` in ``tests/performance/test_performance.py``
-that can be used to generate test cases once you have the dataset generators created.
-
-Create Dataset generators
-~~~~~~~~~~~~~~~~~~~~~~~~~
+To validate the overall code style for your transformer, you can use the custom code validation
+function, ``validate_transformer_code_style``. This function returns a boolean indicating whether
+or not the transformer passed all the code style checks. It also prints a table describing each
+check and whether or not it passed.
 
-In order to test performance, we have a class that is responsible for generating data to test
-the transformer methods against. Each subclass implements one static method, ``generate`` that
-takes in the number of rows to generate. You should make a generator for every type of column
-that you believe would be useful to test against. For some examples, you can look in this
-folder: https://github.com/sdv-dev/RDT/tree/master/tests/performance/datasets
+.. code-block:: Python
 
-The generators also each have the following class variables:
+   In [1]: from tests.contributing import validate_transformer_code_style
 
-1. ``TYPE``
-2. ``SUBTYPE``
+   In [2]: valid = validate_transformer_code_style('rdt.transformers.BooleanTransformer') # Replace BooleanTransformer with your transformer
+   Validating source file C:\Datacebo\RDT\rdt\transformers\boolean.py
 
-These should match the type and subtype of data that your ``transformer`` is used for.
+   SUCCESS: The code style is correct.
 
-Maintainer's Checklist
-~~~~~~~~~~~~~~~~~~~~~~
+   Check                      Correct    Details
+   -------------------------  ---------  ---------------------------------------------------------
+   flake8                     Yes        Code follows PEP8 standards.
+   isort                      Yes        Imports are properly sorted.
+   pylint                     Yes        Code is properly formatted and structured.
+   pydocstyle                 Yes        The docstrings are properly written.
+   Transformer Name           Yes        Transformer name ends with ``Transformer``.
+   Transformer is subclass    Yes        The transformer is subclass of ``BaseTransformer``.
+   Valid module               Yes        The transformer is placed inside a valid module.
+   Valid test module          Yes        The transformer tests are placed inside the valid module.
+   Valid test function names  Yes        The transformer tests are named correctly.
+   Valid transformer addon    Yes        The addon is configured properly.
+   Importable from module     Yes        The transformer can be imported from the parent module.
 
-Once you have done everything above, you can create a PR. Be sure to look over the
-checklist below to make sure your PR is ready for review.
+   In [3]: valid
+   Out[3]: True
 
-1. Verify that the profiling notebook was created and used to find any obvious bottlenecks.
-2. Verify that performance test cases were created.
-3. Verify that the timings and memory values for these test cases are reasonable compared
-   to other similar transformers if possible.
-4. Verify that unit and integration tests were added for the transformers.
-5. Create an issue that is assigned to the user making the PR and verify that the PR resolves
-   that issue.
-6. Review the ``Pull Request Guidelines`` below.
+Unit Tests
+""""""""""
 
-Get Started!
-------------
+* Unit tests should cover specific cases for each of the following methods: ``__init__``,
+  ``fit``, ``transform`` and ``reverse_transform``.
+* Unit tests for a transformer must have 100% coverage based on the code coverage report.
+* The tests should go in a module called ``tests/unit/transformers/{transformer_module}``.
 
-Ready to contribute? Here's how to set up `Reversible Data Transforms` for local development.
+Validating Unit Tests
+*********************
 
-1. Fork the `Reversible Data Transforms` repo on GitHub.
-2. Clone your fork locally::
+The transformer unit tests and their coverage can be validated using the
+``validate_transformer_unit_tests`` function. This function returns a ``float`` value representing
+the test coverage where 1.0 is 100%. It also prints each test and whether or not it passed. It also
+prints a table summarizing the test coverage and provides a link to the full coverage report.
 
-    $ git clone git@github.com:your_name_here/RDT.git
+.. code-block:: Python
 
-3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed,
-   this is how you set up your fork for local development::
+   In [1]: from tests.contributing import validate_transformer_unit_tests
 
-    $ mkvirtualenv RDT
-    $ cd RDT/
-    $ make install-develop
+   In [2]: test_coverage = validate_transformer_unit_tests('rdt.transformers.BooleanTransformer') # Replace BooleanTransformer with your transformer
+   Validating source file C:\Datacebo\RDT\rdt\transformers\boolean.py
 
-4. Create a branch for local development::
+   ================================================= test session starts =================================================
+   collected 12 items
 
-    $ git checkout -b name-of-your-bugfix-or-feature
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test___init__ PASSED                            [  8%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__fit_array PASSED                          [ 16%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__fit_nan_ignore PASSED                     [ 25%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__fit_nan_not_ignore PASSED                 [ 33%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__reverse_transform_2d_ndarray PASSED       [ 41%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__reverse_transform_float_values PASSED     [ 50%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__reverse_transform_float_values_out_of_range PASSED [ 58%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__reverse_transform_nan_ignore PASSED       [ 66%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__reverse_transform_nan_not_ignore PASSED   [ 75%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__reverse_transform_not_null_values PASSED  [ 83%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__transform_array PASSED                    [ 91%]
+   tests/unit/transformers/test_boolean.py::TestBooleanTransformer::test__transform_series PASSED                   [100%]
 
-   Try to use the naming scheme of prefixing your branch with ``gh-X`` where X is
-   the associated issue, such as ``gh-3-fix-foo-bug``. And if you are not
-   developing on your own fork, further prefix the branch with your GitHub
-   username, like ``githubusername/gh-3-fix-foo-bug``.
+   ============================================ 12 passed, 1 warning in 0.08s ============================================
 
-   Now you can make your changes locally.
+   SUCCESS: The unit tests passed.
+   Name                          Stmts   Miss  Cover   Missing
+   -----------------------------------------------------------
+   rdt\transformers\boolean.py      37     19    49%   3-36, 40-55, 68, 88, 100
+   -----------------------------------------------------------
+   TOTAL                            37     19    49%
 
-5. While hacking your changes, make sure to cover all your developments with the required
-   unit tests, and that none of the old tests fail as a consequence of your changes.
-   For this, make sure to run the tests suite and check the code coverage::
+   ERROR: The unit tests only cover 48.649% of your code.
 
-    $ make lint       # Check code styling
-    $ make test       # Run the tests
-    $ make coverage   # Get the coverage report
+   Full coverage report here:
 
-6. When you're done making changes, check that your changes pass all the styling checks and
-   tests, including other Python supported versions, using::
+   file:///C:/Datacebo/RDT/htmlcov/rdt_transformers_boolean_py.html
 
-    $ make test-all
+   In [3]: test_coverage
+   Out [3]: 0.486
 
-7. Make also sure to include the necessary documentation in the code as docstrings following
-   the `Google docstrings style`_.
-   If you want to view how your documentation will look like when it is published, you can
-   generate and view the docs with this command::
+Integration Tests
+"""""""""""""""""
 
-    $ make view-docs
+Integration tests should test the entire workflow of going from input data, to fitting, to
+transforming and finally reverse transforming the data. By default, we run integration tests
+for each transformer that validate the following checks:
 
-8. Commit your changes and push your branch to GitHub::
+1. The Transformer correctly defines the data type that it supports.
+2. At least one Dataset Generator exists for the Transformer data type.
+3. The Transformer can transform data and produces outputs of the indicated data types.
+4. The Transformer can reverse transform the data it produces, recovering the original data type.
+   If ``is_composite_identity``, we expect that the reverse transformed data is equal to the
+   original data.
+5. The HyperTransformer is able to use the Transformer and produce float values.
+6. The HyperTransformer is able to reverse the data that has previously transformed,
+   and restore the original data type.
 
-    $ git add .
-    $ git commit -m "Your detailed description of your changes."
-    $ git push origin name-of-your-bugfix-or-feature
+If you wish to test any specific end-to-end scenarios that were not covered in the above checks, 
+add a new integration test. Integration tests can be added under
+``tests/integration/path/to/test_a_module.py``.
 
-9. Submit a pull request through the GitHub website.
+* Before putting up a PR, confirm that the automatic integration tests pass. If new functionality
+  that isn't covered is added, feel free to add new integration tests.
+* Integration tests should be added under ``tests/unit/transformers/{transformer_module}``.
 
-Pull Request Guidelines
------------------------
+Validating Integration Tests
+****************************
 
-Before you submit a pull request, check that it meets these guidelines:
+Integration tests can be validated using the ``validate_transformer_integration`` function. This
+function returns a boolean representing whether or not the transformer passes all integration
+checks. It also prints a table describing each check and whether or not it passed.
 
-1. It resolves an open GitHub Issue and contains its reference in the title or
-   the comment. If there is no associated issue, feel free to create one.
-2. Whenever possible, it resolves only **one** issue. If your PR resolves more than
-   one issue, try to split it in more than one pull request.
-3. The pull request should include unit tests that cover all the changed code
-4. If the pull request adds functionality, the docs should be updated. Put
-   your new functionality into a function with a docstring, and add the
-   feature to the documentation in an appropriate place.
-5. The pull request should work for all the supported Python versions. Check the `Github actions
-   page`_ and make sure that all the checks pass.
+.. code-block:: Python
 
-Unit Testing Guidelines
------------------------
+   In [1]: from tests.contributing import validate_transformer_integration
 
-All the Unit Tests should comply with the following requirements:
+   In [2]: valid = validate_transformer_integration('rdt.transformers.BooleanTransformer') # Replace BooleanTransformer with your transformer
+   Validating Integration Tests for transformer BooleanTransformer
 
-1. Unit Tests should be based only in unittest and pytest modules.
+   SUCCESS: The integration tests were successful.
 
-2. The tests that cover a module called ``rdt/path/to/a_module.py``
-   should be implemented in a separated module called
-   ``tests/rdt/path/to/test_a_module.py``.
-   Note that the module name has the ``test_`` prefix and is located in a path similar
-   to the one of the tested module, just inside the ``tests`` folder.
+   Check                                   Correct    Details
+   --------------------------------------  ---------  -----------------------------------------------------------------------------------------------------------------------
+   Dataset Generators                      Yes        At least one Dataset Generator exists for the Transformer data type.
+   Output Types                            Yes        The Transformer can transform data and produce output(s) of the indicated data type(s).
+   Reverse Transform                       Yes        The Transformer can reverse transform the data it produces, going back to the original data type.
+   Hypertransformer can transform          Yes        The HyperTransformer is able to use the Transformer and produce float values.
+   Hypertransformer can reverse transform  Yes        The HyperTransformer is able to reverse the data that it has previously transformed and restore the original data type.
 
-3. Each method of the tested module should have at least one associated test method, and
-   each test method should cover only **one** use case or scenario.
+   In [3]: valid
+   Out [3]: True
 
-4. Test case methods should start with the ``test_`` prefix and have descriptive names
-   that indicate which scenario they cover.
-   Names such as ``test_some_methed_input_none``, ``test_some_method_value_error`` or
-   ``test_some_method_timeout`` are right, but names like ``test_some_method_1``,
-   ``some_method`` or ``test_error`` are not.
+.. _Transformer Performance:
 
-5. Each test should validate only what the code of the method being tested does, and not
-   cover the behavior of any third party package or tool being used, which is assumed to
-   work properly as far as it is being passed the right values.
+Transformer Performance
+"""""""""""""""""""""""
 
-6. Any third party tool that may have any kind of random behavior, such as some Machine
-   Learning models, databases or Web APIs, will be mocked using the ``mock`` library, and
-   the only thing that will be tested is that our code passes the right values to them.
+We want to ensure our transformers are as efficient as possible, in terms of time and memory.
+In order to do so, we run performance tests on each transformer, based on the input data type
+specified by the transformer.
 
-7. Unit tests should not use anything from outside the test and the code being tested. This
-   includes not reading or writing to any file system or database, which will be properly
-   mocked.
+We generate test data using Dataset Generators. Each transformer should have at least one
+Dataset Generator that produces data of the transformer's input type.
+If there are any specific dataset characteristics that you think may affect your transformer
+performance (e.g. constant data, mostly null data), consider adding a Dataset Generator
+for that scenario as well.
+
+.. _Creating Dataset Generators:
+
+Creating Dataset Generators
+***************************
+
+In order to test performance, we have a class that is responsible for generating data to test
+the transformer methods against. Each subclass implements two static method, ``generate`` 
+and ``get_performance_thresholds``.
+
+1. ``generate`` takes in the number of rows to generate, and outputs the expected number
+   of data rows.
+2. ``get_performance_thresholds`` returns the time and memory threshold for each of the required
+   transformer methods. These thresolds are per row.
+
+You should make a generator for every type of column that you believe would be useful to test
+against. For some examples, you can look in the `dataset generator folder`_.
+
+The generators each have a ``DATA_TYPE`` class variable. This should match the data type that your
+``transformer`` accepts as input.
+
+More details can be found in the `Development Guide`_.
+
+Common Performance Pitfalls
+***************************
+
+It is important to keep the performance of these transformers as efficient as possible.
+Below are some tips and common pitfalls to avoid when developing your transformer, so as to
+optimize performance.
+
+1. Avoid duplicate operations. If you need to do some change to an array/series, try to only
+   do it once and reuse that variable later.
+2. Try to use vectorized operations when possible.
+3. When working with Pandas Series, a lot of the operations are able to handle nulls. If you
+   need to round, get the max or get the min of a series, there is no need to filter out nulls
+   before doing that calculation.
+4. ``pd.to_numeric`` is preferred over ``as_type``.
+5. ``pd.to_numeric`` also replaces all None values with NaNs that can be operated on since
+   ``np.nan`` is a float type.
+6. If you are working with a series that has booleans and null values, there is a
+   `nullable boolean type`_ that can be leveraged to avoid having to filter out null values.
+
+Validating Performance
+**********************
+
+Validate the performance of your transformer using the ``validate_transformer_performance``
+function. This function returns a ``pandas.DataFrame`` containing the performance results
+of the transformer.
+
+.. code-block:: Python
+
+   In [1]: from tests.contributing import validate_transformer_performance
+
+   In [2]: results = validate_transformer_performance('rdt.transformers.DatetimeTransformer') # Replace DatetimeTransformer with your transformer
+   Validating Performance for transformer DatetimeTransformer
+
+   SUCCESS: The Performance Tests were successful.
 
-Tips
-----
+   In [3]: results
+   Out [3]:
+            Evaluation Metric         Value Acceptable     Units  Compared to Average
+   0                Fit Memory  9.334700e+01        Yes  Mb / row             0.757455
+   1                  Fit Time  6.232677e-07        Yes   s / row             0.574041
+   2  Reverse Transform Memory  1.451382e+02        Yes  Mb / row             0.966153
+   3    Reverse Transform Time  6.641531e-07        Yes   s / row             1.080660
+   4          Transform Memory  8.896317e+01        Yes  Mb / row             0.656664
+   5            Transform Time  5.217231e-07        Yes   s / row             0.484631
 
-To run a subset of tests::
+Fix any performance issues that are reported. If there are no errors but performance
+can be improved, this function should be used for reference.
 
-    $ python -m pytest tests.test_rdt
-    $ python -m pytest -k 'foo'
+.. _Transformer Quality:
 
-Release Workflow
-----------------
+Transformer Quality
+"""""""""""""""""""
 
-The process of releasing a new version involves several steps combining both ``git`` and
-``bumpversion`` which, briefly:
+To assess the quality of a transformer, we run quality tests that apply the Transformer
+on all the real world datasets that contain the Transformer input data type. The quality tests
+look at how well the original correlations are preserved by using transformed data to train
+regression models that predict other columns in the data. We compare the transformer's quality
+results to that of other transformers of the same data type.
 
-1. Merge what is in ``master`` branch into ``stable`` branch.
-2. Update the version in ``setup.cfg``, ``rdt/__init__.py`` and
-   ``HISTORY.md`` files.
-3. Create a new git tag pointing at the corresponding commit in ``stable`` branch.
-4. Merge the new commit from ``stable`` into ``master``.
-5. Update the version in ``setup.cfg`` and ``rdt/__init__.py``
-   to open the next development iteration.
+.. _Adding a Dataset:
 
-.. note:: Before starting the process, make sure that ``HISTORY.md`` has been updated with a new
-          entry that explains the changes that will be included in the new version.
-          Normally this is just a list of the Pull Requests that have been merged to master
-          since the last release.
+Adding a Dataset
+****************
 
-Once this is done, run of the following commands:
+If the transformer you are creating adds a new data type, then a dataset with that type may need to
+be added for the quality tests. This only needs to be done if the transformer being added is 
+expected to preserve or expose relationships in the data. This can be done using the following
+steps:
 
-1. If you are releasing a patch version::
+1. Find a dataset containing the data type your transformer uses as an input.
 
-    make release
+2. Test your transformer against this dataset by loading it into a ``DataFrame`` and using the
+   ``get_transformer_regression_scores`` in the ``test_quality`` package::
 
-2. If you are releasing a minor version::
+    from tests.quality.test_quality import get_transformer_regression_scores
+    get_transformer_regression_scores(data, data_type, dataset_name, [transformer])
 
-    make release-minor
+3. If the scores are higher than the ``TEST_THRESHOLD`` in the ``test_quality`` package, contact
+   one of the `RDT core contributors`_ on GitHub and ask them to add the dataset. Once this is
+   done, the quality tests should pass.
 
-3. If you are releasing a major version::
+Validating Quality
+******************
 
-    make release-major
+Validate the quality of your transformer using the ``validate_transformer_quality`` function.
+This function returns a ``pandas.DataFrame`` containing the scores attained by the transformer
+on each dataset, how that score compares to average and whether or not it is acceptable.
 
-Release Candidates
-~~~~~~~~~~~~~~~~~~
+.. code-block:: Python
 
-Sometimes it is necessary or convenient to upload a release candidate to PyPi as a pre-release,
-in order to make some of the new features available for testing on other projects before they
-are included in an actual full-blown release.
+   In [1]: from tests.contributing import validate_transformer_quality
 
-In order to perform such an action, you can execute::
+   In [2]: results = validate_transformer_quality('rdt.transformers.CategoricalTransformer') # Replace CategoricalTransformer with your transformer
+   Validating Quality Tests for transformer CategoricalTransformer
 
-    make release-candidate
+   SUCCESS: The quality tests were successful.
 
-This will perform the following actions:
+   In [3]: results
+   Out [3]:
+                     Dataset     Score  Compared To Average  Acceptable
+   0                   adult  0.223325             0.443181        True
+   1      student_placements  0.457490             0.994631        True
+   2  student_placements_pii  0.457490             0.988428        True
 
-1. Build and upload the current version to PyPi as a pre-release, with the format ``X.Y.Z.devN``
+Fix any quality issues that are reported.
 
-2. Bump the current version to the next release candidate, ``X.Y.Z.dev(N+1)``
+Finalize Your Transformer
+"""""""""""""""""""""""""
 
-After this is done, the new pre-release can be installed by including the ``dev`` section in the
-dependency specification, either in ``setup.py``::
+Re-run all the previous validations until they pass. For a final verification, run
+``validate_pull_request`` and fix any errors reported. This function runs all the checks described
+above. It also prints a table summarizing the results of all these checks.
 
-    install_requires = [
-        ...
-        'rdt>=X.Y.Z.dev',
-        ...
-    ]
+.. code-block:: Python
 
-or in command line::
+   In [1]: from tests.contributing import validate_pull_request
 
-    pip install 'rdt>=X.Y.Z.dev'
+   In [2]: valid = validate_pull_request('rdt.transformers.BooleanTransformer') # Replace BooleanTransformer with your transformer
+   ...................
 
+   Check              Correct    Details
+   -----------------  ---------  ----------------------------------------------------------------------
+   Code Style         Yes        Code Style is acceptable.
+   Unit Tests         Yes        The unit tests are correct and run successfully.
+   Integration tests  Yes        The integration tests run successfully.
+   Performance Tests  Yes        The performance of the transformer is acceptable.
+   Quality tests      Yes        The output data quality is acceptable.
+   Clean Repository   Yes        There are no unexpected changes in the repository.
+
+   SUCCESS: The Pull Request can be made!
+   You can now commit all your changes, push to GitHub and create a Pull Request.
+
+   In [3]: valid
+   Out [3]: True
+
+Once you have done everything above, you can create a PR. Do this by following the steps in the
+`Pull Request Guidelines`_ section. Review and fill out the checklist in the PR template to ensure
+your code is ready for review.
+
+Summary of Steps to Add a New Transformer
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. If it does not exist, open an Issue in Github and describe the Transformer that will be added,
+   including the data type that it handles and how it will handle it.
+2. Create and clone a fork of the RDT repository.
+3. Create a branch in this repository using the naming convention
+   issue-[issue-number]-[transformer-name] (eg. issue-123-address-transformer).
+4. Implement the Transformer class.
+5. Run the ``validate_transformer_code_stye`` function described in the `Code Style`_ section
+   and fix the reported errors.
+6. Implement Unit Tests for the Transformer.
+7. Run the ``validate_transformer_unit_tests`` function and fix the reported errors.
+8. Run the ``validate_transformer_integration`` function and fix the reported errors.
+9. If required, implement the `Dataset Generators` for the new data type. This is described in the
+   `Creating Dataset Generators`_ section.
+10. Run the ``validate_transformer_performance`` function and fix any errors reported.
+    If there are no errors but performance can be improved, this function should be used for
+    reference.
+11. If this transformer is expected to help preserve relationships in the data, run the
+    ``validate_transformer_quality`` function. If the quality is too low, make the
+    necessary enhancements to the transformer.
+12. If the quality tests fail because there is no dataset for the transformer's data type,
+    follow the steps in the `Adding a Dataset`_ section to add a real world dataset
+    containing the new data type to the quality tests.
+13. Run the ``validate_pull_request`` function as a final check and fix any errors reported.
+14. After all the previous steps pass, all the new and modified files can be committed and pushed
+    to github, and a Pull Request can be submitted. Follow the steps in the
+    `Pull Request Guidelines`_ section to submit your Pull Request.
 
-.. _GitHub issues page: https://github.com/sdv-dev/RDT/issues
 .. _Github actions page: https://github.com/sdv-dev/RDT/actions
-.. _Google docstrings style: https://google.github.io/styleguide/pyguide.html?showone=Comments#Comments
 .. _nullable boolean type: https://pandas.pydata.org/pandas-docs/version/1.0/user_guide/boolean.html
-.. _Colab Notebook: https://colab.research.google.com/drive/1dGnBLMW-5LATGoBUuQKWfOTZFssBmgYu?usp=sharing
+.. _RDT core contributors: https://github.com/orgs/sdv-dev/teams/core-contributors
+.. _dataset generator folder: https://github.com/sdv-dev/RDT/tree/master/tests/datasets
+.. _Development Guide: DEVELOPMENT.rst
\ No newline at end of file
diff --git a/DEVELOPMENT.rst b/DEVELOPMENT.rst
new file mode 100644
index 000000000..a6441b06a
--- /dev/null
+++ b/DEVELOPMENT.rst
@@ -0,0 +1,420 @@
+.. highlight:: shell
+
+=================
+Development Guide
+=================
+
+.. contents:: Table of contents
+   :local:
+   :depth: 3
+
+This guide describes in detail the main technical components of RDT as well as how to develop
+them.
+
+RDT Technical Overview
+----------------------
+
+The goal of RDT is to be able to transform data that is not machine learning ready into data that
+is. By machine learning ready, we mean that the data should consist of data types that most machine
+learning models can process. Usually this means outputting numeric data with no nulls. On top of this,
+RDT also enforces that those transformations can be reversed, so that data of the original form can
+be obtained again. RDT accomplishes this with the use of two main classes:
+
+* ``BaseTransformer``
+* ``HyperTransformer``
+
+BaseTransformer
+"""""""""""""""
+
+Every Transformer in RDT inherits from the ``BaseTransformer``. The goal of this class or any of
+its subclasses is to take data of a certain data type, and convert it into machine learning ready
+data. To enable transformers to do this, the ``BaseTransformer`` has the following attributes:
+
+* ``INPUT_TYPE`` (str) - The input type for the transformer.
+* ``OUTPUT_TYPES`` (dict) - Dictionary mapping transformed column names to their data types.
+* ``DETERMINISTIC_TRANSFORM`` (bool) - Whether or not calling ``transform`` yields a deterministic
+  output.
+* ``DETERMINISTIC_REVERSE`` (bool) - Whether or not calling ``reverse_transform`` yields a
+  deterministic output.
+* ``COMPOSITION_IS_IDENTITY`` (bool) - Whether or not calling ``transform`` and then
+  ``reverse_transform`` back to back yields data identical to the ``transform`` input (the original
+  data).
+* ``NEXT_TRANSFORMERS`` (dict) - Dictionary mapping transformed column names to Transformer class names
+  to use on them. This is optional. It can also be partially defined, meaning that some transformed
+  columns can have an entry in the dict, but not all of them need to.
+* ``columns`` (list) - List of column names that the transformer will transform. Set during ``fit``.
+* ``output_columns`` (list) - List of column names in the output from calling ``transform``. Set
+  during ``fit``.
+* ``column_prefix`` (str) - Prefix that will be added to the output columns to make them unique on
+  the table. For transformers that act on a single column this is the column name, and for
+  transformers that act on multiple columns this is their names joined by `#`. Set during ``fit``.
+
+It also has the following default methods, which the ``Transformer`` subclasses may overwrite when
+necessary:
+
+* ``get_output_types()`` - Returns the name of the columns that the transform method creates. By
+  default this will be the ``OUTPUT_TYPES`` dictionary with the column names prepended with the
+  ``column_prefix`` with a dot (`.`) as the separator.
+* ``is_transform_deterministic()`` - Returns a boolean indicating whether the output of the
+  ``transform`` method is deterministic. By default this is the value of the
+  ``DETERMINISTIC_TRANSFORM`` attribute.
+* ``is_reverse_deterministic()`` - Returns a boolean indicating whether the output of the
+  ``reverse_transform`` method is deterministic. By default this is the value of the
+  ``DETERMINISTIC_REVERSE`` attribute.
+* ``is_composition_identity()`` - Returns a boolean indicating whether the output of calling
+  ``transform`` and then ``reverse_transform`` on the transformed output is deterministic. By
+  default this is the ``COMPOSITION_IS_IDENTITY`` attribute.
+* ``get_next_transformers()`` - Returns a dictionary mapping the names of the columns that the
+  ``transform`` method creates to the next transformer to use for those columns. By default this
+  will be the ``NEXT_TRANSFORMERS`` dictionary with the keys prepended with the ``column_prefix``
+  with a dot (`.`) as a separator.
+* ``fit(data, columns)`` - Takes in ``pandas.DataFrame`` and list of column names and stores
+  the information needed to run ``transform``.
+* ``transform(data, drop)`` - Takes in ``pandas.DataFrame`` and bool saying whether or not to
+  drop original columns in the output. Returns ``pandas.DataFrame`` containing transformed data.
+* ``reverse_transform(data, drop)`` - Takes in ``pandas.DataFrame`` and bool saying whether or
+  not to transformed columns in the output. Returns ``pandas.DataFrame`` containing reverse
+  transformed data.
+
+Any subclass of the ``BaseTransformer`` can add extra methods or attributes that it needs.
+
+HyperTransformer
+""""""""""""""""
+
+The ``HyperTransformer`` class is used to transform an entire table. Under the hood, the
+``HyperTransformer`` figures out which Transformer classes to use on each column in order to
+get a machine learning ready output. It does so using the following methods:
+
+* ``fit(data)`` - Takes in a ``pandas.DataFrame``. For every column or group of columns in the
+  data, it find a transformer to use on it and calls that transformer's `fit` method with those
+  columns. If the output of the transformer is not machine learning ready, it will recursively
+  find transformers to use on that data type until it is. A sequence of transformers to use is
+  created.
+* ``transform(data)`` - Takes in a ``pandas.DataFrame``. Goes through the sequence of transformers
+  created during ``fit`` and calls their underlying ``transform`` method.
+* ``reverse_transform(data)`` - Takes in a ``pandas.DataFrame``. Goes through the sequence of
+  transformers created during ``fit`` in reverse and calls their underlying ``reverse_transform``
+  method.
+
+Implementing a Transformer
+--------------------------
+
+In order to create a new Transformer class, the class should inherit from the ``BaseTransformer``.
+It should also set the values for the attributes defined above.
+
+*Note*: Some attributes might not be able to be determined until after ``fit`` is called. In this
+case, those attributes should be set in the ``_fit`` method.
+
+The only methods that need to be implemented for a new Transformer class are:
+
+* ``_fit(columns_data)``
+* ``_transform(columns_data)``
+* ``_reverse_transform()``
+
+Take note of the `_` preceding each method. The ``BaseTransformer`` will call these methods when
+``fit``, ``transform`` and ``reverse_transform`` are called. This is because the 
+``BaseTransformer`` figures out which columns to pass down behind the scenes. All of the `_`
+methods take in a ``pandas.Series`` or ``pandas.DataFrame`` containing only the columns that will
+be used by the transformer.
+
+If for some reason, the new transformer requires access to all of the data, then the ``fit``,
+``transform`` and ``reverse_transform`` methods can be overwritten.
+
+Example Transformer
+"""""""""""""""""""
+
+Now that we have some background information on how Transformers work in RDT, let's create a new
+one. For this example, we will create a simple ``USPhoneNumberTransformer``. The goal of this
+transformer is to take strings containing phone numbers into numeric data. For the sake of
+simplicity, we will assume all phone numbers are of the format `###-###-####` or
+`#-###-###-####`.
+
+Let's start by setting the necessary attributes and writing the ``__init__`` method.
+
+.. code-block:: Python
+
+    class USPhoneNumberTransformer(BaseTransformer):
+
+        INPUT_TYPE = 'phone_number'
+        DETERMINISTIC_TRANSFORM = True
+        DETERMINISTIC_REVERSE = True
+        COMPOSITION_IS_IDENTITY = True
+
+        def __init__(self):
+            self.has_country_code = None
+
+Now we can write the ``_fit`` method.
+
+.. code-block:: Python
+
+    def _fit(self, columns_data):
+        number = ''.join(columns_data.loc[0].split('-'))
+        self.has_country_code = len(number) == 11
+
+Since the ``country_code`` may or may not be present, we can overwrite the
+``get_next_transformers`` and ``get_output_types`` methods accordingly.
+
+.. code-block:: Python
+
+    def get_output_types(self):
+        output_types = {
+            'area_code': 'categorical',
+            'exchange': 'integer',
+            'line': 'integer'
+        }
+        if self.has_country_code:
+            output_types['country_code'] = 'categorical'
+
+        return self._add_prefix(output_types)
+
+    def get_next_transformers(self):
+        next_transformers = {
+            'country_code': 'CategoricalTransformer',
+            'area_code': 'CategoricalTransformer'
+        }
+        if self.has_country_code:
+            next_transformers['country_code'] = 'CategoricalTransformer'
+        
+        return self._add_prefix(next_transformers)
+
+``_add_prefix`` is a private method that prepends the ``column_prefix`` attributes to every key
+in a dictionary. Now that we have this information, we can write the ``_transform`` and
+``_reverse_transform`` methods.
+
+.. code-block:: Python
+
+    def _transform(self, data):
+        return data.str.split('-', expand=True)
+
+    def _reverse_transform(self, data):
+        if self.has_country_code:
+            country_code = data.iloc[:, 0].astype('str')
+            area_code = data.iloc[:, 1].astype('str')
+            exchange = data.iloc[:, 2].astype('str')
+            line = data.iloc[:, 3].astype('str')
+            return country_code + '-' + area_code + '-' + exchange + '-' + line
+        
+        area_code = data.iloc[:, 0].astype('str')
+        exchange = data.iloc[:, 1].astype('str')
+        line = data.iloc[:, 2].astype('str')
+        return area_code + '-' + exchange + '-' + line
+
+We don't have to worry about the naming of the output columns because the ``BaseTransformer``
+handles that for us. Let's view the complete class below.
+
+.. code-block:: Python
+    class USPhoneNumberTransformer(BaseTransformer):
+
+        INPUT_TYPE = 'phone_number'
+        DETERMINISTIC_TRANSFORM = True
+        DETERMINISTIC_REVERSE = True
+        COMPOSITION_IS_IDENTITY = True
+
+        def __init__(self):
+            self.has_country_code = None
+        
+        def _fit(self, columns_data):
+            number = ''.join(columns_data.loc[0].split('-'))
+            self.has_country_code = len(number) == 11
+
+        def get_output_types(self):
+            output_types = {
+                'area_code': 'categorical',
+                'exchange': 'integer',
+                'line': 'integer'
+            }
+            if self.has_country_code:
+                output_types['country_code'] = 'categorical'
+
+            return self._add_prefix(output_types)
+
+        def get_next_transformers(self):
+            next_transformers = {
+                'country_code': 'CategoricalTransformer',
+                'area_code': 'CategoricalTransformer'
+            }
+            if self.has_country_code:
+                next_transformers['country_code'] = 'CategoricalTransformer'
+
+            return self._add_prefix(next_transformers)
+        
+        def _transform(self, data):
+            return data.str.split('-', expand=True)
+
+        def _reverse_transform(self, data):
+            if self.has_country_code:
+                country_code = data.iloc[:, 0].astype('str')
+                area_code = data.iloc[:, 1].astype('str')
+                exchange = data.iloc[:, 2].astype('str')
+                line = data.iloc[:, 3].astype('str')
+                return country_code + '-' + area_code + '-' + exchange + '-' + line
+            
+            area_code = data.iloc[:, 0].astype('str')
+            exchange = data.iloc[:, 1].astype('str')
+            line = data.iloc[:, 2].astype('str')
+            return area_code + '-' + exchange + '-' + line
+
+Now we can see our `USPhoneNumberTransformer` in action.
+
+.. code-block:: Python
+
+    In [1]: transformer = USPhoneNumberTransformer()
+            data = pd.DataFrame({
+                'phone_numbers': ['1-202-555-0191', '1-202-555-0151', '1-202-867-5309']
+            })
+            transformer.fit(data, ['phone_numbers'])
+            transformed = transformer.transform(data)
+    
+    In [2]: transformed
+    Out [2]:
+        phone_numbers.area_code	phone_numbers.exchange	phone_numbers.line	phone_numbers.country_code
+    0	                      1	                   202	               555	                      0191
+    1	                      1	                   202	               555	                      0151
+    2	                      1	                   202	               867	                      5309
+    
+    In [3] reverse_transformed = transformer.reverse_transform(transformed)
+
+    In [4] reverse_transformed
+    Out [4]
+            phone_numbers
+    0	   1-202-555-0191
+    1	   1-202-555-0151
+    2	   1-202-867-5309
+
+We can also run it using the `HyperTransformer`.
+
+.. code-block:: Python
+
+    In [1]: ht = HyperTransformer(
+                data_type_transformers={'phone_number': USPhoneNumberTransformer},
+                field_types={'phone_numbers': 'phone_number'}
+            )
+            ht.fit(data)
+            transformed = ht.transform(data)
+
+    In [2]: transformed
+    Out [2]:
+        phone_numbers.area_code.value	phone_numbers.exchange	phone_numbers.line	phone_numbers.country_code.value
+    0	                          0.5	                   202	               555	                        0.500000
+    1	                          0.5	                   202	               555	                        0.166667
+    2	                          0.5	                   202	               867	                        0.833333
+
+    In [3]: reverse_transformed = ht.reverse_transform(transformed)
+
+    In [4]: reverse_transformed
+    Out [4]:
+            phone_numbers
+    0	   1-202-555-0191
+    1	   1-202-555-0151
+    2	   1-202-867-5309
+
+Dataset Generators
+------------------
+
+In RDT, performance tests are run to assure that each transformer is efficient. In order to run
+these tests, we have classes that generate datasets of a certain data type. If a new transformer
+introduces a new data type, the a ``DatasetGenerator`` class will need to be added for it.
+
+BaseDatasetGenerator
+""""""""""""""""""""
+
+All dataset generators inherit from the ``BaseDatasetGenerator`` class. It has the following
+class attribute:
+
+* ``DATA_TYPE`` (str) - The data type for the class to generate.
+
+They must implement the following methods.
+
+* ``generate(num_rows)`` - Takes in an int representing the number of rows to generate. Returns a
+  ``numpy.ndarray`` of size ``num_rows`` where each value is of the class' ``DATA_TYPE``.
+
+* ``get_performance_thresholds()`` - Returns a dict mapping each of the main methods for a
+  transformer (``fit``, ``transform``, ``reverse_transform``) to the expected time and memory it
+  takes for those methods to run on 1 row.
+
+Implementing a DatasetGenerator
+"""""""""""""""""""""""""""""""
+
+To create a new ``DatasetGenerator``, the methods described above need to be implemented. The
+class should be placed in a new file in the following location ``tests/datasets/{DATA_TYPE}.py``.
+Each generator must inherit from the base class as well as ``abc.ABC``.
+
+Example DatasetGenerator
+************************
+
+Let's create a ``DatasetGenerator`` for the ``phone_number`` data type that we introduced earlier.
+We can start by implementing the ``generate`` method and setting the ``DATA_TYPE``.
+
+.. code-block:: Python
+
+    from abc import ABC
+
+    import numpy as np
+
+    from tests.datasets.base import BaseDatasetGenerator
+
+    class USPhoneNumberGenerator(BaseDatasetGenerator, ABC):
+        DATA_TYPE = 'phone_number'
+        
+        @staticmethod
+        def generate(num_rows):
+            area_codes = np.random.randint(low=100, high=999, size=num_rows).astype(str)
+            exchange = np.random.randint(low=100, high=999, size=num_rows).astype(str)
+            line = np.random.randint(low=1000, high=9999, size=num_rows).astype(str)
+            return np.apply_along_axis('-'.join, 0, [area_codes, exchange, line])
+
+In order for the tests to run, the generator must also implement the ``get_performance_thresholds``
+method. The times are specified in seconds and the memory in bytes.
+
+.. code-block:: Python
+
+    @staticmethod
+    def get_performance_thresholds():
+        """Return the expected threseholds."""
+        return {
+            'fit': {
+                'time': 1,
+                'memory': 100.0
+            },
+            'transform': {
+                'time': 1,
+                'memory': 1000.0
+            },
+            'reverse_transform': {
+                'time': 1,
+                'memory': 1000.0,
+            }
+        }
+
+To view the result of the generator we can run the following:
+
+.. code-block:: Python
+
+    In [1]: USPhoneNumberGenerator.generate(100)
+    Out [1]:
+    array(['160-919-7653', '347-212-8425', '717-820-4356', '483-675-6853',
+       '656-141-2176', '681-981-5310', '314-989-4289', '138-343-6582',
+       '406-683-8597', '639-156-5496', '625-600-1649', '110-477-8992',
+       '770-731-6200', '166-491-9881', '418-682-9540', '889-169-1878',
+       '660-213-4713', '270-506-9422', '323-691-2507', '189-158-5409',
+       '605-218-6776', '944-980-8854', '773-290-6675', '969-724-8712',
+       '617-979-3609', '145-828-6455', '570-923-8982', '260-800-5404',
+       '301-453-3972', '454-629-5258', '298-394-6958', '700-285-1703',
+       '439-683-2711', '935-387-1178', '151-643-7354', '549-741-6070',
+       '617-142-6518', '759-653-4626', '482-778-1256', '909-538-2919',
+       '772-617-8616', '691-559-2419', '274-200-5514', '744-163-6255',
+       '760-709-7880', '909-782-6044', '826-607-6956', '902-609-2589',
+       '345-796-8422', '818-867-9468', '430-906-3757', '143-788-5794',
+       '340-705-3813', '211-447-7218', '912-799-7431', '840-211-5830',
+       '752-600-1938', '236-659-2646', '591-946-1546', '903-564-4356',
+       '928-847-8630', '315-775-9896', '384-323-8186', '192-282-8873',
+       '861-497-3333', '839-304-2029', '674-261-5948', '721-642-9755',
+       '761-787-2193', '429-720-9832', '126-876-2681', '327-533-3443',
+       '170-210-5689', '916-945-8487', '619-332-6223', '515-453-5862',
+       '509-666-4074', '231-687-8172', '489-862-2525', '602-456-5236',
+       '549-936-9406', '471-989-5828', '424-436-1012', '405-996-8833',
+       '786-811-5453', '851-897-7043', '462-381-9671', '328-267-1474',
+       '482-171-7564', '245-353-7712', '589-535-6689', '864-252-5314',
+       '990-737-8649', '112-189-9047', '126-316-8627', '985-724-3452',
+       '119-612-8449', '456-529-1190', '344-956-1910', '125-962-2067'],
+      dtype='<U12')


Test patch:
diff --git a/tests/contributing.py b/tests/contributing.py
index 43dc570c0..c8350211c 100644
--- a/tests/contributing.py
+++ b/tests/contributing.py
@@ -382,7 +382,6 @@ def validate_transformer_quality(transformer):
     a score comparing the transformer's average score for the dataset to
     the average of the average score for the dataset across all transformers of
     the same data type, and whether or not the score passed the test threshold.
-    3. The table described above is printed when this function is run.
 
     Returns:
         DataFrame containing the following columns for each dataset the transformer
@@ -416,9 +415,7 @@ def validate_transformer_quality(transformer):
     else:
         print('Failure: The quality tests were NOT successful.\n')
 
-    print(tabulate(transformer_results, headers='keys', showindex=False))
-
-    return transformer_results
+    return transformer_results.reset_index(drop=True)
 
 
 def validate_transformer_performance(transformer):
diff --git a/tests/quality/test_quality.py b/tests/quality/test_quality.py
index 9fc764d4c..f2e2df0b8 100644
--- a/tests/quality/test_quality.py
+++ b/tests/quality/test_quality.py
@@ -59,6 +59,26 @@ def find_columns(data, data_type, metadata=None):
 
 
 def get_transformer_regression_scores(data, data_type, dataset_name, transformers, metadata=None):
+    """Returns regression scores for a list of transformers.
+
+    Args:
+        data (pandas.DataFrame):
+            The dataset containing columns to predict and train with.
+        data_type (string):
+            The data type of the transformer.
+        dataset_name (string):
+            The name of the dataset.
+        transformers (list):
+            List of transformer instances.
+        metadata (dict):
+            Dictionary containing metadata for the table.
+
+    Returns:
+        pandas.DataFrame containing the score for each column predicted
+        in the dataset. To get the scores, a regression model is trained.
+        The features used are the output of transforming all the columns
+        of the data type using a transformer in the transformers list.
+    """
     columns_to_predict = find_columns(data, 'numerical')
     columns_to_transform = find_columns(data, data_type, metadata)
     scores = pd.DataFrame()