Repository: KwstasMCPU/BambooTools (language: python)

Issue:
Update README.md



Discussion / hints:

Fix patch:
diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml
index 64947b6..bd6952d 100644
--- a/.github/workflows/python-publish.yml
+++ b/.github/workflows/python-publish.yml
@@ -1,7 +1,7 @@
 # This workflow will build and test the BambooTools package when a push is made. It will
-# only publish to PiPy if a new tag is found
+# only publish to PyPi if a new tag is found
 
-name: Publish and test Python package to PiPy
+name: Publish and test Python package to PyPi
 
 on: push
 
@@ -44,5 +44,3 @@ jobs:
         TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
       run: |
           twine upload dist/*
-
-
diff --git a/.gitignore b/.gitignore
index 9d109c7..b3379a3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -165,4 +165,4 @@ TODO
 #  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
\ No newline at end of file
+#.idea/
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
new file mode 100644
index 0000000..d70bcda
--- /dev/null
+++ b/.pre-commit-config.yaml
@@ -0,0 +1,23 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+-   repo: https://github.com/psf/black
+    rev: 23.10.1
+    hooks:
+    -   id: black
+        args: [--safe, "--line-length=79"]
+
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+    -   id: trailing-whitespace
+    -   id: end-of-file-fixer
+    -   id: check-yaml
+    -   id: check-added-large-files
+        language_version: python3
+
+-   repo: https://github.com/PyCQA/flake8
+    rev: 6.1.0
+    hooks:
+    -   id: flake8
+        exclude: (^bin/|tests)
diff --git a/CHANGES.md b/CHANGES.md
new file mode 100644
index 0000000..3199737
--- /dev/null
+++ b/CHANGES.md
@@ -0,0 +1,51 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [v.0.4.0-beta] - 2023-11-04
+
+### Added
+
+- `duplication_summary()` method has been added. It produces a table with metrics regarding the duplicated records of the given dataset [#19](https://github.com/KwstasMCPU/BambooTools/issues/19).
+- `duplication_frequency_table()` method has been added. It shows how many times a duplicated record appear per the number of its clones [#19](https://github.com/KwstasMCPU/BambooTools/issues/19).
+
+### Changed
+
+- renamed the output columns of `completeness()`:
+  - `'perc': 'completeness ratio'`
+  - `'count': 'complete values'`
+- `format` argument of `completeness()` method has been depreciated.
+- `_conditional_probability()` internal class method converted to a standalone internal function.
+- Docstring style finalized to `numpy`.
+
+### Fixed
+- Fixed `MemoryError` bug in `completeness()` caused in cases of datasets with many columns [#20](https://github.com/KwstasMCPU/BambooTools/issues/20).
+- Removed unnecessary print statemens from `missing_corr_matrix()` [#21](https://github.com/KwstasMCPU/BambooTools/issues/21).
+
+## [v.0.3.0-beta] - 2023-10-02
+
+### Added
+
+- `missing_corr_matrix()` method has been added. It calculates the conditional
+probability of a column's value being NULL if another column's value is NULL.
+
+- `format` argument was added in the `completeness_summary()` method. If True
+the output is formated with pandas Styler.
+
+### Changed
+-  The outlier detector functions have been renamed to indicate that they
+are intended for internal use. A single leading underscore in front of the functions
+name has been added:
+
+    * outlier_detector_std-> _outlier_detector_std
+    * outlier_detector_iqr -> _outlier_detector_iqr
+    * outlier_detector_percentiles -> _outlier_detector_percentiles
+
+### Fixed
+- Fixed bug in outlier_summary() where categorical columns were also present
+in the dataset.
+
+## [v.0.2.0-alpha] - 2023-09-24
+
+### Added
+- Initial release.
diff --git a/CHANGES.txt b/CHANGES.txt
deleted file mode 100644
index bb0c2b4..0000000
--- a/CHANGES.txt
+++ /dev/null
@@ -1,32 +0,0 @@
-# Changelog
-
-All notable changes to this project will be documented in this file.
-
-## [v.0.3.0-beta] - 2023-10-02
-
-### Added
-
-- `missing_corr_matrix()` method has been added. It calculates the conditional 
-probability of a column's value being NULL if another column's value is NULL.
-
-- `format` argument was added in the `completeness_summary()` method. If True
-the output is formated with pandas Styler.
-
-### Changed
--  The outlier detector functions have been renamed to indicate that they 
-are intended for internal use. A single leading underscore in front of the functions 
-name has been added:
-
-    * outlier_detector_std-> _outlier_detector_std
-    * outlier_detector_iqr -> _outlier_detector_iqr
-    * outlier_detector_percentiles -> _outlier_detector_percentiles
-
-### Fixed
-- Fixed bug in outlier_summary() where categorical columns were also present
-in the dataset
-
-## [v.0.2.0-alpha] - 2023-09-24
-
-### Added
-- Initial release
-
diff --git a/README.md b/README.md
index 4c48037..a37deb3 100644
--- a/README.md
+++ b/README.md
@@ -6,7 +6,7 @@ With BambooTools, you can easily identify and handle outliers in your data, enab
 
 ## Installation
 
-Install from PiPy
+Install from PyPi
 
 ```bash
 pip install BambooTools
@@ -44,30 +44,29 @@ df = pd.DataFrame({'Animal': ['Falcon', 'Falcon',
 # check the completeness of the dataset per column
 print(df.bbt.completeness())
 ```
-|           | perc               | count | total |
-|-----------|--------------------|-------|-------|
-| Animal    | 1.0                | 6     | 6     |
-| Max Speed | 0.6666666666666666 | 4     | 6     |
-| Weight    | 0.6666666666666666 | 4     | 6     |
+|           | complete values | completeness ratio | total |
+|-----------|-----------------|--------------------|-------|
+| Animal    | 6               | 1.0                | 6     |
+| Max Speed | 4               | 0.6666666666666666 | 6     |
+| Weight    | 4               | 0.6666666666666666 | 6     |
 
 Specifying a list of categorical columns would result the completeness per category:
 ```python
 # check the completeness of the datataset per category
 print(df.bbt.completeness(by=['Animal']))
 ```
-|        | Max Speed   |           |           | Weight      |        |        |
-|--------|-------------|-----------|-----------|-------------|--------|--------|
-| Animal | perc        | count     | total     | perc        | count  | total  |
-|        |             |           |           |             |        |        |
-| Falcon | 0.666666667 | 2         | 3         | 0.666666667 | 2      | 3      |
-| Lama   | 0           | 0         | 1         | 1           | 1      | 1      |
-| Parrot | 1           | 2         | 2         | 0.5         | 1      | 2      |
+|        | Max Speed       |                    |       | Weight          |                    |       |
+|--------|-----------------|--------------------|-------|-----------------|--------------------|-------|
+| Animal | complete values | completeness ratio | total | complete values | completeness ratio | total |
+| Falcon | 2               | 0.6666666666666666 | 3     | 2               | 0.6666666666666666 | 3     |
+| Lama   | 0               | 0.0                | 1     | 1               | 1.0                | 1     |
+| Parrot | 2               | 1.0                | 2     | 1               | 0.5                | 2     |
 
 ## Missing values correlation matrix
 `missing_corr_matrix()` This matrix aims to help to pintpoint relationships between missing values of different columns. Calculates
-the conditional probability of a column's value being NaN, given the fact another column value is NaN.
+the conditional probability of a record's value being NaN in a specific column, given the fact another value for the same record is missing at a different column.
 
-For a dataset with two columns `'A', 'B'` the conditional probability of a value from column `'A'` being NaN is:
+For a dataset with two columns `'A', 'B'` the conditional probability of a record having a missing value at column `'A'` is:
 
 $$P(A \text{ is NULL } | B \text{ is NULL}) = \frac{P(A \text{ is NULL } \cap B \text{ is NULL})}{P(B \text{ is NULL})}$$
 
@@ -174,6 +173,45 @@ print(penguins.bbt.outlier_bounds(method='iqr', by=['sex', 'species'], factor=1)
 | **Male**   | **Adelie**    | 36.5           | 44             | 17.4          | 20.7          | 181               | 205               | 3300        | 4800        |
 | **Male**   | **Chinstrap** | 48.125         | 53.9           | 17.8          | 20.8          | 189               | 210               | 3362.5      | 4468.75     |
 | **Male**   | **Gentoo**    | 45.7           | 52.9           | 14.3          | 17            | 211               | 232               | 4900        | 6100        |
+
+## Duplication summary
+
+`duplication_summary()` returns metrics regarding the duplicate records of the given dataset. It states the number of total rows, unique rows, unique rows without duplications, unique records with duplications and total duplicated records:
+
+```python
+print(penguins.bbt.duplication_summary(subset=['sex',
+                                               'species',
+                                               'island']))
+```
+|                                     | counts |
+|-------------------------------------|--------|
+| total records                       | 344    |
+| unique records                      | 13     |
+| unique records without duplications | 1      |
+| unique records with duplications    | 12     |
+| total duplicated records            | 343    |
+
+## Duplication frequency table
+`duplication_frequency_table` generates a table which states the frequency of records with duplications. Categorizes the duplicated records according to their number of duplications, and reports the frequency of those categories.
+
+In the example below, we notice that there are 2 cases of 5 identical records.
+
+```python
+print(penguins.bbt.duplication_frequency_table(subset=['sex',
+                                                       'species',
+                                                       'island']))
+```
+| n identical bins | frequency | sum of duplications | percentage to total duplications |
+|------------------|-----------|---------------------|----------------------------------|
+| 2                | 0         | 0                   | 0                                |
+| 3                | 0         | 0                   | 0                                |
+| 4                | 0         | 0                   | 0                                |
+| 5                | 2         | 10                  | 0.029154519                      |
+| [6, 10)          | 0         | 0                   | 0                                |
+| [10, 15)         | 0         | 0                   | 0                                |
+| [15, 50)         | 8         | 214                 | 0.623906706                      |
+| 50>              | 2         | 119                 | 0.346938776                      |
+
 # Contributing
 
 Contributions are more than welcome! You can contribute with several ways:
@@ -182,32 +220,38 @@ Contributions are more than welcome! You can contribute with several ways:
 * Recommendations for new features and implementation of those
 * Writing and or improving existing tests, to ensure quality
 
-Prior any contributions, opening an issue is recommended.
+**Prior yout contribution, opening an issue is recommended.**
 
 It is also recommended to install the package in ["development mode"](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#working-in-development-mode) while working on it. *When installed as editable, a project can be edited in-place without reinstallation.*
 
-To install a Python package in "editable"/"development" mode change directory to the root of the project directory and run:
+To install the Python package in "editable"/"development" mode, change directory to the root of the project directory and run:
 
 ```bash
 pip install -e .
 pip install -r requirements-dev.txt # this will install the development dependencies (e.g. pytest)
 ```
 
-In order to install the package and the development dependencies with a one liner, run the below:
+OR in order to install the package and the development dependencies with a one liner, run the below:
 
 ```bash
 pip install -e ".[dev]"
 ```
 
+To ensure that the development workflow is followed, please also setup the pre-commit hooks:
+
+```bash
+pre-commit install
+```
+
 ## General Guidelines
 
 1. Fork the repository on GitHub.
 2. Clone the forked repository to your local machine.
 3. Make a new branch, from the `develop` branch for your feature or bug fix.
-4. Implement your changes. 
+4. Implement your changes.
    - It is recommended to write tests and examples for them in `tests\test_bambootols.py` and `bin\examples.py` respectively.
-1. Create a Pull Request. Link it to the issue you have opened.
+5. Create a Pull Request. Link it to the issue you have opened.
 
 # Credits
 
-Special thanks to [danikavu](https://github.com/danikavu) for the code reviews
\ No newline at end of file
+Special thanks to [danikavu](https://github.com/danikavu) for the code reviews
diff --git a/bambootools/bambootools.py b/bambootools/bambootools.py
index 588411d..969b324 100644
--- a/bambootools/bambootools.py
+++ b/bambootools/bambootools.py
@@ -2,11 +2,86 @@
 :copyright: (c) 2023 by Maravegias Konstantinos.
 :license: MIT, see LICENSE for more details.
 """
-import pandas as pd
-import numpy as np
 from itertools import combinations
 from typing import List, Tuple, Literal
 
+import pandas as pd
+import numpy as np
+
+
+def _conditional_probability(
+    data: pd.DataFrame, col_a: str, col_b: str
+) -> Tuple[float, float]:
+    """
+    Calculates the probability of a column's value being NULL given the
+    fact another's column value is NULL (conditional probability).
+
+    Parameters
+    ----------
+    _df : DataFrame
+        The dataset holding the column to examine.
+    col_a : str
+        Column name of the first column.
+    col_b : str
+        Column name of the second column.
+
+    Returns
+    -------
+    Tuple[float, float]
+        The conditional probabilities for the values of `col_a` and `col_b`
+        being NULL respectively.
+    """
+
+    col_a_mask = data[col_a].isna()
+    col_b_mask = data[col_b].isna()
+    both_na = (col_a_mask & col_b_mask).sum()
+    p_a_being_na = col_a_mask.sum() / data.shape[0]
+    p_b_being_na = col_b_mask.sum() / data.shape[0]
+    p_both_being_na = both_na / data.shape[0]
+    # check for for possible division with zero warning
+    # column b has no NaN
+    if p_b_being_na == 0:
+        p_a_conditional = np.nan
+    else:
+        p_a_conditional = p_both_being_na / p_b_being_na
+    # column a has no NaN
+    if p_a_being_na == 0:
+        p_b_conditional = np.nan
+    else:
+        p_b_conditional = p_both_being_na / p_a_being_na
+    return p_a_conditional, p_b_conditional
+
+
+def _hash_table(df: pd.DataFrame, subset: List[str] = None) -> pd.Series:
+    """Returns a data hash of the given DataFrame, excluding the index.
+
+    Parameters
+    ----------
+    df : DataFrame
+        The dataframe to be hashed
+    subset :  list of a column label or sequence of labels, optinal
+        The subset of columns to generate the hash series.
+
+    Returns
+    -------
+    Series of uint64
+        Series with same length as the object.
+
+    Raises
+    ------
+    AttributeError
+        Raises error if `subset` is not a list.
+    """
+
+    if not subset:
+        subset = df.columns.to_list()
+    else:
+        if not isinstance(subset, List):
+            raise AttributeError("`subset` arg must be a list of strings")
+
+    hashed_series = pd.util.hash_pandas_object(df[subset], index=False)
+    return hashed_series
+
 
 @pd.api.extensions.register_dataframe_accessor("bbt")
 class BambooToolsDfAccessor:
@@ -19,102 +94,105 @@ def pandas_obj(self) -> pd.DataFrame:
         return self._obj
 
     @staticmethod
-    def _validate(obj) -> None:
-        """Validate that the passed object is a pandas Dataframe
+    def _validate(obj: pd.DataFrame) -> None:
+        """
+        Validates if the passed object is a pandas Dataframe
 
-        Args:
-            * obj: Object passed
+        Parameters
+        ----------
+        obj : Dataframe
 
-        Raises:
-            AttributeError: Prompts the user that pd.DataFrame should be used
+        Raises
+        ------
+        AttributeError
+            Raises error if the object is not a pandas DataFrame.
         """
+
         if not isinstance(obj, pd.DataFrame):
             raise AttributeError("Must be a pandas DataFrame")
 
-    def completeness(self, by: List[str] = None,
-                     format: bool = False) -> pd.DataFrame:
-        """Returns the completeness table of a dataframe. The returned columns
-        are: `count`, `perc` and `total`.
-            `count`: the non NULL values
-            `perc`: the ratio of total records to the `count`
-            `total`: the total records
-
-        Args:
-            * by (List[str], optional): List of columns to aggregate. If
-                passed the completeness per column is measured per group.
-                Defaults to None.
-            * format (bool, optional): If True the perc column is formated
-                to a percentage (eg 50.00%). Note that it returns a pandas
-                Styler object instead of a DataFrame.
-
-        Raises:
-            AttributeError: Checks if by is a list of string
-
-        Returns:
-            pd.DataFrame: The completeness dataframe
+    def completeness(self, by: List[str] = None) -> pd.DataFrame:
+        """
+        Returns the completeness table of a dataframe. The returned columns
+        are:
+            * `complete values`: the number non NULL values per column.
+            * `completeness ratio`: the ratio of `complete values` to the
+              total number of records (column's length).
+            * `total`: the total number of records.
+
+        Parameters
+        ----------
+        by : list of columns, default None
+            The list of column to aggragate. Produces the completeness metrics
+            per the groups specified on `by`.
+
+        Returns
+        -------
+        DataFrame
+            The completeness table.
+
+        Raises
+        ------
+        AttributeError
+            Raises error in case of wrong input for `by` argument.
         """
+
         if by is None:
-            by = self._obj.columns.to_list()
-            counts = self._obj.groupby(by, dropna=False).\
-                apply(lambda x: x.notnull().sum()).sum()
-
-            perc = self._obj.groupby(by, dropna=False).\
-                apply(lambda x: x.notnull().sum()).sum() / self._obj.shape[0]
-
-            _df = pd.concat([counts, perc], axis=1).\
-                rename(columns={0: 'count',
-                                1: 'perc'
-                                }
-                       )
-            _df['total'] = self._obj.shape[0]
-
-            if format:
-                _df = _df.style.format({'perc': '{:.2%}'})
+            complete_values = self._obj.notnull()
+            total_records = self._obj.shape[0]
+            counts = complete_values.sum()
+            counts.name = "complete values"
+            perce = complete_values.sum() / total_records
+            perce.name = "completeness ratio"
+            output = counts.to_frame().join(perce)
+            output["total"] = total_records
         else:
             if not isinstance(by, List):
                 raise AttributeError("`by` arg must be a list of strings")
 
-            _df = self._obj.groupby(by, dropna=False).\
-                agg([('count', lambda x: x.notnull().sum()),
-                     ('perc', lambda x: x.notnull().sum()/x.shape[0]),
-                     ('total', 'size')
-                     ]
-                    )
-            if format:
-                format_dict = dict([(t, '{:.2%}') for t in
-                                    _df.columns if t[1] == 'perc'])
-                _df = _df.style.format(format_dict)
-        return _df
+            output = self._obj.groupby(by, dropna=False).agg(
+                [
+                    ("complete values", lambda x: x.notnull().sum()),
+                    (
+                        "completeness ratio",
+                        lambda x: x.notnull().sum() / x.shape[0],
+                    ),
+                    ("total", "size"),
+                ]
+            )
+        return output
 
     def missing_corr_matrix(self) -> pd.DataFrame:
-        """Returns a missing correlations matrix. Calculates the conditional
-        probability of a column being NULL given the fact another column is
-        NULL.
+        """
+        Returns the missing correlations matrix. Calculates the conditional
+        probability of a record's value being NULL at a specific colunm given
+        the fact, another's column value is missing for the same record.
 
         A missing correlation matrix is a table, which states for every column
-        the probability of its values being NULL given the fact another's
-        column values are NULL. In more details, if comparing columns A and B,
-        it is the ratio between the number of records where both columns are
-        NULL, and the total number of NULL values in column B.
-
-        Note that the ratio between A and B is not equal to the ratio between
-        B and A.
+        the above mentioned contidional probability. In more details, if
+        examining the conditional probability of a record being NULL at column
+        A compared to another column B, it is the ratio between the probability
+        of the values of the same record being NULL in both the A and B columns
+        (intersection), and the probability of a record being NULL at column B.
 
         `P(A is NULL | B is NULL) = P(A is NULL & B is NULL) / P(B is NULL)`
 
-        Returns:
-            pd.DataFrame: Returns an n x n matrix, with n equals the number of
-                the initial dataframe's columns.
+        Returns
+        -------
+        DataFrame
+            Returns an `n x n` matrix, with `n` equals the number of
+            the initial dataframe's columns.
         """
+
         _df = self._obj
         columns_pairs_comb = list(combinations(_df.columns, 2))
         pairs_dict = {}
         for col_a, col_b in columns_pairs_comb:
             # calculate the conditional probabilities for each columns
             # pair
-            result = self._conditional_probability(_df=_df,
-                                                   col_a=col_a,
-                                                   col_b=col_b)
+            result = _conditional_probability(
+                data=_df, col_a=col_a, col_b=col_b
+            )
             if col_a in pairs_dict:
                 pairs_dict[col_a].update({col_b: result[0]})
             else:
@@ -127,295 +205,475 @@ def missing_corr_matrix(self) -> pd.DataFrame:
         matrix = pd.DataFrame(pairs_dict)
         return matrix.reindex(matrix.columns)
 
-    def outlier_bounds(self, method: Literal['std', 'iqr', 'percentiles'],
-                       std_n: float = 3.0, factor: float = 1.5,
-                       lower_thresh: float = 0.0, upper_thresh: float = 1.0,
-                       by: List = None
-                       ) -> pd.DataFrame:
-        """Returns the outlier boundaries of the given dataframe for every
+    def duplication_summary(self, subset: List[str] = None) -> pd.DataFrame:
+        """
+        Generates a duplication summary table. Calculates the number and
+        percentage of duplicate rows.
+
+        Summary table explained:
+
+            * `total records`: Refers to the total row of the dataset.
+            * `unique records`: Refers to the number of the unique records
+            of the dataset.
+            * `unique records without duplications`: Refers to the number of
+            unique records which have no duplications.
+            * `unique records with duplications`: Refers to the number of
+            unique records which have duplications.
+            * `total duplicated records`: Referns to the number of the total
+            duplicated records.
+
+        Parameters
+        ----------
+        subset : list of a column label or sequence of labels, optional
+            Only consider certain columns for identifying duplicates, by
+            default use all of the columns.
+
+        Returns
+        -------
+        DataFrame
+            The duplication summary table.
+        """
+
+        _df = self.pandas_obj.copy()
+        hashed_series = _hash_table(_df, subset)
+        del _df
+
+        total_records = len(hashed_series)
+        n_unique_records = hashed_series.nunique()
+        n_total_duplicate_records = hashed_series.duplicated(keep=False).sum()
+        frequency_of_records = hashed_series.value_counts()
+        n_unique_duplicate_records = (frequency_of_records > 1).sum()
+        n_unique_non_duplicated_records = (frequency_of_records == 1).sum()
+
+        output = pd.DataFrame(
+            index=[
+                "total records",
+                "unique records",
+                "unique records without duplications",
+                "unique records with duplications",
+                "total duplicated records",
+            ],
+            columns=["counts"],
+            data=[
+                total_records,
+                n_unique_records,
+                n_unique_non_duplicated_records,
+                n_unique_duplicate_records,
+                n_total_duplicate_records,
+            ],
+        )
+
+        return output
+
+    def duplication_frequency_table(
+        self, subset: List[str] = None
+    ) -> pd.DataFrame:
+        """
+        Generates a table which states the frequency of records with
+        duplications. Categorizes the duplicated records according to their
+        number of duplications, and reports the frequency of those categories.
+
+        E.g.: if a record has 1 identical record (so 2 in including itself)
+        it is classed in the `2` category (`n identical bins` column.) If there
+        are 10 of those pairs, the frequency is `10` and they account for
+        20 duplications (`sum of duplications` equals 20).
+
+        Frequency table explained:
+
+            * `n identical bins`: States the category of the duplicated
+            records. `2` accounts for pairs of duplicates (two indentical
+            records), `3` for triples, etc.
+            * `frequency`: The frequency of the `d bins`.
+            * `sum of duplications`: States for how many duplicated records
+            those categories generate.
+            * `percentage to total duplications`: Is ratio between the
+            * `sum of duplications` and to total number of duplicated values.
+
+        Parameters
+        ----------
+        subset : list of a column label or sequence of labels, optional
+            Only consider certain columns for identifying duplicates, by
+            default use all of the columns.
+
+        Returns
+        -------
+        DataFrame
+            A table with metrics regarding the duplicated columns.
+        """
+
+        _df = self.pandas_obj.copy()
+        hashed_series = _hash_table(_df, subset)
+        del _df
+
+        frquency_table = (
+            hashed_series.value_counts()
+            .value_counts()
+            .sort_index()
+            .to_frame(name="frequency")
+        )
+        frquency_table["sum of duplications"] = (
+            frquency_table["frequency"] * frquency_table.index
+        )
+        frquency_table["n identical bins"] = pd.cut(
+            frquency_table.index,
+            bins=[2, 3, 4, 5, 6, 10, 15, 50, np.inf],
+            include_lowest=True,
+            labels=[
+                "2",
+                "3",
+                "4",
+                "5",
+                "[6, 10)",
+                "[10, 15)",
+                "[15, 50)",
+                "50>",
+            ],
+            right=False,
+        )
+        frquency_table.dropna(subset=["n identical bins"], inplace=True)
+
+        output = frquency_table.groupby(
+            ["n identical bins"], observed=False
+        ).sum()
+        output["percentage to total duplications"] = (
+            output["sum of duplications"] / output["sum of duplications"].sum()
+        )
+        return output
+
+    def outlier_bounds(
+        self,
+        method: Literal["std", "iqr", "percentiles"],
+        std_n: float = 3.0,
+        factor: float = 1.5,
+        lower_thresh: float = 0.0,
+        upper_thresh: float = 1.0,
+        by: List["str"] = None,
+    ) -> pd.DataFrame:
+        """
+        Returns the outlier boundaries of the given dataframe for every
         numerical column. Outlier boundaries are defined as the values for
-        which any other value above or below is considered as an outlier.
-
-        Args:
-            * method ({'std', 'iqr', 'percentiles'}): Which outlier method
-                to be used.
-                - 'std': Calculates the mean and standard deviation. A value
-                becomes an outlier if exceeds the mean more than `std_n`
-                standard deviations. Use if you assume that your data are
-                normally distributed.
-                - 'iqr': Calculates the IQR, then considers as an outlier
-                every value being `factor`*IQR below or upper the 25%, 75%
-                percentiles respectively.
-                - 'percentiles': Detects as outliers, every value being below
-                or upper the given percentiles.
-            * std_n (float, optional): The number of standard deviations to be
-                used in the `std` method. Defaults to 3.0.
-            * factor (float, optional): The factor of IQR to be used in the
-                `iqr` method. Defaults to 1.5.
-            * lower_thresh (float, optional): The lower percentile threshold
-                to be used in the `percentiles` method. Defaults to 0.0.
-            * upper_thresh (float, optional): The upper percentile threshold
-                to be used in the `percentiles` method. Defaults to 1.0.
-            * by (List[str], optional): If the names of categorical columns are
-                given, then the boundaries are calculated per group.
-
-        Returns:
-            pd.DataFrame: A table of the lower and upper boundary values
+        which any other value above or below is considered an outlier.
+
+        Parameters
+        ----------
+        method : {'std', 'iqr', 'percentiles'}
+            Determines which outlier method to be used.
+
+            * `std`: Calculates the mean and standard deviation. A value
+            becomes an outlier if exceeds the mean for more than `std_n`
+            standard deviations. Use if you assume that your data are
+            normally distributed.
+            * `iqr`: Calculates the IQR, then considers as an outlier
+            every value being `factor`*IQR below or upper the 25%, 75%
+            percentiles respectively.
+            * `percentiles`: Detects as outliers, every value being below
+            or upper the given percentiles.
+
+        std_n : float, default 3.0
+            The number of standard deviations to be used in order to determine
+            if a value is an outlier. Used in the `std` method.
+        factor : float, default 1.5
+            The factor of IQR to be used in order to determine if a value is
+            and outlier. Used in the `iqr` method.
+        lower_thresh : float, default 0.0
+            The lower percentile threshold to be used in the `percentiles`
+            method.
+        upper_thresh : float, default 1.0
+            The upper percentile threshold to be used in the `percentiles`
+            method.
+        by : list of columns, default None
+            The list of column to aggragate. Produces the outlier bounds
+            metrics per the groups specified on `by`.
+
+        Returns
+        -------
+        DataFrame
+            The dataframe table with the boundaries per column.
         """
+
         _df = self.pandas_obj.copy()
         # check if to group by per with any categorical column
         if by:
             # group by per categorical column
             if not isinstance(by, list):
                 raise AttributeError("`by` arg must be a list of strings")
-            cols = _df.select_dtypes(exclude=['category', 'object']).columns
+            cols = _df.select_dtypes(exclude=["category", "object"]).columns
             _df = _df.groupby(by)[cols]
             # select and call outlier method:
-            if method == 'std':
-                bounds = _df.apply(lambda group: group.apply(
-                    self._outlier_detector_std, std_n=std_n)).unstack()
-            if method == 'iqr':
-                bounds = _df.apply(lambda group: group.apply(
-                    self._outlier_detector_iqr, factor=factor)).unstack()
-            if method == 'percentiles':
-                bounds = _df.apply(lambda group: group.apply(
-                    self._outlier_detector_percentiles,
-                    lower_thresh=lower_thresh,
-                    upper_thresh=upper_thresh
-                    )).unstack()
+            if method == "std":
+                bounds = _df.apply(
+                    lambda group: group.apply(
+                        self._outlier_detector_std, std_n=std_n
+                    )
+                ).unstack()
+            if method == "iqr":
+                bounds = _df.apply(
+                    lambda group: group.apply(
+                        self._outlier_detector_iqr, factor=factor
+                    )
+                ).unstack()
+            if method == "percentiles":
+                bounds = _df.apply(
+                    lambda group: group.apply(
+                        self._outlier_detector_percentiles,
+                        lower_thresh=lower_thresh,
+                        upper_thresh=upper_thresh,
+                    )
+                ).unstack()
             return bounds
 
         else:
             # do not group by per any categorical column
             # select and call outlier method:
-            _df = _df.select_dtypes(exclude=['category', 'object'])
-            if method == 'std':
+            _df = _df.select_dtypes(exclude=["category", "object"])
+            if method == "std":
                 bounds = self._outlier_detector_std(_df, std_n)
-            if method == 'iqr':
+            if method == "iqr":
                 bounds = self._outlier_detector_iqr(_df, factor)
-            if method == 'percentiles':
-                bounds = self._outlier_detector_percentiles(_df,
-                                                            lower_thresh,
-                                                            upper_thresh
-                                                            )
-            return pd.DataFrame({'lower': bounds['lower'],
-                                 'upper': bounds['upper']
-                                 }
-                                )
-
-    def outlier_summary(self, method: Literal['std', 'iqr', 'percentiles'],
-                        std_n: float = 3.0, factor: float = 1.5,
-                        lower_thresh: float = 0.0, upper_thresh: float = 1.0,
-                        by: List = None
-                        ) -> pd.DataFrame:
-        """Returns an outlier summary table (counts of outliers) for upper and
-        lower bounds. Utilises existing functions.
-
-        Args:
-            * method ({'std', 'iqr', 'percentiles'}): Which outlier method
-                to be used.
-                - 'std': Calculates the mean and standard deviation. A value
-                becomes an outlier if exceeds the mean more than `std_n`
-                standard deviations. Use if you assume that your data are
-                normally distributed.
-                - 'iqr': Calculates the IQR, then considers as an outlier
-                every value being `factor`*IQR below or upper the 25%, 75%
-                percentiles respectively.
-                - 'percentiles': Detects as outliers, every value being below
-                or upper the given percentiles.
-            * std_n (float, optional): The number of standard deviations to be
-                used in the `std` method. Defaults to 3.0.
-            * factor (float, optional): The factor of IQR to be used in the
-                `iqr` method. Defaults to 1.5.
-            * lower_thresh (float, optional): The lower percentile threshold
-                to be used in the `percentiles` method. Defaults to 0.0.
-            * upper_thresh (float, optional): The upper percentile threshold
-                to be used in the `percentiles` method. Defaults to 1.0.
-            * by (List[str], optional): If the names of categorical columns are
-                given, then the boundaries are calculated per group.
-
-        Returns:
-            pd.DataFrame: The outlier summary table
+            if method == "percentiles":
+                bounds = self._outlier_detector_percentiles(
+                    _df, lower_thresh, upper_thresh
+                )
+            return pd.DataFrame(
+                {"lower": bounds["lower"], "upper": bounds["upper"]}
+            )
+
+    def outlier_summary(
+        self,
+        method: Literal["std", "iqr", "percentiles"],
+        std_n: float = 3.0,
+        factor: float = 1.5,
+        lower_thresh: float = 0.0,
+        upper_thresh: float = 1.0,
+        by: List = None,
+    ) -> pd.DataFrame:
         """
-        bounds = self.outlier_bounds(method, std_n, factor,
-                                     lower_thresh, upper_thresh, by)
+        Generates an outlier summary table. The outlier summary table produces
+        metrics regarding the outliers values of the dataset. It uses different
+        methods to define a values as an outlier.
+
+        Outlier summary table explained:
+
+            * n_outliers_upper: The number of outliers existing above the upper
+            limit.
+            * n_outliers_lower: The number of outliers existing below the lower
+            limit.
+            * n_non_outliers: The number of non outlier values.
+            * total_records: The total records of the dataset.
+
+        Parameters
+        ----------
+        method : {'std', 'iqr', 'percentiles'}
+            Determines which outlier method to be used.
+
+            * `std`: Calculates the mean and standard deviation. A value
+            becomes an outlier if exceeds the mean for more than `std_n`
+            standard deviations. Use if you assume that your data are
+            normally distributed.
+            * `iqr`: Calculates the IQR, then considers as an outlier
+            every value being `factor`*IQR below or upper the 25%, 75%
+            percentiles respectively.
+            * `percentiles`: Detects as outliers, every value being below
+            or upper the given percentiles.
+
+        std_n : float, default 3.0
+            The number of standard deviations to be used in order to determine
+            if a value is an outlier. Used in the `std` method.
+        factor : float, default 1.5
+            The factor of IQR to be used in order to determine if a value is
+            and outlier. Used in the `iqr` method.
+        lower_thresh : float, default 0.0
+            The lower percentile threshold to be used in the `percentiles`
+            method.
+        upper_thresh : float, default 1.0
+            The upper percentile threshold to be used in the `percentiles`
+            method.
+        by : list of columns, default None
+            The list of column to aggragate. Produces the outlier bounds
+            metrics per the groups specified on `by`.
+
+        Returns
+        -------
+        DataFrame
+            The outlier summary dataframe.
+        """
+
+        bounds = self.outlier_bounds(
+            method, std_n, factor, lower_thresh, upper_thresh, by
+        )
         outlier_counts = {}
         _df = self.pandas_obj.copy()
-        cols = _df.select_dtypes(exclude=['category', 'object']).columns
+        cols = _df.select_dtypes(exclude=["category", "object"]).columns
         if by:
             # --> groupby summary table (group)
             for group in bounds.index:
                 for col in cols:
-                    lower_bound = bounds.loc[group, (col, 'lower')]
-                    upper_bound = bounds.loc[group, (col, 'upper')]
+                    lower_bound = bounds.loc[group, (col, "lower")]
+                    upper_bound = bounds.loc[group, (col, "upper")]
                     # Count the outliers below/above the lower/upper bounds
-                    lower_outliers = _df[((_df[by] == group).all(axis=1)
-                                          & (_df[col] < lower_bound))]
-                    upper_outliers = _df[((_df[by] == group).all(axis=1)
-                                          & (_df[col] > upper_bound))]
+                    lower_outliers = _df[
+                        (
+                            (_df[by] == group).all(axis=1)
+                            & (_df[col] < lower_bound)
+                        )
+                    ]
+                    upper_outliers = _df[
+                        (
+                            (_df[by] == group).all(axis=1)
+                            & (_df[col] > upper_bound)
+                        )
+                    ]
                     # count the non outliers
-                    non_outliers = _df[((_df[by] == group).all(axis=1)
-                                        & (_df[col] <= upper_bound)
-                                        & (_df[col] >= lower_bound))]
+                    non_outliers = _df[
+                        (
+                            (_df[by] == group).all(axis=1)
+                            & (_df[col] <= upper_bound)
+                            & (_df[col] >= lower_bound)
+                        )
+                    ]
                     # Store the counts in the dictionary
-                    outlier_counts[(group,
-                                    col,
-                                    'n_outliers_lower')] = len(lower_outliers)
-                    outlier_counts[(group,
-                                    col,
-                                    'n_outliers_upper')] = len(upper_outliers)
-                    outlier_counts[(group,
-                                    col,
-                                    'n_non_outliers')] = len(non_outliers)
+                    outlier_counts[(group, col, "n_outliers_lower")] = len(
+                        lower_outliers
+                    )
+                    outlier_counts[(group, col, "n_outliers_upper")] = len(
+                        upper_outliers
+                    )
+                    outlier_counts[(group, col, "n_non_outliers")] = len(
+                        non_outliers
+                    )
             # generate the summary table
             summary_tbl = pd.Series(outlier_counts).unstack()
         else:
             # --> non groupby summary table
-            _df = _df.select_dtypes(exclude=['category', 'object'])
-            lower_outliers = _df.lt(bounds['lower'], axis=1).sum()
-            upper_outliers = _df.gt(bounds['upper'], axis=1).sum()
+            _df = _df.select_dtypes(exclude=["category", "object"])
+            lower_outliers = _df.lt(bounds["lower"], axis=1).sum()
+            upper_outliers = _df.gt(bounds["upper"], axis=1).sum()
             # count the non outliers
-            non_outliers = (_df.le(bounds['upper'], axis=1)
-                            & _df.ge(bounds['lower'], axis=1)).sum()
+            non_outliers = (
+                _df.le(bounds["upper"], axis=1)
+                & _df.ge(bounds["lower"], axis=1)
+            ).sum()
             # concat all the series into one dataframe
-            summary_tbl = pd.concat([upper_outliers,
-                                     lower_outliers,
-                                     non_outliers], axis=1
-                                    ).rename(columns={
-                                        0: 'n_outliers_upper',
-                                        1: 'n_outliers_lower',
-                                        2: 'n_non_outliers'
-                                        }
-                                             )
+            summary_tbl = pd.concat(
+                [upper_outliers, lower_outliers, non_outliers], axis=1
+            ).rename(
+                columns={
+                    0: "n_outliers_upper",
+                    1: "n_outliers_lower",
+                    2: "n_non_outliers",
+                }
+            )
         # add additional columns in the summary table
-        qry_total_outliers = 'n_outliers_upper + n_outliers_lower'
-        summary_tbl['n_total_outliers'] = summary_tbl.eval(qry_total_outliers)
-        qry_total_records = 'n_non_outliers + n_total_outliers'
-        summary_tbl['total_records'] = summary_tbl.eval(qry_total_records)
+        qry_total_outliers = "n_outliers_upper + n_outliers_lower"
+        summary_tbl["n_total_outliers"] = summary_tbl.eval(qry_total_outliers)
+        qry_total_records = "n_non_outliers + n_total_outliers"
+        summary_tbl["total_records"] = summary_tbl.eval(qry_total_records)
         return summary_tbl
 
-    def _outlier_detector_std(self,
-                              _df: pd.DataFrame = None,
-                              std_n: float = 3.0
-                              ) -> pd.Series:
-        """Returns the upper and lower boundaries which are used to class a
+    def _outlier_detector_std(
+        self, _df: pd.DataFrame = None, std_n: float = 3.0
+    ) -> pd.Series:
+        """
+        Returns the upper and lower boundaries which are used to class a
         value as an outlier. The boundaries are caclulated as `std_n` times
         away from the mean.
-        Note: Not suitable for data which do not follow normal distribution.
 
-        Args:
-            * _df (pd.DataFrame, optional): If called iternally it is passed as
-                an argument. Defaults to None.
-            * std_n (float, optional): The number of standard deviations to be
-                used in the `std` method. Defaults to 3.0.
+        Note: It is not suitable for data which do not follow normal
+        distribution.
+
+        Parameters
+        ----------
+        _df : DataFrame, default None
+            If called iternally it is passed as an argument.
+        std_n : float, default 3.0
+            The number of standard deviations to be used in the `std` method.
 
-        Returns:
-            pd.Series: The lower and upper bound
+        Returns
+        -------
+        Series
+            The lower and upper outlier boundaries.
         """
+
         if _df is None:
             _df = self._obj
-            _df = _df.select_dtypes(exclude=['category', 'object'])
+            _df = _df.select_dtypes(exclude=["category", "object"])
         data_mean = _df.mean(numeric_only=True)
         data_std = _df.std(numeric_only=True)
         cut_off = data_std * std_n
         lower_bound = data_mean - cut_off
         upper_bound = data_mean + cut_off
-        return pd.Series({'lower': lower_bound, 'upper': upper_bound})
-
-    def _outlier_detector_iqr(self,
-                              _df: pd.DataFrame = None,
-                              factor: float = 1.5
-                              ) -> pd.Series:
-        """Returns the upper and lower boundaries which are used to class a
-        value as an outlier. The boundaries are defined as the points, which
+        return pd.Series({"lower": lower_bound, "upper": upper_bound})
+
+    def _outlier_detector_iqr(
+        self, _df: pd.DataFrame = None, factor: float = 1.5
+    ) -> pd.Series:
+        """
+        Returns the upper and lower boundaries which are used to class a
+        value as an outlier. The boundaries are defined as the values, which
         are `factor` times the IQR below and above the Q1 and Q3 quartiles
         respectively.
 
-        Args:
-            * _df (pd.DataFrame, optional): If called iternally it is passed as
-                an argument. Defaults to None.
-            * std_n (float, factor): The multiplayer of the IQR.
-                Defaults to 1.5.
-
-        Returns:
-            pd.Series: The lower and upper bound
+        Parameters
+        ----------
+        _df : DataFrame, default None
+            If called iternally it is passed as an argument.
+        factor : float, default 1.5
+            The multiplayer of the IQR.
+
+        Returns
+        -------
+        Series
+            The lower and upper outlier boundaries.
         """
+
         if _df is None:
             _df = self._obj
-            _df = _df.select_dtypes(exclude=['category', 'object'])
+            _df = _df.select_dtypes(exclude=["category", "object"])
         q25 = _df.quantile(0.25)
         q75 = _df.quantile(0.75)
         iqr = q75 - q25
         cut_off = iqr * factor
         lower_bound = q25 - cut_off
         upper_bound = q75 + cut_off
-        return pd.Series({'lower': lower_bound, 'upper': upper_bound})
-
-    def _outlier_detector_percentiles(self,
-                                      _df: pd.DataFrame = None,
-                                      lower_thresh: float = 0.0,
-                                      upper_thresh: float = 1.0
-                                      ) -> pd.Series:
-        """Returns the upper and lower boundaries which are used to class a
-        value as an outlier. The boundaries are calculated as the returd values
+        return pd.Series({"lower": lower_bound, "upper": upper_bound})
+
+    def _outlier_detector_percentiles(
+        self,
+        _df: pd.DataFrame = None,
+        lower_thresh: float = 0.0,
+        upper_thresh: float = 1.0,
+    ) -> pd.Series:
+        """
+        Returns the upper and lower boundaries which are used to class a
+        value as an outlier. The boundaries are defined as the values
         at the given `lower_thesh` and `upper_thresh` percentiles.
 
-        Args:
-            * _df (pd.DataFrame, optional): If called iternally it is passed as
-                an argument. Defaults to None.
-            * lower_thresh: Value between 0 <= q <= 1, the percentile to
-                compute for the lower boundary value.
-            * upper_thresh: Value between 0 <= q <= 1, the percentile to
-                compute for the upper boundary value.
-
-        Returns:
-            pd.Series: The lower and upper bound
+        Parameters
+        ----------
+        _df : DataFrame, default None
+            If called iternally it is passed as an argument.
+        lower_thresh : float, between [0, 1], default 0.0
+            The percentile to compute for the lower boundary value.
+        upper_thresh : float, between [0, 1], default 0.0
+            The percentile to compute for the upper boundary value.
+
+        Returns
+        -------
+        Series
+            The lower and upper outlier boundaries.
         """
+
         if _df is None:
             _df = self._obj
-            _df = _df.select_dtypes(exclude=['category', 'object'])
+            _df = _df.select_dtypes(exclude=["category", "object"])
+        if lower_thresh < 0 | lower_thresh > 1:
+            raise ValueError("Lower threshold should be within [0, 1]")
+        if upper_thresh < 0 | upper_thresh > 1:
+            raise ValueError("Upper threshold should be within [0, 1]")
+
         lower_bound = _df.quantile(lower_thresh)
         upper_bound = _df.quantile(upper_thresh)
-        return pd.Series({'lower': lower_bound, 'upper': upper_bound})
-
-    def _conditional_probability(self,
-                                 _df: pd.DataFrame,
-                                 col_a: str,
-                                 col_b: str) -> Tuple[float, float]:
-        """Calculates the probability of a column's value being NULL given the
-        fact another's column value is NULL (conditional probability).
-
-        Args:
-            * _df (pd.DataFrame): The dataframe holding the both columns
-            * col_a (str): Column name of the first column
-            * col_b (str): Column name of the second column
-
-        Returns:
-            Tuple[float, float]: The conditional probabilities for a value of
-                `col_a` and `col_b` being NULL respectively
-        """
-        col_a_mask = _df[col_a].isna()
-        col_b_mask = _df[col_b].isna()
-        both_na = (col_a_mask & col_b_mask).sum()
-        p_a_being_na = col_a_mask.sum() / _df.shape[0]
-        p_b_being_na = col_b_mask.sum() / _df.shape[0]
-        p_both_being_na = both_na / _df.shape[0]
-        # check for for possible division with zero warning
-        msg = """`{}` has no NULL values.
-        Conditional probability for `{}` set to NaN."""
-        # column b has no NaN
-        if p_b_being_na == 0:
-            print(msg.format(col_b, col_a))
-            p_a_conditional = np.nan
-        else:
-            p_a_conditional = p_both_being_na / p_b_being_na
-        # column a has no NaN
-        if p_a_being_na == 0:
-            print(msg.format(col_a, col_b))
-            p_b_conditional = np.nan
-        else:
-            p_b_conditional = p_both_being_na / p_a_being_na
-        return p_a_conditional, p_b_conditional
+        return pd.Series({"lower": lower_bound, "upper": upper_bound})
 
 
 @pd.api.extensions.register_series_accessor("bbt")
@@ -430,33 +688,44 @@ def series_obj(self) -> pd.Series:
 
     @staticmethod
     def _validate(obj) -> None:
-        """Validate that the passed object is a pandas Series
+        """
+        Validates if the passed object is a pandas Series
 
-        Args:
-            * obj: Object passed
+        Parameters
+        ----------
+        obj : Series
 
-        Raises:
-            AttributeError: Prompts the user that pd.Series should be used
+        Raises
+        ------
+        AttributeError
+            Raises error if the object is not a pandas Series.
         """
+
         if not isinstance(obj, pd.Series):
             raise AttributeError("Must be a pandas Series")
 
-    def above(self, thresh: float,
-              dropna: bool = False) -> Tuple[float, float]:
-        """Calculates the number of values and their percentage which are
+    def above(
+        self, thresh: float, dropna: bool = False
+    ) -> Tuple[float, float]:
+        """
+        Calculates the number of values and their percentage which are
         above a specific threshold.
 
-        Args:
-            * thresh (float): The threshold given by the user.
-            * dropna (bool, optional): If True drops the NULL records before
-                the calculation of percentage of NULL values. Hence the total
-                number of records equal to the number of non NULL values.
-                Defaults to False.
-
-        Returns:
-            Tuple[float, float]: Counts and percentage of records above the
-                given threshold.
+        Parameters
+        ----------
+        thresh : float
+            The threshold given by the user.
+        dropna : bool, default False
+             If True drops the NULL records before the calculation of
+             percentage of NULL values. Hence the total number of records
+             equal to the number of non NULL values.
+
+        Returns
+        -------
+        Tuple[float, float]
+            Counts and percentage of records above the given threshold.
         """
+
         if dropna:
             values_above = self._obj > thresh
         else:
@@ -466,21 +735,26 @@ def above(self, thresh: float,
 
         return count, perc
 
-    def below(self, thresh: float,
-              dropna: bool = False) -> Tuple[float, float]:
-        """Calculates the number of values and their percentage which are
-        below a specific threshold.
-
-        Args:
-            * thresh (float): The threshold given by the user.
-            * dropna (bool, optional): If True drops the NULL records before
-                the calculation of percentage of NULL values. Hence the total
-                number of records equal to the number of non NULL values.
-                Defaults to False.
-
-        Returns:
-            Tuple[float, float]: Counts and percentage of records below the
-                given threshold.
+    def below(
+        self, thresh: float, dropna: bool = False
+    ) -> Tuple[float, float]:
+        """
+        Calculates the number of values and their percentage which are
+        beloq a specific threshold.
+
+        Parameters
+        ----------
+        thresh : float
+            The threshold given by the user.
+        dropna : bool, default False
+             If True drops the NULL records before the calculation of
+             percentage of NULL values. Hence the total number of records
+             equal to the number of non NULL values.
+
+        Returns
+        -------
+        Tuple[float, float]
+            Counts and percentage of records below the given threshold.
         """
 
         if dropna:
diff --git a/bin/examples.py b/bin/examples.py
index e7628e5..2c51d2a 100644
--- a/bin/examples.py
+++ b/bin/examples.py
@@ -1,56 +1,80 @@
-from bambootools import bambootools
 import pandas as pd
 import numpy as np
 import seaborn as sns
 
-# Set a seed for reproducibility
-np.random.seed(0)
+from bambootools import bambootools
+
+
+def generate_dataset():
+    # Set a seed for reproducibility
+    np.random.seed(0)
+
+    # Define the number of records
+    n_records = 50
 
-# Define the number of records
-n_records = 50
+    # Define the categories for the 'animal' column
+    animals = ["cat", "dog", "lama"]
 
-# Define the categories for the 'animal' column
-animals = ['cat', 'dog', 'lama']
+    # Generate random data
+    df = pd.DataFrame(
+        {
+            "animal": np.random.choice(animals, n_records),
+            "color": np.random.choice(
+                ["black", "white", "brown", "gray"], n_records
+            ),
+            "weight": np.random.randint(1, 100, n_records),
+            "tail length": np.random.randint(1, 50, n_records),
+            "height": np.random.randint(10, 500, n_records),
+            "date_of_observation": pd.date_range(
+                start="2022-01-01", end="2022-02-01", periods=n_records
+            ),
+        }
+    )
 
-# Generate random data
-df = pd.DataFrame({
-    'animal': np.random.choice(animals, n_records),
-    'color': np.random.choice(['black', 'white', 'brown', 'gray'], n_records),
-    'weight': np.random.randint(1, 100, n_records),
-    'tail length': np.random.randint(1, 50, n_records),
-    'height': np.random.randint(10, 500, n_records)
-})
+    # Insert NULL values in the 'animal', 'color', 'weight', 'tail length' and
+    # 'height' columns
+    for col, n_nulls in zip(df.columns, [2, 15, 20, 48, 17, 0]):
+        null_indices = np.random.choice(df.index, n_nulls, replace=False)
+        df.loc[null_indices, col] = np.nan
 
-# Insert NULL values in the 'animal', 'color', 'weight', 'tail length' and
-# 'height' columns
-for col, n_nulls in zip(df.columns, [2, 15, 20, 48, 17]):
-    null_indices = np.random.choice(df.index, n_nulls, replace=False)
-    df.loc[null_indices, col] = np.nan
+    return df
+
+
+df = generate_dataset()
 
 # get dataset's completeness for each column
 print(df.bbt.completeness())
 
 # get dataset's completeness per group
-print(df.bbt.completeness(by=['animal']))
+print(df.bbt.completeness(by=["animal"]))
 
 # get missing correlation matrix
-print(df.bbt.missing_corr_matrix().to_clipboard())
+print(df.bbt.missing_corr_matrix())
 
 # outlier boundaries per group
 penguins = sns.load_dataset("penguins")
-print(penguins.bbt.outlier_bounds(method='std',
-                                  by=['sex', 'species']))
+print(penguins.bbt.outlier_bounds(method="std", by=["sex", "species"]))
 
 # outliers summary
-print(penguins.bbt.outlier_summary(method='std'))
+print(penguins.bbt.outlier_summary(method="std"))
 
 # outliers summary per group
-print(penguins.bbt.outlier_summary(method='iqr',
-                                   by=['sex', 'species'],
-                                   factor=1))
+print(
+    penguins.bbt.outlier_summary(method="iqr", by=["sex", "species"], factor=1)
+)
+
+# get duplication summary
+print(penguins.bbt.duplication_summary(subset=["sex", "species", "island"]))
+
+# get duplication frequency table
+print(
+    penguins.bbt.duplication_frequency_table(
+        subset=["sex", "species", "island"]
+    )
+)
 
 # find how many values and their percentage which are above a threshold
-print(df['weight'].bbt.above(thresh=30))
+print(df["weight"].bbt.above(thresh=30))
 
 # find how many values and their percentage which are below a threshold
-print(df['weight'].bbt.below(thresh=30))
+print(df["weight"].bbt.below(thresh=30))
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 5b31af3..391d511 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -2,3 +2,5 @@ pytest>=7.4.2
 pandas>=1.5.3
 twine
 seaborn
+build
+pre-commit==3.5.0
diff --git a/setup.py b/setup.py
index 428ff41..a26147f 100644
--- a/setup.py
+++ b/setup.py
@@ -3,31 +3,48 @@
 with open("README.md", "r") as f:
     long_description = f.read()
 
+project_urls = {
+    "Documentation": "https://github.com/KwstasMCPU/BambooTools",
+    "Changelog": "https://github.com/KwstasMCPU/BambooTools/releases",
+    "Source": "https://github.com/KwstasMCPU/BambooTools",
+}
+
 setup(
-    name='BambooTools',
-    version='0.3.0',
-    author='Konstantinos Maravegias',
-    author_email='kwstas.maras@gmail.com',
+    name="BambooTools",
+    version="0.4.0",
+    author="Konstantinos Maravegias",
+    author_email="kwstas.maras@gmail.com",
     packages=find_packages(),
-    url='https://github.com/KwstasMCPU/BambooTools',
-    license='MIT',
-    description='Pandas extension to enchance your data analysis.',
+    url="https://github.com/KwstasMCPU/BambooTools",
+    license="MIT",
+    description="Pandas extension to enchance your data analysis.",
     long_description=long_description,
-    long_description_content_type='text/markdown',
-    install_requires=[
-        "pandas>=1.5.3"
-        ],
+    long_description_content_type="text/markdown",
+    install_requires=["pandas>=1.5.3"],
     extras_require={
-        "dev": ["pytest>=7.4.2", "twine", "seaborn"]
-        },
-    classifiers=["Development Status :: 4 - Beta",
-                 "Programming Language :: Python :: 3",
-                 "Intended Audience :: Developers",
-                 "Intended Audience :: Science/Research",
-                 "License :: OSI Approved :: MIT License"
-                 ],
-    keywords=['BambooTools', 'pandas', 'pandas extensions',
-              'data analysis', 'data science'
-              ],
-    python_requires='>=3.8'
+        "dev": [
+            "pytest>=7.4.2",
+            "twine",
+            "seaborn",
+            "build",
+            "pre-commit==3.5.0",
+        ]
+    },
+    classifiers=[
+        "Development Status :: 4 - Beta",
+        "Programming Language :: Python :: 3",
+        "Intended Audience :: Developers",
+        "Intended Audience :: Science/Research",
+        "License :: OSI Approved :: MIT License",
+    ],
+    keywords=[
+        "BambooTools",
+        "pandas",
+        "pandas extensions",
+        "data analysis",
+        "data science",
+        "analytics",
+    ],
+    project_urls=project_urls,
+    python_requires=">=3.8",
 )


Test patch:
diff --git a/tests/test_bambootools.py b/tests/test_bambootools.py
index db5a90d..f34954a 100644
--- a/tests/test_bambootools.py
+++ b/tests/test_bambootools.py
@@ -1,9 +1,10 @@
 import pytest
-from bambootools import bambootools
 import pandas as pd
 import numpy as np
 import seaborn as sns
 
+from bambootools import bambootools
+
 
 @pytest.fixture
 def animals_dataset():
@@ -14,21 +15,23 @@ def animals_dataset():
     n_records = 20
 
     # Define the categories for the 'animal' column
-    animals = ['cat', 'dog', 'lama']
+    animals = ["cat", "dog", "lama"]
 
     # Generate random data for the 'animal', 'weight', n 'tail length' columns
-    df = pd.DataFrame({
-        'animal': np.random.choice(animals, n_records),
-        'weight': np.random.randint(1, 100, n_records),
-        'tail length': np.random.randint(1, 100, n_records),
-        'color': np.random.choice(['black', 'white',
-                                   'brown', 'gray'],
-                                  n_records),
-        'name': [f'name_{i}' for i in range(n_records)]
-    })
+    df = pd.DataFrame(
+        {
+            "animal": np.random.choice(animals, n_records),
+            "weight": np.random.randint(1, 100, n_records),
+            "tail length": np.random.randint(1, 100, n_records),
+            "color": np.random.choice(
+                ["black", "white", "brown", "gray"], n_records
+            ),
+            "name": [f"name_{i}" for i in range(n_records)],
+        }
+    )
 
     # Insert NULL values in the 'weight', 'tail length' and 'name' columns
-    for col, n_nulls in zip(['weight', 'tail length', 'name'], [3, 5, 1]):
+    for col, n_nulls in zip(["weight", "tail length", "name"], [3, 5, 1]):
         null_indices = np.random.choice(df.index, n_nulls, replace=False)
         df.loc[null_indices, col] = np.nan
     return df
@@ -41,68 +44,92 @@ def penguins_dataset():
 
 # tests for BambooToolsDfAccessor
 def test_init_dataframe(animals_dataset):
-    assert animals_dataset.equals(animals_dataset.bbt.pandas_obj), (
-        "Expected equal dataframe.")
+    assert animals_dataset.equals(
+        animals_dataset.bbt.pandas_obj
+    ), "Expected equal dataframe."
 
 
 def test_completeness(animals_dataset):
     result = animals_dataset.bbt.completeness()
     assert result.shape == (5, 3), "Wrong table dimensions."
-    assert result['count'].max() <= animals_dataset.shape[0], (
-        "Max value of non missing cannot exceed total number of records.")
-    assert result['perc'].max() <= 1.0, (
-        "Max value of perc cannot exceed 1.")
+    assert (
+        result["complete values"].max() <= animals_dataset.shape[0]
+    ), "Max value of `complete values` cannot exceed total records."
+    assert (
+        result["completeness ratio"].max() <= 1.0
+    ), "Max value of `completeness ratio` cannot exceed 1."
 
 
 def test_completeness_per_group(animals_dataset):
-    result = animals_dataset.bbt.completeness(by=['animal'])
+    result = animals_dataset.bbt.completeness(by=["animal"])
     assert result.shape == (3, 12), "Wrong table dimensions."
-    assert result['weight']['perc'].max() <= 1.0, (
-        "Max value of perc cannot exceed 1.")
+    assert (
+        result["weight"]["completeness ratio"].max() <= 1.0
+    ), "Max value of perc cannot exceed 1."
     # test if the counts were calculated correctly
-    n_cats = animals_dataset['animal'].value_counts()['cat']
-    assert result['weight'].loc['cat', 'total'] == n_cats, (
-        "Total counts per category must equal the value counts."
-    )
+    n_cats = animals_dataset["animal"].value_counts()["cat"]
+    assert (
+        result["weight"].loc["cat", "total"] == n_cats
+    ), "Total counts per category must equal the value counts."
 
 
 def test_missing_corr_matrix(animals_dataset):
     result = animals_dataset.bbt.missing_corr_matrix()
     n_columns = animals_dataset.shape[1]
-    assert result.shape == (n_columns, n_columns), (
-        "A {}x{} martix was expected".format(n_columns, n_columns)
-    )
+    assert result.shape == (
+        n_columns,
+        n_columns,
+    ), "A {}x{} martix was expected".format(n_columns, n_columns)
 
 
 def test_outlier_summary(penguins_dataset):
-    result = penguins_dataset.bbt.outlier_summary(method='iqr',
-                                                  by=['sex', 'species']
-                                                  )
+    result = penguins_dataset.bbt.outlier_summary(
+        method="iqr", by=["sex", "species"]
+    )
     assert result.shape == (24, 5), "Shape should be (24, 5). sex*species."
-    assert result['n_outliers_lower'].sum() == 8, "N of outliers should be 8."
+    assert result["n_outliers_lower"].sum() == 8, "N of outliers should be 8."
     # test if the counts were calculated correctly
-    n_penguins_grp = penguins_dataset.groupby(['sex', 'species']).size()
-    assert n_penguins_grp['Female', 'Adelie'] == result['total_records'].\
-        loc[('Female', 'Adelie'), 'bill_depth_mm'], (
-            "Total counts per category must equal the original data counts."
-        )
+    n_penguins_grp = penguins_dataset.groupby(["sex", "species"]).size()
+    assert (
+        n_penguins_grp["Female", "Adelie"]
+        == result["total_records"].loc[("Female", "Adelie"), "bill_depth_mm"]
+    ), "Total counts per category must equal the original data counts."
+
+
+def test_duplication_summary(penguins_dataset):
+    subset = ["sex", "species", "island"]
+    result = penguins_dataset.bbt.duplication_summary(subset=subset)
+    total_duplicates = penguins_dataset.duplicated(
+        subset=subset, keep=False
+    ).sum()
+    assert result.shape == (5, 1), "Shape should be 5, 1."
+    assert (
+        result.loc["total records"].item() == penguins_dataset.shape[0]
+    ), "Total records should equal dataframes length."
+    assert (
+        result.loc["total duplicated records"].item() == total_duplicates
+    ), "Total duplicates must equal to result of pd.duplicated()"
+
+
+def test_duplication_frequency_table(penguins_dataset):
+    result = penguins_dataset.bbt.duplication_frequency_table()
+    assert result.shape == (8, 3), "Shape should be 8, 3"
 
 
 # tests for BambooToolsSeriesAccessor
 def test_init_series(animals_dataset):
-    assert animals_dataset['weight'].equals(
-        animals_dataset['weight'].bbt.series_obj), (
-        "Expected equal series."
-    )
+    assert animals_dataset["weight"].equals(
+        animals_dataset["weight"].bbt.series_obj
+    ), "Expected equal series."
 
 
 def test_above(animals_dataset):
-    result = animals_dataset['weight'].bbt.above(thresh=30)
+    result = animals_dataset["weight"].bbt.above(thresh=30)
     assert isinstance(result, tuple), "Did not return a tuple."
     assert result[0] == 10, "Value counts should be 10."
 
 
 def test_below(animals_dataset):
-    result = animals_dataset['weight'].bbt.below(thresh=30)
+    result = animals_dataset["weight"].bbt.below(thresh=30)
     assert isinstance(result, tuple), "Did not return a tuple."
     assert result[0] == 6, "Value counts should be 10."