Repository: dirty-cat/dirty_cat
Pull request #409 (2022-11-10 09:53:02)

FEA Adding FeatureAugmenter
After the discussion in #403, here is a first version of the Feature Augmenter.

This is a transformer that will allow the users to pass a main table and some auxiliary tables to fuzzy join with.

(I need to add an example before making it ready for review)

jovan-stojanovic: When using the FeatureAugmenter, I found out that tuning the `ngram_range` and `analyzer` parameters of the `fuzzy_join` may be also useful sometimes. Do you think it's a good idea to introduce them in the class as parameters? Or would this add too much complexity for a first version? @GaelVaroquaux @LilianBoulard
GaelVaroquaux: > When using the FeatureAugmenter, I found out that tuning the ngram_range and analyzer parameters of the fuzzy_join may be also useful sometimes. Do you think it's a good idea to introduce them in the class as parameters?

If you find them useful than yes, we should add them.

jovan-stojanovic: Ok, thanks, I added it to the FA.

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index ab91735..fad8b66 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -2,6 +2,9 @@
 
 .. currentmodule:: dirty_cat
 
+Release 0.5.0
+==============
+
 Release 0.4.0 (upcoming)
 =========================
 
@@ -11,6 +14,9 @@ Major changes
 * New experimental feature: joining tables using :func:`fuzzy_join` by approximate key matching. Matches are based
   on string similarities and the nearest neighbors matches are found for each category.
 :pr:`291` by :user:`Jovan Stojanovic <jovan-stojanovic>` and :user:`Leo Grinsztajn <LeoGrin>`
+* New experimental feature: :class:`FeatureAugmenter`, a transformer
+  that augments with :func:`fuzzy_join` the number of features in a main table
+  by using information from auxilliary tables. :pr:`409` by :user:`Jovan Stojanovic <jovan-stojanovic>`
 * **datasets.fetching**: contains a new function :func:`fetch_world_bank_indicator`
   that can be used to download any indicator from the World Bank Open Data platform.
   It only needs the indicator ID that can be found on the website. :pr:`291` by :user:`Jovan Stojanovic <jovan-stojanovic>`
diff --git a/dirty_cat/__init__.py b/dirty_cat/__init__.py
index b6bf6d3..40f30b8 100644
--- a/dirty_cat/__init__.py
+++ b/dirty_cat/__init__.py
@@ -16,6 +16,7 @@ except ModuleNotFoundError:
 
 from ._datetime_encoder import DatetimeEncoder
 from ._deduplicate import compute_ngram_distance, deduplicate
+from ._feature_augmenter import FeatureAugmenter
 from ._fuzzy_join import fuzzy_join
 from ._gap_encoder import GapEncoder
 from ._minhash_encoder import MinHashEncoder
@@ -28,13 +29,14 @@ with open(_Path(__file__).parent / "VERSION.txt") as _fh:
 
 
 __all__ = [
-    "SimilarityEncoder",
-    "TargetEncoder",
-    "MinHashEncoder",
-    "GapEncoder",
     "DatetimeEncoder",
-    "SuperVectorizer",
+    "FeatureAugmenter",
     "fuzzy_join",
+    "GapEncoder",
+    "MinHashEncoder",
+    "SimilarityEncoder",
+    "SuperVectorizer",
+    "TargetEncoder",
     "deduplicate",
     "compute_ngram_distance",
 ]
diff --git a/dirty_cat/_feature_augmenter.py b/dirty_cat/_feature_augmenter.py
new file mode 100644
index 0000000..7dcf36d
--- /dev/null
+++ b/dirty_cat/_feature_augmenter.py
@@ -0,0 +1,197 @@
+"""
+Transformer that allows multiple fuzzy joins to be performed on a table.
+The principle is as follows:
+  1. The main table and the key column name are provided at initialisation.
+  2. The auxilliary tables are provided for fitting, and will be joined sequentially
+  when the transform is called.
+It is advised to use hyper-parameter tuning tools such as scikit-learn's
+GridSearchCV to determine the best `match_score` parameter, as this can
+significantly improve your results.
+(see example 'Fuzzy joining dirty tables with the FeatureAugmenter' for an illustration)
+For more information on how the join is performed, see fuzzy_join's documentation.
+"""
+
+from typing import List, Literal, Tuple
+
+import pandas as pd
+from sklearn.base import BaseEstimator, TransformerMixin
+
+from dirty_cat._fuzzy_join import fuzzy_join
+
+
+class FeatureAugmenter(BaseEstimator, TransformerMixin):
+    """Transformer augmenting the number of features in a table by joining multiple tables.
+
+    Given a list of tables and key column names,
+    fuzzy join them to the main table.
+
+    Parameters
+    ----------
+    tables : list of 2-tuples of (:class:`~pandas.DataFrame`, str)
+        List of (table, column name) tuples
+        specyfying the transformer objects to be applied.
+        table: str
+            Name of the table to be joined.
+        column name: str,
+            Name of table column to join on.
+    main_key : str
+        The key column name in the main table on which
+        the join will be performed.
+    match_score : float, default=0
+        Distance score between the closest matches that will be accepted.
+        In a [0, 1] interval. 1 means that only a perfect match will be
+        accepted, and zero means that the closest match will be accepted,
+        no matter how distant.
+    analyzer : typing.Literal["word", "char", "char_wb"], default=`char_wb`
+        Analyzer parameter for the CountVectorizer used for the string
+        similarities.
+        Options: {`word`, `char`, `char_wb`}, describing whether the matrix V
+        to factorize should be made of word counts or character n-gram counts.
+        Option `char_wb` creates character n-grams only from text inside word
+        boundaries; n-grams at the edges of words are padded with space.
+    ngram_range : tuple (min_n, max_n), default=(2, 4)
+        The lower and upper boundary of the range of n-values for different
+        n-grams used in the string similarity. All values of n such
+        that min_n <= n <= max_n will be used.
+
+    Examples
+    --------
+    >>> X = pd.DataFrame(['France', 'Germany', 'Italy'], columns=['Country'])
+    >>> X
+    Country
+    0   France
+    1  Germany
+    2    Italy
+
+    >>> aux_table_1 = pd.DataFrame([['Germany', 84_000_000], ['France', 68_000_000], ['Italy', 59_000_000]], columns=['Country', 'Population']) # noqa
+    >>> aux_table_1
+       Country  Population
+    0  Germany    84000000
+    1   France    68000000
+    2    Italy    59000000
+
+    >>> aux_table_2 = pd.DataFrame([['French Republic', 2937], ['Italy', 2099], ['Germany', 4223], ['UK', 3186]], columns=['Country name', 'GDP (billion)']) # noqa
+    >>> aux_table_2
+        Country name  GDP (billion)
+    0   French Republic      2937
+    1        Italy           2099
+    2      Germany           4223
+    3           UK           3186
+
+    >>> aux_table_3 = pd.DataFrame([['France', 'Paris'], ['Italia', 'Rome'], ['Germany', 'Berlin']], columns=['Countries', 'Capital']) # noqa
+    >>> aux_table_3
+      Countries Capital
+    0    France   Paris
+    1     Italia   Rome
+    2   Germany  Berlin
+
+    >>> aux_tables = [(aux_table_1, "Country"), (aux_table_2, "Country name"), (aux_table_3, "Countries")] # noqa
+
+    >>> fa = FeatureAugmenter(tables=aux_tables, main_key='Country')
+
+    >>> augmented_table = fa.fit_transform(X)
+    >>> augmented_table
+        Country Country_aux  Population Country name  GDP (billion) Countries Capital
+    0   France      France    68000000  French Republic       2937    France   Paris
+    1  Germany     Germany    84000000      Germany           4223   Germany  Berlin
+    2    Italy       Italy    59000000        Italy           2099    Italia    Rome
+    """
+
+    def __init__(
+        self,
+        tables: List[Tuple[pd.DataFrame, str]],
+        main_key: str,
+        match_score: float = 0.0,
+        analyzer: Literal["word", "char", "char_wb"] = "char_wb",
+        ngram_range: Tuple[int, int] = (2, 4),
+    ):
+        self.tables = tables
+        self.main_key = main_key
+        self.match_score = match_score
+        self.analyzer = analyzer
+        self.ngram_range = ngram_range
+
+    def fit(self, X, y=None) -> "FeatureAugmenter":
+        """Fit the Feature Augmenter to the main table.
+
+        In practice, just checks if the key columns in X,
+        the main table, and in the auxilliary tables exist.
+
+        Parameters
+        ----------
+        X : DataFrame, shape [n_samples, n_features]
+            The main table, to be joined to the
+            auxilliary ones.
+        y : array-like of shape (n_samples,...), default=None
+            Targets for supervised learning.
+
+        Returns
+        -------
+        FeatureAugmenter
+            Fitted FeatureAugmenter instance.
+        """
+
+        if self.main_key not in X.columns:
+            raise ValueError(
+                f"Got main_key={self.main_key!r}, but column not in {list(X.columns)}."
+            )
+
+        for pairs in self.tables:
+            if pairs[1] not in pairs[0].columns:
+                raise ValueError(
+                    f"Got column key {pairs[1]!r}, "
+                    f"but column not in {pairs[0].columns}."
+                )
+        return self
+
+    def transform(self, X, y=None) -> pd.DataFrame:
+        """Transform X using the specified encoding scheme.
+
+        Parameters
+        ----------
+        X : DataFrame, shape [n_samples, n_features]
+            The main table, to be joined to the
+            auxilliary ones.
+        y : array-like of shape (n_samples,...), default=None
+            Targets for supervised learning.
+
+        Returns
+        -------
+        DataFrame
+            The final joined table.
+        """
+
+        for pairs in self.tables:
+            # TODO: Add an option to fuzzy_join on multiple columns at once
+            # (will be if len(inter_col)!=0)
+            aux_table = pairs[0]
+            X = fuzzy_join(
+                X,
+                aux_table,
+                left_on=self.main_key,
+                right_on=pairs[1],
+                match_score=self.match_score,
+                analyzer=self.analyzer,
+                ngram_range=self.ngram_range,
+                suffixes=("", "_aux"),
+            )
+        return X
+
+    def fit_transform(self, X, y=None) -> pd.DataFrame:
+        """
+        Fit the FeatureAugmenter to X, then transforms it.
+
+        Parameters
+        ----------
+        X : DataFrame, shape [n_samples, n_features]
+            The main table, to be joined to the
+            auxilliary ones.
+        y : array-like of shape (n_samples,...), default=None
+            Targets for supervised learning.
+
+        Returns
+        -------
+        DataFrame
+            The final joined table.
+        """
+        return self.fit(X).transform(X)
diff --git a/dirty_cat/_fuzzy_join.py b/dirty_cat/_fuzzy_join.py
index 9a3313a..00b442d 100644
--- a/dirty_cat/_fuzzy_join.py
+++ b/dirty_cat/_fuzzy_join.py
@@ -51,8 +51,8 @@ def fuzzy_join(
         A table to merge.
     right : :class:`~pandas.DataFrame`
         A table used to merge with.
-    how : {"left", "right"}, optional, default=`left`
-        Type of merge to be performed. Note that unlike pandas' merge,
+    how: typing.Literal["left", "right"], default=`left`
+        Type of merge to be performed. Note that unlike :func:`~pandas.merge`,
         only "left" and "right" are supported so far, as the fuzzy-join comes
         with its own mechanism to resolve lack of correspondence between
         left and right tables.
@@ -86,9 +86,9 @@ def fuzzy_join(
         the nearest matched categories.
     match_score : float, default=0.0
         Distance score between the closest matches that will be accepted.
-        In a [0, 1] interval. Closer to 1 means the matches need to be very
-        close to be accepted, and closer to 0 that a bigger matching distance
-        is tolerated.
+        In a [0, 1] interval. 1 means that only a perfect match will be
+        accepted, and zero means that the closest match will be accepted,
+        no matter how distant.
     drop_unmatched : boolean, default=False
         Remove categories for which a match was not found in the two tables.
     sort : boolean, default=False
diff --git a/doc/index.rst b/doc/index.rst
index 282059b..138b0d5 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -140,6 +140,7 @@ Joining tables
    :nosignatures:
 
    fuzzy_join
+   FeatureAugmenter
 
 Data download
 -------------
diff --git a/examples/05_joining_tables_with_FuzzyJoin.py b/examples/05_fuzzy_joining_with_FeatureAugmenter.py
similarity index 76%
rename from examples/05_joining_tables_with_FuzzyJoin.py
rename to examples/05_fuzzy_joining_with_FeatureAugmenter.py
index 6f1d561..94f5d6d 100755
--- a/examples/05_joining_tables_with_FuzzyJoin.py
+++ b/examples/05_fuzzy_joining_with_FeatureAugmenter.py
@@ -1,6 +1,6 @@
 """
-Merging dirty tables: fuzzy join
-================================
+Fuzzy joining dirty tables with the FeatureAugmenter
+=====================================================
 
 Here we show how to combine data from different sources,
 with a vocabulary not well normalized.
@@ -12,12 +12,17 @@ In this example, the |fj| function allows us to join
 tables without cleaning the data by taking into account the
 label variations.
 
+Moreover, the |fa| makes fuzzy joining of multiple tables with
+|fj| easy and adjustable with the introduction of a pipeline.
+
 To illustrate, we will join data from the `2022 World Happiness Report <https://worldhappiness.report/>`_.
 with tables provided in `the World Bank open data platform <https://data.worldbank.org/>`_
-in order to create a satisfying first prediction model.
+in order to create a first prediction model.
 
 
 .. |fj| replace:: :func:`~dirty_cat.fuzzy_join`
+
+.. |fa| replace:: :func:`~dirty_cat.FeatureAugmenter`
 """
 
 ###############################################################################
@@ -110,6 +115,11 @@ gdppc.sort_values(by="Country Name").tail(7)
 # To join them with dirty_cat, we only need to do the following:
 from dirty_cat import fuzzy_join
 
+# We will ignore the warnings:
+import warnings
+
+warnings.filterwarnings("ignore")
+
 df1 = fuzzy_join(
     df,  # our table to join
     gdppc,  # the table to join with
@@ -269,10 +279,6 @@ plt.show()
 ###############################################################################
 # 3. Joining legal rights strength table
 # ......................................
-# .. topic:: Note:
-#
-#    Here, we use the `keep='left'` option to keep only the left key matching
-#    column, so as not to have unnecessary overlaping column with country names.
 #
 # And the table with a measure of legal rights strength in the country:
 df3 = fuzzy_join(
@@ -341,10 +347,7 @@ cv_results_t = cross_validate(hgdb, X, y, cv=cv, scoring="r2")
 
 cv_r2_t = cv_results_t["test_score"]
 
-print(
-    f"Mean R2 score with {len(X.columns) - 2} feature columns is"
-    f" {cv_r2_t.mean():.2f} +- {cv_r2_t.std():.2f}"
-)
+print(f"Mean R2 score is {cv_r2_t.mean():.2f} +- {cv_r2_t.std():.2f}")
 
 #################################################################
 # We have a satisfying first result: an R2 of 0.66!
@@ -358,3 +361,98 @@ print(
 #
 # Now up to you, try improving our model by adding information into it and
 # beating our result!
+
+#######################################################################
+# Using the |fa| to fuzzy join multiple tables
+# --------------------------------------------
+# A faster way to merge different tables from the World Bank
+# to `X` is to use the |fa|.
+#
+# The |fa| is a transformer that can easily chain joins of tables on
+# a main table.
+
+#######################################################################
+# Instantiating the transformer
+# ............................
+
+y = df["Happiness score"]
+#######################################################################
+# We gather the auxilliary tables into a
+# list of (tables, keys) for the `tables` parameter.
+# An instance of the transformer with the necessary information is:
+from dirty_cat import FeatureAugmenter
+
+fa = FeatureAugmenter(
+    tables=[
+        (gdppc, "Country Name"),
+        (life_exp, "Country Name"),
+        (legal_rights, "Country Name"),
+    ],
+    main_key="Country",
+)
+
+#################################################################
+# Fitting and transforming into the final table
+# ............................................
+# To get our final joined table we will fit and transform the main table (df)
+# with our create instance of the |fa|:
+df_final = fa.fit_transform(df)
+
+df_final.head(10)
+
+##########################################################################
+# And that's it! As previously, we now have a big table
+# ready for machine learning.
+# Let's create our machine learning pipeline:
+from sklearn.pipeline import make_pipeline
+from sklearn.compose import make_column_transformer
+
+# We include only the columns that will be pertinent for our regression:
+encoder = make_column_transformer(
+    (
+        "passthrough",
+        [
+            "GDP per capita (current US$)",
+            "Life expectancy at birth, total (years)",
+            "Strength of legal rights index (0=weak to 12=strong)",
+        ],
+    ),
+    remainder="drop",
+)
+
+pipeline = make_pipeline(fa, encoder, HistGradientBoostingRegressor())
+
+##########################################################################
+# And the best part is that we are now able to evaluate the paramaters of the |fj|.
+# For instance, the ``match_score`` was manually picked and can now be
+# introduced into a grid search:
+
+from sklearn.model_selection import GridSearchCV
+
+# We will test four possible values of match_score:
+params = {"featureaugmenter__match_score": [0.2, 0.3, 0.4, 0.5]}
+
+grid = GridSearchCV(pipeline, param_grid=params)
+grid.fit(df, y)
+
+print(grid.best_params_)
+##########################################################################
+# The grid searching gave us the best value of 0.5 for the parameter
+# ``match_score``. Let's use this value in our regression:
+#
+
+print(f"Mean R2 score with pipeline is {grid.score(df, y):.2f}")
+
+##########################################################################
+#
+# .. topic:: Note:
+#
+#    Here, ``grid.score()`` takes directly the best model
+#    (with ``match_score=0.5``) that was found during the grid search.
+#    Thus, it is equivalent to fixing the ``match_score`` to 0.5 and
+#    refitting the pipeline on the data.
+#
+#
+# Great, by evaluating the correct ``match_score`` we improved our
+# results significantly!
+#


Test patch:
diff --git a/dirty_cat/tests/test_feature_augmenter.py b/dirty_cat/tests/test_feature_augmenter.py
new file mode 100644
index 0000000..4840fa1
--- /dev/null
+++ b/dirty_cat/tests/test_feature_augmenter.py
@@ -0,0 +1,92 @@
+import pandas as pd
+import pytest
+
+from dirty_cat import FeatureAugmenter
+
+
+def test_feature_augmenter():
+    main_table = pd.DataFrame(
+        [
+            "France",
+            "Germany",
+            "Italy",
+        ],
+        columns=["Country"],
+    )
+
+    aux_table_1 = pd.DataFrame(
+        [
+            ["Germany", 84_000_000],
+            ["French Republic", 68_000_000],
+            ["Italia", 59_000_000],
+        ],
+        columns=["Country", "Population"],
+    )
+
+    aux_table_2 = pd.DataFrame(
+        [
+            ["France", 2937],
+            ["Italy", 2099],
+            ["Germany", 4223],
+            ["UK", 3186],
+        ],
+        columns=["Country name", "GDP (billion)"],
+    )
+
+    aux_table_3 = pd.DataFrame(
+        [
+            ["La France", "Paris"],
+            ["Italy", "Rome"],
+            ["Germany", "Berlin"],
+        ],
+        columns=["Countries", "Capital"],
+    )
+
+    aux_tables = [
+        (aux_table_1, "Country"),
+        (aux_table_2, "Country name"),
+        (aux_table_3, "Countries"),
+    ]
+
+    fa = FeatureAugmenter(tables=aux_tables, main_key="Country")
+
+    fa.fit(main_table)
+
+    number_of_cols = tuple(
+        map(
+            sum,
+            zip(
+                main_table.shape,
+                aux_table_1.shape,
+                aux_table_2.shape,
+                aux_table_3.shape,
+            ),
+        )
+    )[1]
+
+    big_table = fa.transform(main_table)
+    assert big_table.shape == (main_table.shape[0], number_of_cols)
+
+    big_table = fa.fit_transform(main_table)
+    assert big_table.shape == (main_table.shape[0], number_of_cols)
+
+    false_fa = FeatureAugmenter(tables=aux_tables, main_key="Countryy")
+
+    with pytest.raises(
+        ValueError,
+        match=r"Got main_key",
+    ):
+        false_fa.fit(main_table)
+
+    false_aux_tables = [
+        (aux_table_1, "Countrys"),
+        (aux_table_2, "Country name"),
+        (aux_table_3, "Countries"),
+    ]
+
+    false_fa2 = FeatureAugmenter(tables=false_aux_tables, main_key="Country")
+    with pytest.raises(
+        ValueError,
+        match=r"Got column key",
+    ):
+        false_fa2.fit(main_table)