Repository: INRIA/scikit-learn-mooc
Pull request #811: Add tip to read about TargetEncoder in the documentation
Author: login: fritshermans
Created: 2025-03-30T17:48:23Z | Merged: 2025-04-09T08:02:32Z

As proposed by @ArturoAmorQ in issue [#792, ](https://github.com/INRIA/scikit-learn-mooc/issues/792), I added a note to read about the TargetEncoder. Please let me know if this is what you had in mind.


Diff:
diff --git a/notebooks/03_categorical_pipeline_ex_02.ipynb b/notebooks/03_categorical_pipeline_ex_02.ipynb
index 83fafffcf..8a6d52408 100644
--- a/notebooks/03_categorical_pipeline_ex_02.ipynb
+++ b/notebooks/03_categorical_pipeline_ex_02.ipynb
@@ -155,6 +155,68 @@
    "source": [
     "# Write your code here."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Analysis\n",
+    "\n",
+    "From an accuracy point of view, the result is almost exactly the same. The\n",
+    "reason is that `HistGradientBoostingClassifier` is expressive and robust\n",
+    "enough to deal with misleading ordering of integer coded categories (which was\n",
+    "not the case for linear models).\n",
+    "\n",
+    "However from a computation point of view, the training time is much longer:\n",
+    "this is caused by the fact that `OneHotEncoder` generates more features than\n",
+    "`OrdinalEncoder`; for each unique categorical value a column is created.\n",
+    "\n",
+    "Note that the current implementation `HistGradientBoostingClassifier` is still\n",
+    "incomplete, and once sparse representation are handled correctly, training\n",
+    "time might improve with such kinds of encodings.\n",
+    "\n",
+    "The main take away message is that arbitrary integer coding of categories is\n",
+    "perfectly fine for `HistGradientBoostingClassifier` and yields fast training\n",
+    "times.\n",
+    "\n",
+    "Which encoder should I use?\n",
+    "\n",
+    "|                  | Meaningful order              | Non-meaningful order |\n",
+    "| ---------------- | ----------------------------- | -------------------- |\n",
+    "| Tree-based model | `OrdinalEncoder`              | `OrdinalEncoder` with reasonable depth    |\n",
+    "| Linear model     | `OrdinalEncoder` with caution | `OneHotEncoder`      |"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "lines_to_next_cell": 0
+   },
+   "source": [
+    "<div class=\"admonition important alert alert-info\">\n",
+    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Important</p>\n",
+    "<ul class=\"last simple\">\n",
+    "<li><tt class=\"docutils literal\">OneHotEncoder</tt>: always does something meaningful, but can be unnecessary\n",
+    "slow with trees.</li>\n",
+    "<li><tt class=\"docutils literal\">OrdinalEncoder</tt>: can be detrimental for linear models unless your category\n",
+    "has a meaningful order and you make sure that <tt class=\"docutils literal\">OrdinalEncoder</tt> respects this\n",
+    "order. Trees can deal with <tt class=\"docutils literal\">OrdinalEncoder</tt> fine as long as they are deep\n",
+    "enough. However, when you allow the decision tree to grow very deep, it might\n",
+    "overfit on other features.</li>\n",
+    "</ul>\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next to one-hot-encoding and ordinal encoding categorical features,\n",
+    "scikit-learn offers the [`TargetEncoder`](https://scikit-learn.org/stable/modules/preprocessing.html#target-encoder).\n",
+    "This encoder is well suited for nominal, categorical features with high\n",
+    "cardinality. This encoding strategy is beyond the scope of this course,\n",
+    "but the interested reader is encouraged to explore this encoder."
+   ]
   }
  ],
  "metadata": {
diff --git a/notebooks/03_categorical_pipeline_sol_02.ipynb b/notebooks/03_categorical_pipeline_sol_02.ipynb
index 161d0cbdd..a7aa19e8d 100644
--- a/notebooks/03_categorical_pipeline_sol_02.ipynb
+++ b/notebooks/03_categorical_pipeline_sol_02.ipynb
@@ -245,8 +245,8 @@
     "not the case for linear models).\n",
     "\n",
     "However from a computation point of view, the training time is much longer:\n",
-    "this is caused by the fact that `OneHotEncoder` generates approximately 10\n",
-    "times more features than `OrdinalEncoder`.\n",
+    "this is caused by the fact that `OneHotEncoder` generates more features than\n",
+    "`OrdinalEncoder`; for each unique categorical value a column is created.\n",
     "\n",
     "Note that the current implementation `HistGradientBoostingClassifier` is still\n",
     "incomplete, and once sparse representation are handled correctly, training\n",
@@ -257,45 +257,48 @@
     "times."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Which encoder should I use?\n",
+    "\n",
+    "|                  | Meaningful order              | Non-meaningful order |\n",
+    "| ---------------- | ----------------------------- | -------------------- |\n",
+    "| Tree-based model | `OrdinalEncoder`              | `OrdinalEncoder` with reasonable depth    |\n",
+    "| Linear model     | `OrdinalEncoder` with caution | `OneHotEncoder`      |"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {
-    "tags": [
-     "solution"
-    ]
+    "lines_to_next_cell": 0
    },
    "source": [
     "<div class=\"admonition important alert alert-info\">\n",
     "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Important</p>\n",
-    "<p>Which encoder should I use?</p>\n",
-    "<table border=\"1\" class=\"docutils\">\n",
-    "<thead valign=\"bottom\">\n",
-    "<tr><th class=\"head\"></th>\n",
-    "<th class=\"head\">Meaningful order</th>\n",
-    "<th class=\"head\">Non-meaningful order</th>\n",
-    "</tr>\n",
-    "</thead>\n",
-    "<tbody valign=\"top\">\n",
-    "<tr><td>Tree-based model</td>\n",
-    "<td><tt class=\"docutils literal\">OrdinalEncoder</tt></td>\n",
-    "<td><tt class=\"docutils literal\">OrdinalEncoder</tt></td>\n",
-    "</tr>\n",
-    "<tr><td>Linear model</td>\n",
-    "<td><tt class=\"docutils literal\">OrdinalEncoder</tt> with caution</td>\n",
-    "<td><tt class=\"docutils literal\">OneHotEncoder</tt></td>\n",
-    "</tr>\n",
-    "</tbody>\n",
-    "</table>\n",
     "<ul class=\"last simple\">\n",
     "<li><tt class=\"docutils literal\">OneHotEncoder</tt>: always does something meaningful, but can be unnecessary\n",
     "slow with trees.</li>\n",
     "<li><tt class=\"docutils literal\">OrdinalEncoder</tt>: can be detrimental for linear models unless your category\n",
     "has a meaningful order and you make sure that <tt class=\"docutils literal\">OrdinalEncoder</tt> respects this\n",
     "order. Trees can deal with <tt class=\"docutils literal\">OrdinalEncoder</tt> fine as long as they are deep\n",
-    "enough.</li>\n",
+    "enough. However, when you allow the decision tree to grow very deep, it might\n",
+    "overfit on other features.</li>\n",
     "</ul>\n",
     "</div>"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next to one-hot-encoding and ordinal encoding categorical features,\n",
+    "scikit-learn offers the [`TargetEncoder`](https://scikit-learn.org/stable/modules/preprocessing.html#target-encoder).\n",
+    "This encoder is well suited for nominal, categorical features with high\n",
+    "cardinality. This encoding strategy is beyond the scope of this course,\n",
+    "but the interested reader is encouraged to explore this encoder."
+   ]
   }
  ],
  "metadata": {
diff --git a/python_scripts/03_categorical_pipeline_ex_02.py b/python_scripts/03_categorical_pipeline_ex_02.py
index 7f48a8ac9..c7f160c2d 100644
--- a/python_scripts/03_categorical_pipeline_ex_02.py
+++ b/python_scripts/03_categorical_pipeline_ex_02.py
@@ -110,3 +110,48 @@
 
 # %%
 # Write your code here.
+
+# %% [markdown]
+# ### Analysis
+#
+# From an accuracy point of view, the result is almost exactly the same. The
+# reason is that `HistGradientBoostingClassifier` is expressive and robust
+# enough to deal with misleading ordering of integer coded categories (which was
+# not the case for linear models).
+#
+# However from a computation point of view, the training time is much longer:
+# this is caused by the fact that `OneHotEncoder` generates more features than
+# `OrdinalEncoder`; for each unique categorical value a column is created.
+#
+# Note that the current implementation `HistGradientBoostingClassifier` is still
+# incomplete, and once sparse representation are handled correctly, training
+# time might improve with such kinds of encodings.
+#
+# The main take away message is that arbitrary integer coding of categories is
+# perfectly fine for `HistGradientBoostingClassifier` and yields fast training
+# times.
+
+# Which encoder should I use?
+#
+# |                  | Meaningful order              | Non-meaningful order |
+# | ---------------- | ----------------------------- | -------------------- |
+# | Tree-based model | `OrdinalEncoder`              | `OrdinalEncoder` with reasonable depth    |
+# | Linear model     | `OrdinalEncoder` with caution | `OneHotEncoder`      |
+
+# %% [markdown]
+# ```{important}
+#
+# - `OneHotEncoder`: always does something meaningful, but can be unnecessary
+#   slow with trees.
+# - `OrdinalEncoder`: can be detrimental for linear models unless your category
+#   has a meaningful order and you make sure that `OrdinalEncoder` respects this
+#   order. Trees can deal with `OrdinalEncoder` fine as long as they are deep
+#   enough. However, when you allow the decision tree to grow very deep, it might
+#   overfit on other features.
+# ```
+# %% [markdown]
+# Next to one-hot-encoding and ordinal encoding categorical features,
+# scikit-learn offers the [`TargetEncoder`](https://scikit-learn.org/stable/modules/preprocessing.html#target-encoder).
+# This encoder is well suited for nominal, categorical features with high
+# cardinality. This encoding strategy is beyond the scope of this course,
+# but the interested reader is encouraged to explore this encoder.
diff --git a/python_scripts/03_categorical_pipeline_sol_02.py b/python_scripts/03_categorical_pipeline_sol_02.py
index 36e8ddcdc..a2653df4e 100644
--- a/python_scripts/03_categorical_pipeline_sol_02.py
+++ b/python_scripts/03_categorical_pipeline_sol_02.py
@@ -179,8 +179,8 @@
 # not the case for linear models).
 #
 # However from a computation point of view, the training time is much longer:
-# this is caused by the fact that `OneHotEncoder` generates approximately 10
-# times more features than `OrdinalEncoder`.
+# this is caused by the fact that `OneHotEncoder` generates more features than
+# `OrdinalEncoder`; for each unique categorical value a column is created.
 #
 # Note that the current implementation `HistGradientBoostingClassifier` is still
 # incomplete, and once sparse representation are handled correctly, training
@@ -190,19 +190,28 @@
 # perfectly fine for `HistGradientBoostingClassifier` and yields fast training
 # times.
 
-# %% [markdown] tags=["solution"]
-# ```{important}
-# Which encoder should I use?
+# %% [markdown]
+# ## Which encoder should I use?
 #
 # |                  | Meaningful order              | Non-meaningful order |
 # | ---------------- | ----------------------------- | -------------------- |
-# | Tree-based model | `OrdinalEncoder`              | `OrdinalEncoder`     |
+# | Tree-based model | `OrdinalEncoder`              | `OrdinalEncoder` with reasonable depth    |
 # | Linear model     | `OrdinalEncoder` with caution | `OneHotEncoder`      |
+
+# %% [markdown]
+# ```{important}
 #
 # - `OneHotEncoder`: always does something meaningful, but can be unnecessary
 #   slow with trees.
 # - `OrdinalEncoder`: can be detrimental for linear models unless your category
 #   has a meaningful order and you make sure that `OrdinalEncoder` respects this
 #   order. Trees can deal with `OrdinalEncoder` fine as long as they are deep
-#   enough.
+#   enough. However, when you allow the decision tree to grow very deep, it might
+#   overfit on other features.
 # ```
+# %% [markdown]
+# Next to one-hot-encoding and ordinal encoding categorical features,
+# scikit-learn offers the [`TargetEncoder`](https://scikit-learn.org/stable/modules/preprocessing.html#target-encoder).
+# This encoder is well suited for nominal, categorical features with high
+# cardinality. This encoding strategy is beyond the scope of this course,
+# but the interested reader is encouraged to explore this encoder.


Reviews:
user: login: ArturoAmorQ
body: Just a nitpick on the current changes.

Otherwise I think you are right with respect to the comment on `OneHotEncoder` resulting in 10x more features than an `OrdinalEncoder`. It does depend on the cardinality.

And also you are right that we should explain that "reasonable depth" means. This can be done by completing the phrase "Trees can deal with OrdinalEncoder fine as long as they are deep enough [to sort things out without overfitting...]" inside the `Important` banner.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T08:55:16Z
user: login: ArturoAmorQ
body: I took the liberty to sync the notebooks in your PR, hope you don't mind :)

Thanks again @fritshermans for your time and enthusiasm. Merging!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-09T08:00:58Z

Review comments:
diff_hunk: @@ -170,7 +170,7 @@
     f"with a fitting time of {elapsed_time:.3f}"
 )
 
-# %% [markdown] tags=["solution"]
+# %% [markdown]
path: python_scripts/03_categorical_pipeline_sol_02.py
user: login: ArturoAmorQ
body: Sorry for the second thoughts, but I now realize it's weird to show the interpretation of the results in the exercise. We can rather split a bit further below, to display the conclusions starting from the "Which encoder should I use" section.

```suggestion
# %% [markdown] tags=["solution"]
```
created_at: 2025-04-07T08:30:51Z
author_association: COLLABORATOR
diff_hunk: @@ -190,14 +190,15 @@
 # perfectly fine for `HistGradientBoostingClassifier` and yields fast training
 # times.
 
-# %% [markdown] tags=["solution"]
-# ```{important}
 # Which encoder should I use?
path: python_scripts/03_categorical_pipeline_sol_02.py
user: login: ArturoAmorQ
body: ```suggestion
# %% [markdown]
# ## Which encoder should I use?
```
created_at: 2025-04-07T08:32:14Z
author_association: COLLABORATOR

PR comments:
user: login: ArturoAmorQ
created_at: 2025-04-02T13:49:14Z
body: Thanks for the PR @fritshermans!

In https://github.com/INRIA/scikit-learn-mooc/issues/792#issuecomment-2619296610 I suggested adding a small comment in the [Importance of decision tree hyperparameters on generalization notebook](https://inria.github.io/scikit-learn-mooc/python_scripts/trees_hyperparameters.html) about ordinal encoding when there are too many unique categorical values.

On a second thought, I think your current contribution and the comment on the tree `max_depth` are more suited for the [Wrap-up analysis in the Exercise M1.05](https://inria.github.io/scikit-learn-mooc/python_scripts/03_categorical_pipeline_sol_02.html#id1).

Notice there we already have a small comment "Trees can deal with `OrdinalEncoder` fine as long as they are deep enough." that can be easily overlooked at, so maybe we can:
- take the table off from the `Important` banner;
- improve the entry for `Tree-based model` vs `Non-meaningful order` to something similar to "`OrdinalEncoder` with a reasonable depth";
- move the paragraph in this PR at the very end of this notebook;
- maybe remove the `tags=["solution"]` so that this whole wrap-up appears in the exercise and not only in it's solution.
author_association: COLLABORATOR
user: login: fritshermans
created_at: 2025-04-06T11:20:55Z
body: Thanks for your remarks @ArturoAmorQ!

I applied your suggestions. The only thing that I would propose to change is to add some explanation on 'reasonable depth' before the table. Moreover, why would a `OneHotEncoder` result in 10x more features than an `OrdinalEncoder`? That depends on the number of categories right?

Please let me know what you think!
author_association: CONTRIBUTOR
user: login: fritshermans
created_at: 2025-04-07T16:05:34Z
body: Done! Let me know what you think of it 😊
author_association: CONTRIBUTOR
user: login: fritshermans
created_at: 2025-04-09T08:30:58Z
body: thanks @ArturoAmorQ ! What should I have done for syncing the notebooks? They didn't show up as 'changed' by git on my local computer I think
author_association: CONTRIBUTOR
user: login: ArturoAmorQ
created_at: 2025-04-09T09:28:48Z
body: You can use either `make quizzes` or just `make` to make all of the notebooks, or you can make one notebook at the time as documented in [Contributing -Updating the notebooks ](https://github.com/INRIA/scikit-learn-mooc/blob/main/CONTRIBUTING.md#updating-the-notebooks).

It's just that we prefer to keep reviews focused on the python files, and then make the notebooks on our side at the end. The downside of such workflow is having too many PRs like https://github.com/INRIA/scikit-learn-mooc/pull/824 and https://github.com/INRIA/scikit-learn-mooc/pull/825 :sweat_smile: 
author_association: COLLABORATOR