Repository: autogluon/autogluon
Pull request: Add basic feature engineering documentation
*Issue #, if available:*
1456

*Description of changes:*
Add one card to tabular documentation on feature engineering.  Mostly lists what AutoGluon does by default with a worked example.  Basics on how to add your own, and a pointer to the detailed python script in examples/tabular (already written by  innixma).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


Problem statement:
Documentation of feature engineering
Add documentation (initially just for the TabularPredictor) explaining what feature engineering AutoGluon does for the user by default:

- what is the universe of types each column can be
- how do we infer types per column
- for each type, what further feature engineering is done (e.g. ngrams and the date parsing I just submitted)
- what are the parameters used for the heuristics
- how does the user disable/enable bits of feature engineering.

Hints from issue discussion:
@Innixma please assign this to me.
I have a question as I work on this documentation:
- can I create hyperlinks to the source from the markdown?  E.g. .. I think core/utils/utils.py might be "../../../core/src/autogluon/core/utils/utils.py".  This is necessary to point people to implementation that they can't access directly via the API.  I guess the risk is that these links might be brittle.  I'd rather do this than mention very detailed implementation in the documentation, which might go stale.
> I have a question as I work on this documentation:
> 
> * can I create hyperlinks to the source from the markdown?  E.g. .. I think core/utils/utils.py might be "../../../core/src/autogluon/core/utils/utils.py".  This is necessary to point people to implementation that they can't access directly via the API.  I guess the risk is that these links might be brittle.  I'd rather do this than mention very detailed implementation in the documentation, which might go stale.

This is a good question. Of course we could link to the github source but that isn't ideal. I had similar difficulties in the past and wasn't sure how to do it. One thing I was able to do was get a button that links API doc to source code.

Example: https://auto.gluon.ai/stable/api/autogluon.features.html#autogluon.features.generators.AbstractFeatureGenerator

Click the [source] button in top right of API description brings you to

https://auto.gluon.ai/stable/_modules/autogluon/features/generators/abstract.html#AbstractFeatureGenerator

However, I'm unsure if all API docs exist here or only the ones that have [source] links for them somewhere in docs. If they don't all exist, we could maybe change it so it does generate it for all the code including private, but not sure how to enable that without a deeper dive.
Related to #570

Commit message: Add basic feature engineering documentation

Patch to review:
diff --git a/docs/tutorials/tabular_prediction/index.rst b/docs/tutorials/tabular_prediction/index.rst
index 760c5ee850fd..ae96e4db7e85 100644
--- a/docs/tutorials/tabular_prediction/index.rst
+++ b/docs/tutorials/tabular_prediction/index.rst
@@ -54,6 +54,12 @@ For standard datasets that are represented as tables (stored as CSV file, parque
 
       How to add a custom metric to AutoGluon.
 
+   .. card::
+      :title: Feature Engineering
+      :link: tabular-feature-engineering.html
+
+      AutoGluon's default feature engineering and how to extend it.
+
    .. card::
       :title: FAQ
       :link: tabular-faq.html
@@ -72,4 +78,5 @@ For standard datasets that are represented as tables (stored as CSV file, parque
    tabular-multilabel
    tabular-custom-model
    tabular-custom-metric
+   tabular-feature-engineering
    tabular-faq
diff --git a/docs/tutorials/tabular_prediction/tabular-feature-engineering.md b/docs/tutorials/tabular_prediction/tabular-feature-engineering.md
new file mode 100644
index 000000000000..6f8eebd29f75
--- /dev/null
+++ b/docs/tutorials/tabular_prediction/tabular-feature-engineering.md
@@ -0,0 +1,184 @@
+
+:label:`sec_feature_engineering`
+
+### Introduction ###
+
+Feature engineering involves taking raw tabular data and 
+
+1. converting it into a format ready for the machine learning model to read
+2. trying to enhance some columns ('features' in ML jargon) to give the ML models more information, hoping to get more accurate results.
+
+Autogluon does some of this for you.  This document describes how that works, and how you can extend it.  We describe here the default behaviour, much of which is configurable, as well as pointers to how to alter the behaviour from the default.
+
+### Column Types ###
+
+AutoGluon.Tabular recognises the following types of features, and has separate processing for them:
+
+| Feature Type        | Example Values           |
+| :------------------ | :----------------------- |
+| boolean             | A, B                     |
+| numerical           | 1.3, 2.0, -1.6           |
+| categorical         | Red, Blue, Yellow        |
+| datetime            | 1/31/2021, Mar-31        |
+| text                | Mary had a little lamb   |
+
+In addition, other AutoGluon prediction modules recognise additional feature types, these can also be enabled in `AutoGluon.Tabular` by using the [MultiModal](tabular-multimodal.html) option. 
+
+| Feature Type        | Example Values           |
+| :------------------ | :----------------------- |
+| image               | path/image123.png        |
+
+### Column Type Detection ###
+
+- Boolean columns are any columns with only 2 unique values.
+
+- Any string columns are deemed categorical unless they are text (see below).  Some models perform better if you tell them which columns are categorical and which are continuous. 
+
+- Numeric columns are passed through without change, except to identify them as `float` or `int`.  Currently, numeric columns are not tested to determine if they are likely to be categorical.  You can force them to be treated as categorical with the Pandas syntax `.astype(category)`, see below.
+
+- Text columns are detected by firstly checking that most rows are unique.  If they are, and there are multiple separate words detected in most rows, the row is a text column.  For details see `common/features/infer_types.py` in the source.
+
+- Datetime columns are detected by trying to convert them to Pandas datetimes.  Pandas detects a wide range of datetime formats.  If many of the values in a column are successfully converted, they are datetimes.  Currently datetimes that appear to be purely numeric (e.g. 20210530) are not correctly detected.  Any NaN values are set to the column mean.  For details see `common/features/infer_types.py`.
+
+
+### Problem Type Detection ###
+
+If the user does not specify whether the problem is a classification problem or a regression problem, the 'label' column is examined to try to guess.  Several things point towards a regression problem : the values are floating point non-integers, and there are a large amount of unique values.  Within classification, both multiclass and binary (n=2 categories) are detected.  For details see `utils/utils.py`.
+
+To override the automatic inference, explicitly pass the problem_type (one of 'binary', 'regression', 'multiclass') to `TabularPredictor()`.  For example:
+```
+predictor = TabularPredictor(label='class', problem_type='multiclass').fit(train_data)
+```
+
+
+### Automatic Feature Engineering ###
+
+## Numerical Columns ##
+
+Numeric columns, both integer and floating point, currently have no automated feature engineering.
+
+## Categorical Columns ##
+
+Since many downstream models require categories to be encoded as integers, each categorical feature is mapped to monotonically increasing integers.
+
+## Datetime Columns ##
+
+Columns recognised as datetime, are converted into several features:
+
+- a numerical Pandas datetime.  Note this has maximum and minimum values specified at [pandas.Timestamp.min](https://pandas.pydata.org/docs/reference/api/pandas.Timestamp.min.html) and [pandas.Timestamp.max](https://pandas.pydata.org/docs/reference/api/pandas.Timestamp.min.html) respectively, which may affect extremely dates very far into the future or past.
+- several extracted columns, the default is `[year, month, day, dayofweek]`.  This is configrable via the [DatetimeFeatureGenerator](../../api/autogluon.features.html#datetimefeaturegenerator)
+
+Note that missing, invalid and out-of-range features generated by the above logic will be converted to the mean value across all valid rows.
+
+
+## Text Columns ##
+
+If the [MultiModal](tabular-multi.model.html) option is enabled, then text columns are processed using a full Transformer neural network model with pretrained NLP models.  See the [TextPredictor](../text_prediction/index.html) section for more information.
+
+Otherwise, they are processed in two more simple ways:
+
+- an n-gram feature generator extracts n-grams (short strings) from the text feature, adding many additional columns, one for each n-gram feature.  These columns are 'n-hot' encoded, containing 1 or more if the original feature contains the n-gram 1 or more times, and 0 otherwise.  By default, all text columns are concatenated before applying this stage, and the n-grams are individual words, not substrings of words.  You can configure this via the [TextNgramFeatureGenerator](../../api/autogluon.features.html#textngramfeaturegenerator) class. The n-gram generation is done in `generators/text_ngram.py`
+- Some additional numerical features are calculated, such as word counts, character counts, proportion of uppercase characters, etc.  This is configurable via the [TextSpecialFeatureGenerator](../../api/autogluon.features.html#textspecialfeaturegenerator).  This is done in `generators/text_special.py`
+
+## Additional Processing ##
+
+- Columns containing only 1 value are dropped before passing to models.
+- Columns containing duplicates of other columns are removed before passing to models.
+
+### Feature Engineering Example ###
+
+By default a feature generator called [AutoMLPipelineFeatureGenerator](../../api/autogluon.features.html#autogluon.features.generators.AutoMLPipelineFeatureGenerator) is used.  Let's see this in action.  We'll create a dataframe containing a floating point column, an integer column, a datetime column,  a categorical column.  We'll first take a look at the raw data we created.
+
+```{.python .input}
+from autogluon.tabular import TabularDataset, TabularPredictor
+import pandas as pd
+import numpy as np
+import random
+from sklearn.datasets import make_regression
+from datetime import datetime
+
+x, y = make_regression(n_samples = 100,n_features = 5,n_targets = 1, random_state = 1)
+dfx = pd.DataFrame(x, columns=['A','B','C','D','E'])
+dfy = pd.DataFrame(y, columns=['label'])
+
+# Create an integer column, a datetime column, a categorical column and a string column to demonstrate how they are processed.
+dfx['B'] = (dfx['B']).astype(int)
+dfx['C'] = datetime(2000,1,1) + pd.to_timedelta(dfx['C'].astype(int), unit='D')
+dfx['D'] = pd.cut(dfx['D'] * 10, [-np.inf,-5,0,5,np.inf],labels=['v','w','x','y'])
+dfx['E'] = pd.Series(list(' '.join(random.choice(["abc", "d", "ef", "ghi", "jkl"]) for i in range(4)) for j in range(100)))
+dataset=TabularDataset(dfx)
+print(dfx)
+```
+
+Now let's call the default feature generator AutoMLPipeLineFeatureGenerator with no parameters and see what it does.
+
+```{.python .input}
+from autogluon.features.generators import AutoMLPipelineFeatureGenerator
+auto_ml_pipeline_feature_generator = AutoMLPipelineFeatureGenerator()
+auto_ml_pipeline_feature_generator.fit_transform(X=dfx)
+```
+
+We can see that:
+
+- The floating point and integer columns 'A' and 'B' are unchanged.
+- The datetime column 'C' has been converted to a raw value (in nanoseconds), as well as parsed into additional columns for the year, month, day and dayofweek.
+- The string categorical column 'D' has been mapped 1:1 to integers - a lot of models only accept numerical input.
+- The freeform text column has been mapped into some summary features ('char_count' etc) as well as a N-hot matrix saying whether each text contained each word.
+
+To get more details, we should call the pipeline as part of `TabularPredictor.fit()`.  We need to combine the `dfx` and `dfy` DataFrames since fit() expects a single dataframe.
+```{.python .input}
+df = pd.concat([dfx, dfy], axis=1)
+predictor = TabularPredictor(label='label')
+predictor.fit(df, hyperparameters={'GBM' : {}}, feature_generator=auto_ml_pipeline_feature_generator)
+```
+
+Reading the output, note that:
+
+- the string-categorical column 'D', despite being mapped to integers, is still recognised as categorical. 
+- the integer column 'B' has not been identified as categorical, even though it only has a few unique values:
+```{.python .input}
+print(len(set(dfx['B'])))
+```
+To mark it as categorical, we can explicitly mark it as categorical in the original dataframe:
+```{.python .input}
+dfx["B"] = dfx["B"].astype("category")
+auto_ml_pipeline_feature_generator = AutoMLPipelineFeatureGenerator()
+auto_ml_pipeline_feature_generator.fit_transform(X=dfx)
+```
+
+### Missing Value Handling ###
+To illustrate missing value handling, let's set the first row to all NaNs:
+```{.python .input}
+dfx.iloc[0] = np.nan
+dfx.head()
+```
+
+Now if we reprocess:
+```{.python .input}
+auto_ml_pipeline_feature_generator = AutoMLPipelineFeatureGenerator()
+auto_ml_pipeline_feature_generator.fit_transform(X=dfx)
+```
+
+We see that the floating point, integer, categorical and text fields 'A', 'B', 'D', and 'E' have retained the NaNs, but the datetime column 'C' has been set to the mean of the non-NaN values.
+
+
+### Customization of Feature Engineering ###
+To customize your feature generation pipeline, it is recommended to call [PipelineFeatureGenerator](../../api/autogluon.features.html#autogluon.features.generators.PipelineFeatureGenerator), passing in non-default parameters to other feature generators as required.  For example, if we think downstream models would benefit from removing rare categorical values and replacing with NaN, we can supply the parameter maximum_num_cat to CategoryFeatureGenerator, as below:
+```{.python .input}
+from autogluon.features.generators import PipelineFeatureGenerator, CategoryFeatureGenerator, IdentityFeatureGenerator
+from autogluon.common.features.types import R_INT, R_FLOAT
+mypipeline = PipelineFeatureGenerator(
+    generators = [[        
+        CategoryFeatureGenerator(maximum_num_cat=10),  # Overridden from default.
+        IdentityFeatureGenerator(infer_features_in_args=dict(valid_raw_types=[R_INT, R_FLOAT])),
+    ]]
+)
+```
+If we then dump out the transformed data, we can see that all columns have been converted to numeric, because that's what most models require, and the rare categorical values have been replaced with NaN:
+```{.python .input}
+mypipeline.fit_transform(X=dfx)
+```
+
+For more on custom feature engineering, see the detailed notebook `examples/tabular/example_custom_feature_generator.py`.
+
+


Code review comments:
diff_hunk: @@ -0,0 +1,184 @@
+
+:label:`sec_feature_engineering`
+
+### Introduction ###
+
+Feature engineering involves taking raw tabular data and 
+
+1. converting it into a format ready for the machine learning model to read
+2. trying to enhance some columns ('features' in ML jargon) to give the ML models more information, hoping to get more accurate results.
+
+Autogluon does some of this for you.  This document describes how that works, and how you can extend it.  We describe here the default behaviour, much of which is configurable, as well as pointers to how to alter the behaviour from the default.
path: docs/tutorials/tabular_prediction/tabular-feature-engineering.md
text: @user1:
nit: AutoGluon

@author:
Have updated as suggested.  Sorry, not clear if I'm supposed to press 'Resolve' or you are here.

@user1:
I normally suggest the contributor marks as resolved after responding to the comment and pushing the fix to the PR.