Repository: dirty-cat/dirty_cat
Pull request #90 (2018-12-14 18:32:56)

DOC first version of the non-linear online example
Example about non-linear starts stabilizing IMO, so I can start taking feedback. 

In short, the example shows a classification problem for the `traffic_violations` dataset.

5 types of models are implemented:
* `SimilarityEncoder` + `SVC`
* `SimilarityEncoder` + `RBFSampler` + `SVC`
* `SimilarityEncoder` + `RBFSampler` + `SGD`
* `SimilarityEncoder` fitted on a subset of the data + `RBFSampler` + `SGD`
* `SimilarityEncoder` fitted on a subset of the data  + `RBFSampler` + online `SGD`

Training set sizes still dummy to make CI pass fast. For now, the scripts takes around 15 sec to run.
`RBFSampler` can easily be switched in favor of `Nystroem`.


codecov[bot]: # [Codecov](https://codecov.io/gh/dirty-cat/dirty_cat/pull/90?src=pr&el=h1) Report
> Merging [#90](https://codecov.io/gh/dirty-cat/dirty_cat/pull/90?src=pr&el=desc) into [master](https://codecov.io/gh/dirty-cat/dirty_cat/commit/a926bbb373c7128d43f40c06c47d542efdf289f4?src=pr&el=desc) will **decrease** coverage by `2.08%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/dirty-cat/dirty_cat/pull/90/graphs/tree.svg?width=650&token=9p6uHCyC5u&height=150&src=pr)](https://codecov.io/gh/dirty-cat/dirty_cat/pull/90?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master     #90      +/-   ##
=========================================
- Coverage   61.38%   59.3%   -2.09%     
=========================================
  Files          11      11              
  Lines         720     720              
  Branches      139     139              
=========================================
- Hits          442     427      -15     
- Misses        242     258      +16     
+ Partials       36      35       -1
```


| [Impacted Files](https://codecov.io/gh/dirty-cat/dirty_cat/pull/90?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [dirty\_cat/string\_distances.py](https://codecov.io/gh/dirty-cat/dirty_cat/pull/90/diff?src=pr&el=tree#diff-ZGlydHlfY2F0L3N0cmluZ19kaXN0YW5jZXMucHk=) | `37.39% <0%> (-13.05%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/dirty-cat/dirty_cat/pull/90?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/dirty-cat/dirty_cat/pull/90?src=pr&el=footer). Last update [a926bbb...1e48d46](https://codecov.io/gh/dirty-cat/dirty_cat/pull/90?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

GaelVaroquaux: Thanks for the example. I just had a quick look and here are my immediate reactions:

* It's very didactic. You did a good job inserting text and mark up to make it easy to follow
* It's a bit long, we should find things to remove. We really need to keep the online part. Maybe we should remove things before
* The example limits itself to n=3000. I would like to run it on the full dataset, which is 50 times bigger. This means that non online methods will not be scalable. We can compare non online methods on a subset of the data (3000 for instance) to online methods on the full data.
pierreglaser: > It's a bit long

Agreed. We can remove one model. `SimilarityEncoder fitted on a subset of the data + RBFSampler + SGD` (4) seems like a good candidate.

>The example limits itself to n=3000

Actually, there are 2 variables defining the training set size:

* `train_set_size`, that defines the training set size for non-online models. Currently set to 3000
* `online_train_set_size` that defines the training set size for the online model. Currently set to 10000.

We can scale it up to x4 (12000 and 40000), which I did on my laptop. Afterwards, we reach our few minutes time budget for the example. Note that the test set is the same for every (online and non-online) models.
GaelVaroquaux: In the list, I would probably remove the following:

* SimilarityEncoder + RBFSampler + SVC
* SimilarityEncoder + RBFSampler + SGD
* SimilarityEncoder fitted on a subset of the data + RBFSampler + SGD

I would only keep SimilarityEncoder + SVC rbf and the online solution. I
would explain that the RBFSampler is necessary to approximate the
non-linear model fit by a non-linear feature transformation, that can
then be fit online.

> We can scale it up to x4 (12000 and 40000), which I did on my laptop.
> Afterwards, we reach our few minutes time budget for the example.

OK. But does the time cost come from the online part, or the other one. I
would be in favor of keeping the non online small. I think that it is
didactic to show that only enables basically unbounded fits.

Also, in terms of speed, it may be interesting to reduce a bit the
embedding dimension of the SimilarityEncoder using tricks in
https://dirty-cat.github.io/stable/auto_examples/04_dimension_reduction_and_performance.html
maybe with 200 protoptypes and the most_frequent strategy. This should
make everything faster, and I would hope that it doesn't change much in
terms of generalization.

> Note that the test set is the same for every (online and non-online) models.

Good to have the same test set!

pierreglaser: >  it may be interesting to reduce a bit the embedding dimension of the SimilarityEncoder
I already use only 100 categories (the first 100 rows of the training set). So dimension is already pretty low. But I can switch to using the most frequent strategy.
GaelVaroquaux: >     I already use only 100 categories (the first 100 rows of the training set).
>     So dimension is already pretty low. But I can switch to using the most
>     frequent strategy.

Most frequent on a large subset might be a better recommendation.

pierreglaser: Just updated the example taking the previous remarks into consideration.
pierreglaser: (There is some noise in the diff - `conf.py` -  that comes from an unfortunate rebase, don't worry about it)
GaelVaroquaux: A few quick comments:

* It would be good to have a plot: we try to have all our example quite visual. Maybe comparing batch SVM RBF on 2000 data points to the RBF sampler + online on 50000 data points? Or a convergence plot as a function of the number of iterations.
* The RBF embedding dimension should probably be significantly higher. Indeed, the price to pay to approximate kernels is to have a dimensionality much higher than the input dimensionality.
* I think that I would prefer running on the for data, rather than limiting to 50000.
GaelVaroquaux: Can you merge with master, to fix the conflict and have the CI run

Patch:
diff --git a/build_tools/circle/build_doc.sh b/build_tools/circle/build_doc.sh
index 7722c7c..51a43d5 100755
--- a/build_tools/circle/build_doc.sh
+++ b/build_tools/circle/build_doc.sh
@@ -125,8 +125,7 @@ pip install sphinx-gallery
 pip install python-Levenshtein
 pip install memory_profiler
 
-#cloning and installing from the ColumnTransformer branch
-pip install scikit-learn==0.20
+pip install scikit-learn
 
 # Build and install the project in dev mode
 python setup.py develop
diff --git a/dirty_cat/datasets/__init__.py b/dirty_cat/datasets/__init__.py
index 26c11de..76df361 100644
--- a/dirty_cat/datasets/__init__.py
+++ b/dirty_cat/datasets/__init__.py
@@ -4,7 +4,7 @@ datasets module of dirty cat
 
 from .fetching import fetch_midwest_survey, fetch_employee_salaries, \
     fetch_open_payments, fetch_road_safety, fetch_medical_charge, \
-    fetch_traffic_violations
+    fetch_traffic_violations, fetch_drug_directory
 from .fetching import get_data_dir
 
 __all__ = [
@@ -14,5 +14,6 @@ __all__ = [
     'fetch_employee_salaries',
     'fetch_road_safety',
     'fetch_open_payments',
+    'fetch_drug_directory',
     'fetch_traffic_violations'
 ]
diff --git a/dirty_cat/datasets/fetching.py b/dirty_cat/datasets/fetching.py
index 4c5929b..67f1391 100644
--- a/dirty_cat/datasets/fetching.py
+++ b/dirty_cat/datasets/fetching.py
@@ -37,7 +37,8 @@ DatasetInfo = namedtuple('DatasetInfo',
 # a DatasetInfo Object is basically a tuple of UrlInfos object
 # an UrlInfo object is composed of an url and the filenames contained
 # in the request content
-UrlInfo = namedtuple('UrlInfo', ['url', 'filenames', 'uncompress'])
+UrlInfo = namedtuple(
+    'UrlInfo', ['url', 'filenames', 'uncompress', 'encoding'])
 
 ROAD_SAFETY_CONFIG = DatasetInfo(
     name='road_safety',
@@ -50,12 +51,12 @@ ROAD_SAFETY_CONFIG = DatasetInfo(
                 "Vehicles_2015.csv",
                 "Accidents_2015.csv"
             ),
-            uncompress=True),
+            uncompress=True, encoding='utf-8'),
         UrlInfo(
             url="http://data.dft.gov.uk/road-accidents-safety-data/"
                 "MakeModel2015.zip",
             filenames=("2015_Make_Model.csv",),
-            uncompress=True
+            uncompress=True, encoding='utf-8'
         )
     ),
     main_file="Accidents_2015.csv",  # for consistency, all files are relevant,
@@ -68,7 +69,7 @@ OPEN_PAYMENTS_CONFIG = DatasetInfo(
     (
         UrlInfo(
             url='http://download.cms.gov/openpayments/PGYR13_P011718.ZIP',
-            filenames=None, uncompress=True
+            filenames=None, uncompress=True, encoding='utf-8'
         ),
     ),
     main_file='OP_DTL_GNRL_PGYR2013_P01172018.csv',  # same
@@ -83,7 +84,7 @@ MIDWEST_SURVEY_CONFIG = DatasetInfo(
                 "master/region-survey/FiveThirtyEight_Midwest_Survey.csv",
             filenames=(
                 "FiveThirtyEight_Midwest_Survey.csv",
-            ), uncompress=False
+            ), uncompress=False, encoding='utf-8'
         ),
     ),
     main_file="FiveThirtyEight_Midwest_Survey.csv",
@@ -99,7 +100,7 @@ MEDICAL_CHARGE_CONFIG = DatasetInfo(
             filenames=(
                 "Medicare_Provider_Charge_Inpatient_DRG100_FY2011.csv",
             ),
-            uncompress=True
+            uncompress=True, encoding='utf-8'
 
         ),
     ),
@@ -116,7 +117,7 @@ EMPLOYEE_SALARIES_CONFIG = DatasetInfo(
             url="https://data.montgomerycountymd.gov/api/views/"
                 "xj3h-s2i7/rows.csv?accessType=DOWNLOAD",
             filenames=("rows.csv",),
-            uncompress=False
+            uncompress=False, encoding='utf-8'
         ),
     ),
     main_file="rows.csv",
@@ -131,12 +132,28 @@ TRAFFIC_VIOLATIONS_CONFIG = DatasetInfo(
                 "4mse-ku6q/rows.csv?accessType=DOWNLOAD",
             filenames=(
                 "rows.csv",
-            ), uncompress=False,
+            ), uncompress=False, encoding='utf-8'
         ),
     ),
     main_file="rows.csv",
     source="https://catalog.data.gov/dataset/ traffic-violations-56dda"
 )
+
+DRUG_DIRECTORY_CONFIG = DatasetInfo(
+    name='drug_directory',
+    urlinfos=(
+        UrlInfo(
+            url="https://www.accessdata.fda.gov/cder/ndctext.zip",
+            filenames=(
+                "product.txt",
+                "package.txt",
+            ), uncompress=True, encoding='latin-1'
+        ),
+    ),
+    main_file="product.txt",
+    source="https://www.fda.gov/Drugs/InformationOnDrugs/ucm142438.htm"
+)
+
 FOLDER_PATH = os.path.dirname(os.path.realpath(__file__))
 
 
@@ -144,6 +161,24 @@ class FileChangedError(Exception):
     pass
 
 
+def _change_file_encoding(file_name, initial_encoding, target_encoding):
+
+    temp_file_name = file_name + '.temp'
+    try:
+        with open(file_name, "r", encoding=initial_encoding) as source:
+            with open(temp_file_name, "w", encoding=target_encoding) as target:
+                while True:
+                    contents = source.read(100000)
+                    if not contents:
+                        break
+                    target.write(contents)
+        shutil.move(temp_file_name, file_name)
+
+    finally:
+        if os.path.exists(temp_file_name):
+            os.unlink(temp_file_name)
+
+
 def _download_and_write(url, file, show_progress=True):
     if show_progress:  # maybe not an ideal design, should we mark
         # clint as mandatory?
@@ -183,7 +218,8 @@ def fetch_dataset(configfile: DatasetInfo, show_progress=True):
     data_dir = os.path.join(get_data_dir(), configfile.name)
     for urlinfo in configfile.urlinfos:
         _fetch_file(urlinfo.url, data_dir, filenames=urlinfo.filenames,
-                    uncompress=urlinfo.uncompress, show_progress=show_progress)
+                    uncompress=urlinfo.uncompress, show_progress=show_progress,
+                    initial_encoding=urlinfo.encoding)
     # returns the absolute path of the csv file where the data is
     result_dict = {
         'description': 'The downloaded data contains the {} dataset.\n'
@@ -195,7 +231,8 @@ def fetch_dataset(configfile: DatasetInfo, show_progress=True):
 
 
 def _fetch_file(url, data_dir, filenames=None, overwrite=False,
-                md5sum=None, uncompress=True, show_progress=True):
+                md5sum=None, uncompress=True, show_progress=True,
+                initial_encoding='utf-8'):
     """fetches the content of a requested url
 
     IF the downloaded file is compressed, then the fetcher
@@ -289,6 +326,10 @@ def _fetch_file(url, data_dir, filenames=None, overwrite=False,
     if _check_if_exists(full_name, remove=False) and uncompress:
         _uncompress_file(full_name, delete_archive=True)
 
+    if download and (initial_encoding != 'utf-8'):
+        for file in os.listdir(data_dir):
+            _change_file_encoding(
+                os.path.join(data_dir, file), initial_encoding, 'utf-8')
     return full_name
 
 
@@ -423,3 +464,24 @@ def fetch_traffic_violations():
     https://catalog.data.gov/dataset/traffic-violations-56dda
     """
     return fetch_dataset(TRAFFIC_VIOLATIONS_CONFIG, show_progress=False)
+
+
+def fetch_drug_directory():
+    """fetches the drug directory dataset
+
+    Returns
+    -------
+    dict
+        a dictionary containing:
+
+            - a short description of the dataset (under the ``description``
+              key)
+            - an absolute path leading to the csv file where the data is stored
+              locally (under the ``path`` key)
+
+
+    References
+    ----------
+    https://www.fda.gov/Drugs/InformationOnDrugs/ucm142438.htm
+    """
+    return fetch_dataset(DRUG_DIRECTORY_CONFIG, show_progress=False)
diff --git a/examples/05_scaling_non_linear_models.py b/examples/05_scaling_non_linear_models.py
new file mode 100644
index 0000000..d71035d
--- /dev/null
+++ b/examples/05_scaling_non_linear_models.py
@@ -0,0 +1,529 @@
+r"""
+Fitting scalable, non-linear models on data with dirty categories
+=================================================================
+
+A very classic dilemna when training a machine learning model consists in
+choosing between using a linear model or a non-linear one.
+
+Linear models are very well studied and understood. They are generally fast
+and easy to optimize, and generate interpretable results. However, for some
+problems with a complex relationship between the input and the output, linear
+models reach their expressivity limit: whatever the number of samples the
+training set may have, past some point, it's precision won't get any better.
+
+Non-linear models, however, tend to *scale* better with sample size: they are
+able to digest the information in the additional samples to get a better
+estimate of the link between the input and the output.
+
+Non-linear models form a very large model class. Among others, this class
+includes:
+
+* Neural Networks
+* Tree-based methods such as Random Forests, and the very powerful Gradient
+  Boosting Machines [#xgboost]_
+* Kernel Methods.
+
+However, reaching the phase where the non-linear model outpeforms the linear
+one can be complicated. Indeed, a more complex models means often a longer
+fitting/tuning process:
+
+* Neural networks often necessitate extended model tuning time, in order to
+  achieve good optimization and network architecture.
+* Gradient Boosting Machines do not tend to scale extremely well with
+  increasing sample size, as all the data needs to be loaded into the main
+  memory.
+* For kernel methods, parameter fitting requires the inversion of a gram matrix
+  of size :math:`n \times n` (:math:`n` being the number of samples), yiedling
+  a quadratic dependency (with n) in the final compmutation time.
+
+
+In order to make the best out of a non-linear model, one has to **make it
+scalable**. For kernel methods, there exist approximation algorithms that
+drop the quadratic dependency with the sample size while ensuring almost the
+same model capacity.
+
+In this example, you will learn how to:
+    1. Build a ML pipeline that uses a kernel method.
+    2. Make this pipeline scalable, by using online algorithms and dimension
+       reduction methods.
+
+
+.. note::
+   This example assumes the reader to be familiar with similarity encoding and
+   its use-cases.
+
+   * For an introduction to dirty categories, see :ref:`this example<sphx_glr_auto_examples_01_dirty_categories.py>`.
+   * To learn with dirty categories using the SimilarityEncoder, see :ref:`this example<sphx_glr_auto_examples_02_predict_employee_salaries.py>`.
+
+
+.. |NYS| replace:: :class:`Nystroem <sklearn.kernel_approximation.Nystroem>`
+
+.. |NYS_EXAMPLE|
+    replace:: :ref:`scikit-learn documentation <nystroem_kernel_approx>`
+
+.. |RBF|
+    replace:: :class:`~sklearn.kernel_approximation.RBFSampler`
+
+.. |SVC|
+    replace:: :class:`SupportVectorClassifier <sklearn.svm.SVC>`
+
+.. |SE| replace:: :class:`~dirty_cat.SimilarityEncoder`
+
+.. |SGDClassifier| replace::
+    :class:`~sklearn.linear_model.SGDClassifier`
+
+.. |APS| replace::
+    :func:`~sklearn.metrics.average_precision_score`
+
+.. |OneHotEncoder| replace::
+    :class:`~sklearn.preprocessing.OneHotEncoder`
+
+.. |ColumnTransformer| replace::
+    :class:`~sklearn.compose.ColumnTransformer`
+
+.. |LabelEncoder| replace::
+    :class:`~sklearn.preprocessing.LabelEncoder`
+
+.. |SGDClassifier_partialfit| replace::
+    :meth:`~sklearn.linear_model.SGDClassifier.partial_fit`
+
+.. |Pipeline| replace::
+    :class:`~sklearn.pipeline.Pipeline`
+
+.. |pd_read_csv| replace::
+    :func:`pandas.read_csv`
+
+.. |sklearn| replace::
+    :std:doc:`scikit-learn <sklearn:index>`
+
+.. |transform| replace::
+    :std:term:`transform <sklearn:transform>`
+
+.. |fit| replace::
+    :std:term:`fit <sklearn:fit>`
+
+"""
+###############################################################################
+# Training a first simple pipeline
+# --------------------------------
+# The data that the model will fit is the :code:`drug_directory` dataset.
+from dirty_cat.datasets import fetch_drug_directory
+
+info = fetch_drug_directory()
+print(info['description'])
+
+###############################################################################
+# .. topic:: Problem Setting
+#
+#    We set the goal of our machine learning problem as follows:
+#
+#    .. centered::
+#       **predict the type of a drug given its composition.**
+#
+#
+# The :code:`NONPROPRIETARYNAME` column, is composed of text observations with
+# describing each drug's composition. The :code:`PRODUCTTYPENAME` column
+# consists of categorial values: therefore, our problem is a classification
+# problem. You can have a glimpse of the values here:
+import pandas as pd
+
+df = pd.read_csv(info['path'], nrows=10, sep='\t').astype(str)
+print(df[['NONPROPRIETARYNAME', 'PRODUCTTYPENAME']].head())
+# This will be useful further down in the example.
+columns_names = df.columns
+
+###############################################################################
+# Estimators construction
+# -----------------------
+# Our input is categorical, thus needs to be encoded. As observations often
+# consist in variations around a few concepts (for instance,
+# :code:`'Amlodipine Besylate'` and
+# :code:`'Amlodipine besylate and atorvastatin calcium'`
+# have one ingredient in common), we need an encoding able to
+# capture similarities between observations.
+
+from dirty_cat import SimilarityEncoder
+from sklearn.compose import make_column_transformer
+from sklearn.preprocessing import OneHotEncoder
+similarity_encoder = SimilarityEncoder(similarity='ngram')
+
+###############################################################################
+# Two other columns are used to predict the output: ``DOSAGEFORMNAME`` and
+# ``ROUTENAME``. They are both categorical and can be encoded with a
+# |OneHotEncoder|. We use a |ColumnTransformer| to stack the |OneHotEncoder|
+# and the |SE|.  We can now choose a kernel method, for instance a |SVC|, to
+# fit the encoded inputs.
+from sklearn.pipeline import Pipeline
+from sklearn.svm import SVC
+
+column_transformer = make_column_transformer(
+    (similarity_encoder, ['NONPROPRIETARYNAME']),
+    (OneHotEncoder(handle_unknown='ignore'), ['DOSAGEFORMNAME', 'ROUTENAME']),
+    sparse_threshold=1)
+
+# The classifier and the ColumnTransformer are stacked into a Pipeline object
+classifier = SVC(kernel='rbf', random_state=42, gamma=1)
+steps = [('transformer', column_transformer), ('classifier', classifier)]
+model = Pipeline(steps)
+
+###############################################################################
+# Data Loading and Preprocessing
+# ------------------------------
+# Like in most machine learning setups, the data has to be splitted into 2
+# exclusive parts:
+#
+# * One for model training.
+# * One for model testing.
+#
+# For this reason, we create a simple wrapper around |pd_read_csv|, that
+# extracts the :code:`X`, and :code:`y` from the dataset.
+#
+# .. topic:: Note about class imbalance:
+#
+#    The :code:`y` labels are composed of 7 unique classes. However, ``HUMAN
+#    OTC DRUG`` and ``HUMAN PRESCRIPTION DRUG`` represent around 97% of the
+#    data, in a fairly balanced manner. The last 5 classes are much rarer.
+#    Dealing with class imbalance is out of the scope of this example, so the
+#    models will be trained on the first two classes only.
+
+
+def preprocess(df, label_encoder):
+    df = df.loc[df['PRODUCTTYPENAME'].isin(
+        ['HUMAN OTC DRUG', 'HUMAN PRESCRIPTION DRUG'])]
+
+    df = df[['NONPROPRIETARYNAME', 'DOSAGEFORMNAME', 'ROUTENAME', 'PRODUCTTYPENAME']]
+    df = df.dropna()
+
+    X = df[['NONPROPRIETARYNAME', 'DOSAGEFORMNAME', 'ROUTENAME']]
+    y = df[['PRODUCTTYPENAME']].values
+
+    y_int = label_encoder.transform(np.squeeze(y))
+
+    return X, y_int
+
+
+def get_X_y(**kwargs):
+    """simple wrapper around pd.read_csv that extracts features and labels
+
+    Some systematic preprocessing is also carried out to avoid doing this
+    transformation repeatedly in the code.
+    """
+    global label_encoder
+    df = pd.read_csv(info['path'], sep='\t', **kwargs)
+    return preprocess(df, label_encoder)
+
+###############################################################################
+# Classifier objects in |sklearn| often require :code:`y` to be integer labels.
+# Additionally, |APS| requires a binary version of the labels.  For these two
+# purposes, we create:
+#
+# * a |LabelEncoder|, that we pre-fitted on the known :code:`y` classes
+# * a |OneHotEncoder|, pre-fitted on the resulting integer labels.
+#
+# Their |transform| methods can the be called at appopriate times.
+import numpy as np
+from sklearn.preprocessing import LabelEncoder, OneHotEncoder
+
+label_encoder = LabelEncoder()
+label_encoder.fit(['HUMAN OTC DRUG', 'HUMAN PRESCRIPTION DRUG'])
+
+one_hot_encoder = OneHotEncoder(categories="auto", sparse=False)
+one_hot_encoder.fit([[0], [1]])
+
+###############################################################################
+# .. warning::
+#
+#    During the following training procedures of this example, we will assume
+#    that the dataset was shuffled prior to its loading. As a reason, we can
+#    take the first :math:`n` observations for the training set, the next
+#    :math:`m` observations for the test set and so on. This may not be the
+#    case for all datasets, so be careful before applying this code to your own
+#    !
+#
+#
+# Finally, the :code:`X` and :code:`y` are loaded.
+#
+#
+# .. topic:: Note: offsetting the test set
+#
+#    We create an offset to separate the training and the test set. The reason
+#    for this, is that the online procedures of this example will consume far
+#    more rows, but we still would like to compare accuracies with the same the
+#    same test set, and not change it each time. Therefore, we "reserve" the
+#    first 100000 rows for the training phase. The rest is made available to
+#    the test set.
+train_set_size = 5000
+test_set_size = 10000
+offset = 100000
+
+X_train, y_train = get_X_y(skiprows=1, names=columns_names,
+                           nrows=train_set_size)
+
+X_test, y_test = get_X_y(skiprows=offset, names=columns_names,
+                         nrows=test_set_size)
+
+###############################################################################
+# Evaluating time and sample complexity
+# -------------------------------------
+# Let's get an idea of model precision and performance depending on the number
+# of the samples used in the train set.
+# The |Pipeline| is trained over different training set sizes. For this,
+# :code:`X_train` and :code:`y_train` get sliced into subsets of increasing
+# size, while :code:`X_test` and :code:`y_test` do not change when the
+# sample size varies.
+import time
+from sklearn.metrics import average_precision_score
+
+# define the different train set sizes on which to evaluate the model
+train_set_sizes = [train_set_size // 10, train_set_size // 3, train_set_size]
+
+train_times_svc, test_scores_svc = [], []
+
+for n in train_set_sizes:
+
+    t0 = time.perf_counter()
+    model.fit(X_train[:n], y_train[:n])
+    train_time = time.perf_counter() - t0
+
+    y_pred = model.predict(X_test)
+
+    y_pred_onehot = one_hot_encoder.transform(y_pred.reshape(-1, 1))
+    y_test_onehot = one_hot_encoder.transform(y_test.reshape(-1, 1))
+
+    test_score = average_precision_score(y_test_onehot, y_pred_onehot)
+
+    train_times_svc.append(train_time)
+    test_scores_svc.append(test_score)
+
+    msg = ("using {:>5} samples: model fitting took {:.1f}s, test accuracy of "
+           "{:.3f}")
+    print(msg.format(n, train_time, test_score))
+
+
+###############################################################################
+# Increasing training size cleary improves model accuracy. However, the
+# training time and the input size increase quadratically with the training set
+# size.  Indeed, kernel methods need to process an entire :math:`n \times n`
+# matrix at once. In order for this matrix to be loaded into memory, :math:`n`
+# has to remain low: Using 30000 observations, the input is a :math:`30000
+# \times 30000` matrix.  If composed of 32-bit floats, its total size is around
+# :math:`30000^2 \times 32 = 2.8 \times 10^{10} \text{bits} = 4\text{GB}`
+
+###############################################################################
+# Reducing input dimension using kernel approximation methods.
+# ------------------------------------------------------------
+#
+# The main scalability issues with kernels methods is the processing of a large
+# square matrix. To understand where this matrix comes from, we need to delve a
+# little bit deeper these methods internals.
+#
+# .. topic:: Kernel methods
+#
+#    Kernel methods address non-linear problems by leveraging similarities
+#    between each pair of inputs. Using a similarity matrix to solve a machine
+#    learning problem allows to catch complex, non-linear relationships within
+#    the data.  But it requires inverting this matrix, which can be a
+#    computational burden when the sample sizes increases.
+
+
+###############################################################################
+# Kernel approximation methods
+# ----------------------------
+# From what was said below, two criterions are limiting a kernel algorithm to
+# scale:
+#
+# * It processes a matrix, whose size increases quadratically with the number
+#   of samples.
+# * During fitting time, this matrix is **inverted**, meaning it has to be
+#   loaded into main memory.
+#
+# Kernel approximation methods such as |RBF| or |NYS| [#nys_ref]_ try to
+# approximate this similarity matrix, without actually creating it. By allowing
+# the program to not compute the perfect similarity matrix, the problem
+# complexity becomes linear! Plus, the samples also do not need to be processed
+# at once into main memory. We are not bound to use a |SVC| anymore, and can
+# instead use an online optimization that will process the input by batch.
+#
+# .. topic:: Online algorithms
+#
+#    An online algorithm [#online_ref]_ is an algorithm that treats its input
+#    piece by piece in a serial fashion. An famous example is the stochastic
+#    gradient descent [#sgd_ref]_, where an estimation the objective function's
+#    gradient is computed on a batch of the data at each step.
+#
+#
+
+###############################################################################
+# Reducing the transformers dimensionality
+# ----------------------------------------
+# There is one last scalability issue in our pipeline: the |SE| and the |RBF|
+# both implement the |fit| method. How to adapt those method to an online
+# setting, where the data is never loaded as a whole?
+#
+# A simple solution is to partially fit the |SE| and the |RBF| on a subset of
+# the data, prior to the online fitting step.
+
+from sklearn.kernel_approximation import RBFSampler
+n_out_encoder = 1000
+n_out_rbf = 5000
+n_samples_encoder = 10000
+
+X_encoder, _ = get_X_y(nrows=n_samples_encoder, names=columns_names)
+
+similarity_encoder = SimilarityEncoder(
+    similarity='ngram', categories='most_frequent', n_prototypes=n_out_encoder,
+    random_state=42, ngram_range=(2, 4))
+
+# Fit the rbf_sampler with the similarity matrix.
+column_transformer = make_column_transformer(
+    (similarity_encoder, ['NONPROPRIETARYNAME']),
+    (OneHotEncoder(handle_unknown='ignore'), ['DOSAGEFORMNAME', 'ROUTENAME']),
+    sparse_threshold=1)
+
+transformed_categories = column_transformer.fit_transform(X_encoder)
+
+# gamma is a parameter of the rbf function, that sets how fast the similarity
+# between two points should decrease as the distance between them rises. It
+# is data-specific, and needs to be chosen carefully, for example using
+# cross-validation.
+rbf_sampler = RBFSampler(
+    gamma=0.5, n_components=n_out_rbf, random_state=42)
+rbf_sampler.fit(transformed_categories)
+
+
+def encode(X, y_int, one_hot_encoder, column_transformer, rbf_sampler):
+    X_sim_encoded = column_transformer.transform(X)
+
+    X_highdim = rbf_sampler.transform(X_sim_encoded.toarray())
+
+    y_onehot = one_hot_encoder.transform(y_int.reshape(-1, 1))
+
+    return X_highdim, y_onehot
+
+
+# The inputs and labels of the val and test sets have to be pre-processed the
+# same way the training set was processed:
+X_test_kernel_approx, y_true_test_onehot = encode(
+    X_test, y_test, one_hot_encoder, column_transformer, rbf_sampler)
+
+
+###############################################################################
+# Online training for out-of-memory data
+# --------------------------------------
+# We now have all the theoretical elements to create an non-linear, online
+# kernel method.
+import warnings
+from sklearn.linear_model.stochastic_gradient import SGDClassifier
+
+online_train_set_size = 100000
+# Filter warning on max_iter and tol
+warnings.filterwarnings('ignore', module='sklearn.linear_model')
+sgd_classifier = SGDClassifier(
+    max_iter=1, tol=None, random_state=42, average=10)
+
+
+###############################################################################
+# We can now start the training, by looping over batches one by one. Note that
+# only one pass over the whole dataset is done. It may be worth doing several
+# passes, but for very large sample sizes, the increase in test accuracy is
+# likely to be marginal.
+batchsize = 1000
+test_scores_rbf = []
+train_times_rbf = []
+online_train_set_sizes = []
+t0 = time.perf_counter()
+
+iter_csv = pd.read_csv(
+    info['path'], nrows=online_train_set_size, chunksize=batchsize,
+    skiprows=1, names=columns_names, sep='\t')
+
+for batch_no, batch in enumerate(iter_csv):
+    X_batch, y_batch = preprocess(batch, label_encoder)
+    X_batch_kernel_approx, y_batch_onehot = encode(
+        X_batch, y_batch, one_hot_encoder, column_transformer, rbf_sampler)
+
+    # make one pass of stochastic gradient descent over the batch.
+    sgd_classifier.partial_fit(
+        X_batch_kernel_approx, y_batch, classes=[0, 1])
+
+    # print train/test accuracy metrics every 5 batch
+    if (batch_no % 5) == 0:
+        message = "batch {:>4} ".format(batch_no)
+        for origin, X, y_true_onehot in zip(
+                ('train', 'val'),
+                (X_batch_kernel_approx, X_test_kernel_approx),
+                (y_batch_onehot, y_true_test_onehot)):
+
+            y_pred = sgd_classifier.predict(X)
+
+            # preprocess correctly the labels and prediction to match
+            # average_precision_score expectations
+            y_pred_onehot = one_hot_encoder.transform(
+                y_pred.reshape(-1, 1))
+
+            score = average_precision_score(y_true_onehot, y_pred_onehot)
+            message += "{} precision: {:.4f}  ".format(origin, score)
+            if origin == 'val':
+                test_scores_rbf.append(score)
+                train_times_rbf.append(time.perf_counter() - t0)
+                online_train_set_sizes.append((batch_no + 1)*batchsize)
+
+        print(message)
+
+###############################################################################
+# So far, we fitted two kinds of models: a exact kernel algorithm, and an
+# approximate, online one. Lets compare both the accuracies and the number of
+# visited samples for each model as we increase our time budget:
+import matplotlib.pyplot as plt
+
+f, axs = plt.subplots(2, 1, sharex=True, figsize=(10, 7))
+ax_score, ax_capacity = axs
+
+ax_score.set_ylabel('score')
+ax_capacity.set_ylabel('training set size')
+ax_capacity.set_xlabel('time')
+
+ax_score.plot(train_times_svc, test_scores_svc, 'b-', label='exact')
+ax_score.plot(train_times_rbf, test_scores_rbf, 'r-', label='online')
+
+ax_capacity.plot(train_times_svc, train_set_sizes, 'b-', label='exact')
+ax_capacity.plot(train_times_rbf, online_train_set_sizes, 'r-', label='online')
+ax_capacity.set_yscale('log')
+
+ax_score.legend(
+    bbox_to_anchor=(0., 1.02, 1., .102),
+    loc=3, ncol=2, mode='expand',
+    borderaxespad=0.)
+
+# compare the two methods in their common time range
+ax_score.set_xlim(0, min(train_times_svc[-1], train_times_rbf[-1]))
+
+title = """Test set accuracy and number of samples visited
+samples seen by the SGDClassifier"""
+f.suptitle(title)
+
+
+###############################################################################
+# This plot shows us that for the time budget, the online model will eventually
+# process more samples, be faster and reach a far higher test accuracy that the
+# non-online, exact kernel method.
+#
+# Our online model also outperforms online **linear** models (for instance,
+# |SE| + |SGDClassifier|). We did not fit two online models here for simplicity
+# purposes, but to train an online linear model, simply comment out the line in
+# encode :code:`X_highdim = rbf_sampler.transform(X_sim_encoded.toarray())` and
+# change it for example with :code:`X_highdim = X_sim_encoded`.
+#
+# In particular, this hierarchy between the linear model and the non-linear
+# one shows that there were some significant non-linear relashionships
+# between the input and the output. By scaling a kernel method, we succesfully
+# took this non-linearity into account in our model, which was a far from
+# trivial task at the beginning of this example!
+
+###############################################################################
+# .. rubric:: Footnotes
+# .. [#xgboost] `Slides on gradient boosting by Tianqi Chen, the founder of XGBoost <https://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf>`_
+# .. [#online_ref] `Wikipedia article on online algorithms <https://en.wikipedia.org/wiki/Online_algorithm>`_
+# .. [#sgd_ref] `Leon Bouttou's article on stochastic gradient descent <http://khalilghorbal.info/assets/spa/papers/ML_GradDescent.pdf>`_
+# .. [#nys_ref] |NYS_EXAMPLE|
+# .. [#dual_ref] `Introduction to duality <https://pdfs.semanticscholar.org/0373/e7289a1978108d6455218160a529c85842c0.pdf>`_


Test patch:
diff --git a/dirty_cat/datasets/tests/test_fetching.py b/dirty_cat/datasets/tests/test_fetching.py
index 7b1471f..beaecd6 100644
--- a/dirty_cat/datasets/tests/test_fetching.py
+++ b/dirty_cat/datasets/tests/test_fetching.py
@@ -129,6 +129,63 @@ def test_fetch_file_overwrite():
         shutil.rmtree(test_dir)
 
 
+@utils.with_setup(setup=setup_mock, teardown=teardown_mock)
+def test_convert_file_to_utf8(monkeypatch):
+    from dirty_cat.datasets import fetching
+    datadir = os.path.join(datasets_utils.get_data_dir(),
+                           'testdata')
+    try:
+        # Specify some content encoded in latin-1, and make sure the final file
+        # contains the same content, but in utf-8. Here, '\xe9' in latin-1 is
+        # '\xc3\xa9' in utf-8
+
+        with monkeypatch.context() as m:
+            m.setattr(utils.MockResponse, "zipresult", False)
+            m.setattr(utils.MockResponse, "_file_contents", b'\xe9')
+
+            dataset_info = fetching.DatasetInfo(
+                name='testdata',
+                urlinfos=(fetching.UrlInfo(
+                    url='http://foo/data',
+                    filenames=('data',),
+                    uncompress=False,
+                    encoding='latin-1'),),
+                main_file='data',
+                source='http://foo/')
+
+            info = fetching.fetch_dataset(dataset_info, show_progress=False)
+
+            with open(info['path'], 'rb') as f:
+                content = f.read()
+
+            assert content == b'\xc3\xa9'
+            os.unlink(info['path'])
+
+            m.setattr(utils.MockResponse, "zipresult", True)
+
+            dataset_info_with_zipfile = fetching.DatasetInfo(
+                name='testdata',
+                urlinfos=(fetching.UrlInfo(
+                    url='http://foo/data.zip',
+                    filenames=('unzipped_data.txt',),
+                    uncompress=True,
+                    encoding='latin-1'),),
+                main_file='unzipped_data.txt',
+                source='http://foo/')
+
+            info_unzipped = fetching.fetch_dataset(
+                dataset_info_with_zipfile, show_progress=False)
+
+            with open(info_unzipped['path'], 'rb') as f:
+                content_unzipped = f.read()
+
+            assert content_unzipped == b'\xc3\xa9'
+
+    finally:
+        if os.path.exists(datadir):
+            shutil.rmtree(datadir)
+
+
 def test_md5_sum_file():
     # Create dummy temporary file
     out, f = mkstemp()
@@ -149,7 +206,8 @@ def test_fetch_dataset():
                                        urlinfos=(fetching.UrlInfo(
                                            url='http://foo/data',
                                            filenames=('data',),
-                                           uncompress=False),),
+                                           uncompress=False,
+                                           encoding='utf-8'),),
                                        main_file='data',
                                        source='http://foo/')
         fetching.fetch_dataset(urlinfo, show_progress=False)
@@ -163,8 +221,9 @@ def test_fetch_dataset():
                                        urlinfos=(fetching.UrlInfo(
                                            url='http://foo/data.zip',
                                            filenames=('unzipped_data.txt',),
-                                           uncompress=True),),
-                                       main_file='data',
+                                           uncompress=True,
+                                           encoding='utf-8'),),
+                                       main_file='unzipped_data.txt',
                                        source='http://foo/')
         fetching.fetch_dataset(urlinfo, show_progress=False)
         assert os.path.exists(os.path.join(datadir, 'unzipped_data.txt'))
diff --git a/dirty_cat/datasets/tests/utils.py b/dirty_cat/datasets/tests/utils.py
index fd4e6c2..e521c03 100644
--- a/dirty_cat/datasets/tests/utils.py
+++ b/dirty_cat/datasets/tests/utils.py
@@ -26,6 +26,7 @@ class with_setup:
 class MockResponse(requests.Response):
     with_content_length = True
     zipresult = False
+    _file_contents = ' '.encode('utf-8')
 
     @classmethod
     def set_with_content_length(cls, b):
@@ -50,7 +51,7 @@ class MockResponse(requests.Response):
     def set_to_zipfile(cls, b):
         cls.zipresult = b
 
-    def __init__(self, url, params, content: str, stream=True):
+    def __init__(self, url, params, stream=True):
         super(MockResponse, self).__init__()
         self.url = url
         self.params = params
@@ -60,24 +61,27 @@ class MockResponse(requests.Response):
             self.raw = BytesIO()
             with zipfile.ZipFile(self.raw, mode='w',
                                  compression=zipfile.ZIP_DEFLATED) as mf:
-                mf.writestr('unzipped_data.txt', 'test')
+                mf.writestr('unzipped_data.txt', self._file_contents)
+
+            # go to the start of the stream
+            self.raw.seek(0)
             # self.raw = self.raw.wr(file=self.raw)
             if stream:
                 self._content = self.raw.getvalue()
 
         else:
             if stream:
-                self.raw = BytesIO(str.encode(content))
+                self.raw = BytesIO(self._file_contents)
             else:
-                self._content = content
+                self._content = self._file_contents
 
         if self.with_content_length:
             if self.zipresult:
                 self.headers['Content-Length'] = len(self.raw.getvalue())
                 self.total_length = len(self.raw.getvalue())
             else:
-                self.headers['Content-Length'] = len(content)
-                self.total_length = len(content)
+                self.headers['Content-Length'] = len(self._file_contents)
+                self.total_length = len(self._file_contents)
 
     def iter_content(self, chunk_size=1, decode_unicode=False):
         """
@@ -103,4 +107,4 @@ class MockResponse(requests.Response):
 
 def mock_request_get(url, params=None, stream=True):
     # should we be able to input a specialized content for the mockrequest?
-    return MockResponse(url, params=None, content=' ')
+    return MockResponse(url, params=None)