Repository: scikit-learn/scikit-learn
Pull request: [MRG] Add iterable values support for dictvectorizer
#### Reference Issues/PRs
Closes #6045
Closes #6369
Resumes #8750

#### What does this implement/fix? Explain your changes.
Add iterable support for value types in dictvectorizer

Problem statement:
Accept multiple values for one categorical feature
Hello,

I think it is convenient to accept `list` as a value of `dict` in `DictVectorizer.fit()`
to accept multiple values for one categorical feature.

Assume that we use the category of movies as a feature,
and this category can be 'action', 'love-comedy', 'science-fiction', etc...
Some movies match some of these categories,
'The Matrix' can be both 'action' and 'science-fiction', for example.

Ideal form in my mind is as follows:

``` python
>>> dv = DictVectorizer(sparse=False)
>>> movie_features = [{'category': ['action', 'science-fiction']}, {'category': 'action'}]
>>> dv.fit_transform(movie_features)
array([[1, 1],
       [1, 0]])
>>> dv.get_feature_names()
['category=action', 'category=science-fiction']
```

But with present API of `DictVectorizer.fit()`, it only accepts one value for 'category', `int` or `str`
and we have to write redundant codes to achieve this, like this:

``` python
>>> movie_features = [{'category': ['action', 'science-fiction']}, {'category': ['action']}]
>>> new_movie_features = []
>>> for movie in movie_features:
...     cate_list = movie['category']
...     feature_dict = {}
...     for cate in cate_list:
...         key = 'category={}'.format(cate)
...         feature_dict[key] = 1
...     new_movie_features.append(feature_dict)
>>> new_movie_features
[{'category=science-fiction': 1, 'category=action': 1}, {'category=action': 1}]
```

If there exists a way to use multiple values for one categorical feature
or there is already similar issue, could you please let me know?
Otherwise, if accepting `list` as a values of `dict` is a reasonable way, I'll send a PR later.

Hints from issue discussion:
Came across exactly this issue today. Seems appropriate to leave this remark here as a show of interest.

I think we've generally been avoiding making `DictVectorizer` too complex. It was developed as a bare-bones general solution after `text.CountVectorizer` blew out. I personally think this is reasonable functionality; but once we've introduced it into `DictVectorizer`, we need similar support in `FeatureHasher`.

Yes, this would be very useful.  Reading the documentation, I formed the impression that I could achieve multiple categories by supplying a list of pairs:
`[('cat1':'foo'), ('cat1':'bar')]`
but that doesn't work either.
#8750 has a fair chance of being merged before the next release

Commit message: Address comments.

Patch to review:
diff --git a/doc/whats_new/v0.24.rst b/doc/whats_new/v0.24.rst
index 60bf06cef8d02..1335577e67479 100644
--- a/doc/whats_new/v0.24.rst
+++ b/doc/whats_new/v0.24.rst
@@ -89,6 +89,13 @@ Changelog
   :pr:`17694` by :user:`Markus Rempfler <rempfler>` and
   :user:`Tsutomu Kusanagi <t-kusanagi2>`.
 
+:mod:`sklearn.feature_extraction`
+.................................
+
+- |Enhancement| :class:`feature_extraction.DictVectorizer` accepts multiple
+  values for one categorical feature. :pr:`17367` by :user:`Peng Yu <yupbank>`
+  and :user:`Chiara Marmo <cmarmo>`
+
 :mod:`sklearn.feature_selection`
 ................................
 
diff --git a/sklearn/feature_extraction/_dict_vectorizer.py b/sklearn/feature_extraction/_dict_vectorizer.py
index 1e7d53aa3c5a0..7ae4a0e2228ab 100644
--- a/sklearn/feature_extraction/_dict_vectorizer.py
+++ b/sklearn/feature_extraction/_dict_vectorizer.py
@@ -3,8 +3,9 @@
 # License: BSD 3 clause
 
 from array import array
-from collections.abc import Mapping
+from collections.abc import Mapping, Iterable
 from operator import itemgetter
+from numbers import Number
 
 import numpy as np
 import scipy.sparse as sp
@@ -35,6 +36,10 @@ class DictVectorizer(TransformerMixin, BaseEstimator):
     a feature "f" that can take on the values "ham" and "spam" will become two
     features in the output, one signifying "f=ham", the other "f=spam".
 
+    When feature values are iterables of strings (not mappings), this
+    transformer will iterate over the values to perform the same transformation
+    to every element.
+
     However, note that this transformer will only do a binary one-hot encoding
     when feature values are of type string. If categorical features are
     represented as numeric values such as int, the DictVectorizer can be
@@ -78,12 +83,28 @@ class DictVectorizer(TransformerMixin, BaseEstimator):
     >>> X
     array([[2., 0., 1.],
            [0., 1., 3.]])
-    >>> v.inverse_transform(X) == \
-        [{'bar': 2.0, 'foo': 1.0}, {'baz': 1.0, 'foo': 3.0}]
+    >>> v.inverse_transform(X) == [{'bar': 2.0, 'foo': 1.0},
+    ...                            {'baz': 1.0, 'foo': 3.0}]
     True
     >>> v.transform({'foo': 4, 'unseen_feature': 3})
     array([[0., 0., 4.]])
 
+    Example with multiple values for one categorical values:
+
+    >>> D2 = [{'foo': '1', 'bar': '2'}, {'foo': '3', 'baz': '1'},
+    ...       {'foo': ['one', 'three']}]
+    >>> X = v.fit_transform(D2)
+    >>> X
+    array([[1., 0., 1., 0., 0., 0.],
+           [0., 1., 0., 1., 0., 0.],
+           [0., 0., 0., 0., 1., 1.]])
+    >>> v.inverse_transform(X) == [{'bar=2': 1.0, 'foo=1': 1.0},
+    ...                            {'baz=1': 1.0, 'foo=3': 1.0},
+    ...                            {'foo=one': 1.0, 'foo=three': 1.0}]
+    True
+    >>> v.transform({'foo': '1', 'unseen_feature': [3]})
+    array([[0., 0., 1., 0., 0., 0.]])
+
     See also
     --------
     FeatureHasher : performs vectorization using only a hash function.
@@ -98,6 +119,27 @@ def __init__(self, *, dtype=np.float64, separator="=", sparse=True,
         self.sparse = sparse
         self.sort = sort
 
+    def _add_iterable_element(self, f, v, feature_names, vocab, fitting=True,
+                              transforming=False, indices=None, values=None):
+        """Add feature names for iterable of strings"""
+        for vv in v:
+            if isinstance(vv, str):
+                feature_name = "%s%s%s" % (f, self.separator, vv)
+                vv = 1
+            else:
+                raise TypeError(f'Unsupported type {type(vv)} in iterable '
+                                'value. Only iterables of string are '
+                                'supported.')
+            if fitting and feature_name not in vocab:
+                vocab[feature_name] = len(feature_names)
+                feature_names.append(feature_name)
+
+            if transforming and feature_name in vocab:
+                indices.append(vocab[feature_name])
+                values.append(self.dtype(vv))
+
+        return
+
     def fit(self, X, y=None):
         """Learn a list of feature name -> indices mappings.
 
@@ -118,10 +160,22 @@ def fit(self, X, y=None):
         for x in X:
             for f, v in x.items():
                 if isinstance(v, str):
-                    f = "%s%s%s" % (f, self.separator, v)
-                if f not in vocab:
-                    feature_names.append(f)
-                    vocab[f] = len(vocab)
+                    feature_name = "%s%s%s" % (f, self.separator, v)
+                    v = 1
+                elif isinstance(v, Number) or (v is None):
+                    feature_name = f
+                elif isinstance(v, Mapping):
+                    raise TypeError(f'Unsupported value Type {type(v)} '
+                                    f'for {f}: {v}.\n'
+                                    'Mapping objects are not supported.')
+                elif isinstance(v, Iterable):
+                    feature_name = None
+                    self._add_iterable_element(f, v, feature_names, vocab)
+
+                if feature_name is not None:
+                    if feature_name not in vocab:
+                        vocab[feature_name] = len(feature_names)
+                        feature_names.append(feature_name)
 
         if self.sort:
             feature_names.sort()
@@ -150,6 +204,8 @@ def _transform(self, X, fitting):
             feature_names = self.feature_names_
             vocab = self.vocabulary_
 
+        transforming = True
+
         # Process everything as sparse regardless of setting
         X = [X] if isinstance(X, Mapping) else X
 
@@ -164,17 +220,28 @@ def _transform(self, X, fitting):
         for x in X:
             for f, v in x.items():
                 if isinstance(v, str):
-                    f = "%s%s%s" % (f, self.separator, v)
+                    feature_name = "%s%s%s" % (f, self.separator, v)
                     v = 1
-                if f in vocab:
-                    indices.append(vocab[f])
-                    values.append(dtype(v))
-                else:
-                    if fitting:
-                        feature_names.append(f)
-                        vocab[f] = len(vocab)
-                        indices.append(vocab[f])
-                        values.append(dtype(v))
+                elif isinstance(v, Number) or (v is None):
+                    feature_name = f
+                elif isinstance(v, Mapping):
+                    raise TypeError(f'Unsupported value Type {type(v)} '
+                                    f'for {f}: {v}.\n'
+                                    'Mapping objects are not supported.')
+                elif isinstance(v, Iterable):
+                    feature_name = None
+                    self._add_iterable_element(f, v, feature_names, vocab,
+                                               fitting, transforming,
+                                               indices, values)
+
+                if feature_name is not None:
+                    if fitting and feature_name not in vocab:
+                        vocab[feature_name] = len(feature_names)
+                        feature_names.append(feature_name)
+
+                    if feature_name in vocab:
+                        indices.append(vocab[feature_name])
+                        values.append(self.dtype(v))
 
             indptr.append(len(indices))
 
diff --git a/sklearn/feature_extraction/tests/test_dict_vectorizer.py b/sklearn/feature_extraction/tests/test_dict_vectorizer.py
index 22a7402908cf1..571749fa7d387 100644
--- a/sklearn/feature_extraction/tests/test_dict_vectorizer.py
+++ b/sklearn/feature_extraction/tests/test_dict_vectorizer.py
@@ -76,6 +76,52 @@ def test_one_of_k():
     assert "version" not in names
 
 
+def test_iterable_value():
+    D_names = ['ham', 'spam', 'version=1', 'version=2', 'version=3']
+    X_data = [[2.0, 0.0, 1.0, 1.0, 0.0],
+              [0.0, 0.3, 0.0, 1.0, 0.0],
+              [0.0, -1.0, 0.0, 0.0, 1.0]]
+    D_in = [{"version": ["1", "2"], "ham": 2},
+            {"version": "2", "spam": .3},
+            {"version=3": True, "spam": -1}]
+    v = DictVectorizer()
+    X = v.fit_transform(D_in)
+    X = X.toarray()
+    assert_array_equal(X, X_data)
+
+    D_out = v.inverse_transform(X)
+    assert D_out[0] == {"version=1": 1, "version=2": 1, "ham": 2}
+
+    names = v.get_feature_names()
+
+    assert names == D_names
+
+
+def test_iterable_not_string_error():
+    error_value = ("Unsupported type <class 'int'> in iterable value. "
+                   "Only iterables of string are supported.")
+    D2 = [{'foo': '1', 'bar': '2'},
+          {'foo': '3', 'baz': '1'},
+          {'foo': [1, 'three']}]
+    v = DictVectorizer(sparse=False)
+    with pytest.raises(TypeError) as error:
+        v.fit(D2)
+    assert str(error.value) == error_value
+
+
+def test_mapping_error():
+    error_value = ("Unsupported value Type <class 'dict'> "
+                   "for foo: {'one': 1, 'three': 3}.\n"
+                   "Mapping objects are not supported.")
+    D2 = [{'foo': '1', 'bar': '2'},
+          {'foo': '3', 'baz': '1'},
+          {'foo': {'one': 1, 'three': 3}}]
+    v = DictVectorizer(sparse=False)
+    with pytest.raises(TypeError) as error:
+        v.fit(D2)
+    assert str(error.value) == error_value
+
+
 def test_unseen_or_no_features():
     D = [{"camelot": 0, "spamalot": 1}]
     for sparse in [True, False]:


Code review comments:
diff_hunk: @@ -35,6 +36,10 @@ class DictVectorizer(TransformerMixin, BaseEstimator):
     a feature "f" that can take on the values "ham" and "spam" will become two
     features in the output, one signifying "f=ham", the other "f=spam".
 
+    When feature values are iterables of strings (not mappings), this
path: sklearn/feature_extraction/_dict_vectorizer.py
text: @user1:
Maybe less precise here would be more helpful than the precise but uninstructive "iterables of strings (not mappings)". "If a feature value is a sequence or set of strings, this ..."

diff_hunk: @@ -78,12 +83,28 @@ class DictVectorizer(TransformerMixin, BaseEstimator):
     >>> X
     array([[2., 0., 1.],
            [0., 1., 3.]])
-    >>> v.inverse_transform(X) == \
-        [{'bar': 2.0, 'foo': 1.0}, {'baz': 1.0, 'foo': 3.0}]
+    >>> v.inverse_transform(X) == [{'bar': 2.0, 'foo': 1.0},
+    ...                            {'baz': 1.0, 'foo': 3.0}]
     True
     >>> v.transform({'foo': 4, 'unseen_feature': 3})
     array([[0., 0., 4.]])
 
+    Example with multiple values for one categorical values:
+
+    >>> D2 = [{'foo': '1', 'bar': '2'}, {'foo': '3', 'baz': '1'},
+    ...       {'foo': ['one', 'three']}]
+    >>> X = v.fit_transform(D2)
+    >>> X
+    array([[1., 0., 1., 0., 0., 0.],
+           [0., 1., 0., 1., 0., 0.],
+           [0., 0., 0., 0., 1., 1.]])
+    >>> v.inverse_transform(X) == [{'bar=2': 1.0, 'foo=1': 1.0},
path: sklearn/feature_extraction/_dict_vectorizer.py
text: @user1:
I don't think we need to demonstrate `inverse_transform` here, but I think `get_feature_names` would be very instructive.

diff_hunk: @@ -98,6 +119,27 @@ def __init__(self, *, dtype=np.float64, separator="=", sparse=True,
         self.sparse = sparse
         self.sort = sort
 
+    def _add_iterable_element(self, f, v, feature_names, vocab, fitting=True,
+                              transforming=False, indices=None, values=None):
+        """Add feature names for iterable of strings"""
+        for vv in v:
+            if isinstance(vv, str):
+                feature_name = "%s%s%s" % (f, self.separator, vv)
+                vv = 1
+            else:
+                raise TypeError(f'Unsupported type {type(vv)} in iterable '
+                                'value. Only iterables of string are '
+                                'supported.')
+            if fitting and feature_name not in vocab:
+                vocab[feature_name] = len(feature_names)
+                feature_names.append(feature_name)
+
+            if transforming and feature_name in vocab:
+                indices.append(vocab[feature_name])
+                values.append(self.dtype(vv))
path: sklearn/feature_extraction/_dict_vectorizer.py
text: @user1:
I've just realised that if a string is repeated in the iterable, then this will increment it twice (since Scipy sparse matrices sum values over repeated indices).

I think this behaviour is okay, but we need to document it differently. Or we need to forbid this behaviour by raising an error with duplicate strings.

@author:
I'm slightly in favour of forcing unique strings... 

@user1:
> I'm slightly in favour of forcing unique strings...

It's certainly the one that is easiest to change later, since we would be turning an error into a supported case.

@author:
> It's certainly the one that is easiest to change later, since we would be turning an error into a supported case.

I was also thinking about the use case described in #6045: attaching categories to movies, or labels to pull requests, we probably don't want to duplicate them... but, sure, who knows, at least it will be easy to get rid of the error... 

@user2:
I think it's fine to count repeated labels twice. This is consistent with what `CountVectorizer` for strings with repeated tokens.

I a way, `DictVectorizer` would behave as a natural complement for `CountVectorizer`  when the tokenization step would have already been performed as a preprocessing step.

@user2:
Also it's easy for the users to filter duplicates in their inputs by using Python `set` instead of lists if they don't want this.