Repository: RasaHQ/rasa
Pull request: Add option 'use_lemma' to CountVectorsFeaturizer
**Proposed changes**:

closes #6536 

**Status (please check what you already did)**:
- [x] added some tests for the functionality
- [x] updated the documentation
- [x] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/master/changelog) for instructions)
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
Lemmatization and CountVectorFeaturizers
**Rasa version**:

1.8 onwards 

**Python version**:

3.6 or 3.7

**Operating system** (windows, osx, ...):

All 

**Issue**:

Currently if you use spaCy components in your pipeline then the behaviour of the CountVectorFeaturizer changes.  Currently we use the `lemma` that spaCy provides as an alternative for the text when we use the "word" analyser. There's potentially merit to this idea but currently:

1. The user does not know this is happening because this phenomenon is undocumented. At the very least we need to update the docs to reflect this.
2. The user cannot configure this, the user must use the `lemma` tokens at all times the user wants to use spaCy. 

I think the best way forward is to implement a configuration in the `CountVectorFeaturizer` that makes it possible to not use the `lemma`. Once this is implemented we can update the documentation.

Hints from issue discussion:
A related discussion needs to take place here too. Is this how we want to deal with lemmatisation in Rasa? 

My impression is that we've always used the spaCy `lemma` features inside of the CountVectorFeaturizer and never really considered how we think about lemmatisation in general. The reason I want to bring it up is related to a [feature request](https://github.com/RasaHQ/rasa-nlu-examples/issues/23) in [rasa-nlu-examples](https://github.com/RasaHQ/rasa-nlu-examples). There are other tools for lemmatization/tokenization that offer support for languages that spaCy currently does not cover. I'd like to add support for them but it might be good to formalise how we want these components to behave. 

@Ghostvv might be in favour of decoupling lemmatisation and the CountVectorFeaturizer.
As soon as we add an option to the `CountVectorFeaturizer` to use the lemma or not, we are decoupling the lemmatisation from the `CountVectorFeaturizer`, aren't we? I think if users want to add another component to the pipeline that does lemmatisation, that is fine as they would need to reuse our interface, e.g. updating the `lemma` attribute of the tokens. 
@tabergma is the `lemma` attribute the only attribute that we want to "countvectorize"? The reason I'm asking is related to (yet another) [feature request](https://github.com/RasaHQ/rasa-nlu-examples/issues/29). In spaCy there are many attributes that could be interesting to featurize. There's flags for stopwords, sentiment and out-of-vocabulary terms. There are all interesting but there's two paths towards an implementation. 

1. We can have a tokenizer that adds all of these attributes to the token and then we can have the CountVectorFeaturizer turn them all into sparse vectors for DIET.
2. We can have a seperate featurizer that handles this directly without the need to attach anything to the tokens. 

I'm currently leaning towards implementing #2 but since there's an option to do something "more general" here I figured I'd at least mention it. 
I think CVF should only featuring tokens/words. The rest is the job of `LexicalSyntacticFeaturizer`

Commit message: update docs

Patch to review:
diff --git a/changelog/6536.feature.md b/changelog/6536.feature.md
new file mode 100644
index 000000000000..db96b0742f25
--- /dev/null
+++ b/changelog/6536.feature.md
@@ -0,0 +1,4 @@
+Add option `use_lemma` to `CountVectorsFeaturizer`. By default it is set to `True`.
+
+`use_lemma` indicates whether the featurizer should use the lemma of a word for counting (if available) or not.
+If this option is set to `False` it will use the word as it is.
diff --git a/docs/docs/components/featurizers.mdx b/docs/docs/components/featurizers.mdx
index 206626af5b7e..c6435323dad2 100644
--- a/docs/docs/components/featurizers.mdx
+++ b/docs/docs/components/featurizers.mdx
@@ -330,6 +330,10 @@ Note: The `feature-dimension` for sequence and sentence features does not have t
   If you want to use character n-grams, set `analyzer` to `char` or `char_wb`.
   The lower and upper boundaries of the n-grams can be configured via the parameters `min_ngram` and `max_ngram`.
   By default both of them are set to `1`.
+  By default the featurizer takes the lemma of a word instead of the word directly if it is available. The lemma
+  of a word is for example set by the `SpacyTokenizer`. You can disable this behavior
+  by setting `use_lemma` to `False`.
+
 
   :::note
   Option `char_wb` creates character n-grams only from text inside word boundaries;
@@ -455,6 +459,8 @@ Note: The `feature-dimension` for sequence and sentence features does not have t
     +-------------------+-------------------------+--------------------------------------------------------------+
     | alias             | CountVectorFeaturizer   | Alias name of featurizer.                                    |
     +-------------------+-------------------------+--------------------------------------------------------------+
+    | use_lemma         | True                    | Use the lemma of words for featurization.                    |
+    +-------------------+-------------------------+--------------------------------------------------------------+
     ```
 
   </details>
diff --git a/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py b/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
index 6b841d6dfc30..f48eb7b5a2c2 100644
--- a/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
+++ b/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
@@ -77,6 +77,9 @@ def required_components(cls) -> List[Type[Component]]:
         # will be converted to lowercase if lowercase is True
         "OOV_token": None,  # string or None
         "OOV_words": [],  # string or list of strings
+        # indicates whether the featurizer should use the lemma of a word for
+        # counting (if available) or not
+        "use_lemma": True,
     }
 
     @classmethod
@@ -114,6 +117,9 @@ def _load_count_vect_params(self) -> None:
         # if convert all characters to lowercase
         self.lowercase = self.component_config["lowercase"]
 
+        # use the lemma of the words or not
+        self.use_lemma = self.component_config["use_lemma"]
+
     # noinspection PyPep8Naming
     def _load_OOV_params(self) -> None:
         self.OOV_token = self.component_config["OOV_token"]
@@ -210,13 +216,15 @@ def __init__(
         # declare class instance for CountVectorizer
         self.vectorizers = vectorizers
 
-    @staticmethod
     def _get_message_tokens_by_attribute(
-        message: "Message", attribute: Text
+        self, message: "Message", attribute: Text
     ) -> List[Text]:
         """Get text tokens of an attribute of a message"""
         if message.get(TOKENS_NAMES[attribute]):
-            return [t.lemma for t in message.get(TOKENS_NAMES[attribute])]
+            return [
+                t.lemma if self.use_lemma else t.text
+                for t in message.get(TOKENS_NAMES[attribute])
+            ]
 
         return message.get(attribute).split()
 
@@ -492,7 +500,11 @@ def train(
         spacy_nlp = kwargs.get("spacy_nlp")
         if spacy_nlp is not None:
             # create spacy lemma_ for OOV_words
-            self.OOV_words = [t.lemma_ for w in self.OOV_words for t in spacy_nlp(w)]
+            self.OOV_words = [
+                t.lemma_ if self.use_lemma else t.text
+                for w in self.OOV_words
+                for t in spacy_nlp(w)
+            ]
 
         # process sentences and collect data for all attributes
         processed_attribute_tokens = self._get_all_attributes_processed_tokens(
diff --git a/tests/nlu/featurizers/test_count_vectors_featurizer.py b/tests/nlu/featurizers/test_count_vectors_featurizer.py
index 42effc649aeb..1e7e33b072a6 100644
--- a/tests/nlu/featurizers/test_count_vectors_featurizer.py
+++ b/tests/nlu/featurizers/test_count_vectors_featurizer.py
@@ -1,10 +1,13 @@
+from typing import List, Text, Any
+
 import numpy as np
 import pytest
 import scipy.sparse
 
+from nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
-from rasa.nlu.constants import TOKENS_NAMES, TEXT, INTENT, RESPONSE
+from rasa.nlu.constants import TOKENS_NAMES, TEXT, INTENT, RESPONSE, SPACY_DOCS
 from rasa.nlu.tokenizers.tokenizer import Token
 from rasa.nlu.training_data import Message
 from rasa.nlu.training_data import TrainingData
@@ -397,3 +400,46 @@ def test_count_vectors_featurizer_train():
     assert sen_vec is None
     assert (1, 1) == seq_vec.shape
     assert np.all(seq_vec.toarray()[0] == np.array([1]))
+
+
+@pytest.mark.parametrize(
+    "sentence, sequence_features, sentence_features, use_lemma",
+    [
+        ("go goes went go", [[1, 0, 0]], [[2, 1, 1]], False),
+        ("go goes went go", [[1]], [[4]], True),
+    ],
+)
+def test_count_vector_featurizer_use_lemma(
+    spacy_nlp: Any,
+    sentence: Text,
+    sequence_features: List[List[int]],
+    sentence_features: List[List[int]],
+    use_lemma: bool,
+):
+    ftr = CountVectorsFeaturizer({"use_lemma": use_lemma})
+
+    train_message = Message(sentence)
+    train_message.set(SPACY_DOCS[TEXT], spacy_nlp(sentence))
+    test_message = Message(sentence)
+    test_message.set(SPACY_DOCS[TEXT], spacy_nlp(sentence))
+
+    SpacyTokenizer().process(train_message)
+    SpacyTokenizer().process(test_message)
+
+    ftr.train(TrainingData([train_message]))
+
+    ftr.process(test_message)
+
+    seq_vecs, sen_vecs = test_message.get_sparse_features(TEXT, [])
+
+    assert isinstance(seq_vecs, scipy.sparse.coo_matrix)
+    assert isinstance(sen_vecs, scipy.sparse.coo_matrix)
+
+    actual_seq_vecs = seq_vecs.toarray()
+    actual_sen_vecs = sen_vecs.toarray()
+
+    print(actual_sen_vecs, sentence_features)
+    print(actual_seq_vecs, sequence_features)
+
+    assert np.all(actual_seq_vecs[0] == sequence_features)
+    assert np.all(actual_sen_vecs[-1] == sentence_features)


Code review comments:
diff_hunk: @@ -397,3 +400,46 @@ def test_count_vectors_featurizer_train():
     assert sen_vec is None
     assert (1, 1) == seq_vec.shape
     assert np.all(seq_vec.toarray()[0] == np.array([1]))
+
+
+@pytest.mark.parametrize(
+    "sentence, sequence_features, sentence_features, use_lemma",
+    [
+        ("go goes went go", [[1, 0, 0]], [[2, 1, 1]], False),
+        ("go goes went go", [[1]], [[4]], True),
+    ],
+)
+def test_count_vector_featurizer_use_lemma(
+    spacy_nlp: Any,
+    sentence: Text,
+    sequence_features: List[List[int]],
+    sentence_features: List[List[int]],
+    use_lemma: bool,
+):
+    ftr = CountVectorsFeaturizer({"use_lemma": use_lemma})
+
+    train_message = Message(sentence)
+    train_message.set(SPACY_DOCS[TEXT], spacy_nlp(sentence))
+    test_message = Message(sentence)
+    test_message.set(SPACY_DOCS[TEXT], spacy_nlp(sentence))
+
+    SpacyTokenizer().process(train_message)
+    SpacyTokenizer().process(test_message)
+
+    ftr.train(TrainingData([train_message]))
+
+    ftr.process(test_message)
+
+    seq_vecs, sen_vecs = test_message.get_sparse_features(TEXT, [])
+
+    assert isinstance(seq_vecs, scipy.sparse.coo_matrix)
+    assert isinstance(sen_vecs, scipy.sparse.coo_matrix)
+
+    actual_seq_vecs = seq_vecs.toarray()
+    actual_sen_vecs = sen_vecs.toarray()
+
+    print(actual_sen_vecs, sentence_features)
path: tests/nlu/featurizers/test_count_vectors_featurizer.py
text: @user1:
what's the goal of printing that in a test? To have the logs in case something fails?

@author:
oh, was just for me when debugging, will remove this