Repository: RasaHQ/rasa
Pull request: Add RegexEntityExtractor
**Proposed changes**:
- closes https://github.com/RasaHQ/rasa/issues/5957
- Add an entity extractor that extracts entities based on a lookup table/regexes

original PR https://github.com/RasaHQ/rasa/pull/6183

**Status (please check what you already did)**:
- [x] added some tests for the functionality
- [x] updated the documentation
- [x] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/master/changelog) for instructions)
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
Add a lookup entity extractor
**Description of Problem**:
Lookup tables are used to create additional features for our entity extractors. In order for the features (aka "the word is part of lookup table xyz, so it is likely that is has the entity xyz") to work, you need to have some examples in your training data with entires from the lookup table.

However, many users think that all entries in the lookup table are extracted as an entity right away. So, as soon as an entry from the lookup table is detected it is extracted as an entity. Many users on the forum struggling with this.

**Overview of the Solution**:
- Create a new component `LookupEntityExtractor` that takes a lookup table. It will extract exact matches of entries in the lookup table and marks them as certain entity. (What about plural/singular/etc. of words? Should we also consider those?)
- We should update the docs to clarify the difference between lookup tables and lookup entity extractor.

Hints from issue discussion:
@tabergma this is an interesting idea -- i definitely think users are confused about this. Do you think a `RegexEntityExtractor` would similarly be helpful? I think some users have implemented it themselves already
@erohmensing Good point 👍 `RegexFeaturizer` could cause a similar issue, might make sense to also add an extractor for it.
@JiteshGaikwad if case you're interested, we'd be interested in a PR relating to your work [here](https://github.com/JiteshGaikwad/LookupEntityExtractor/). 
hey @koaning @tabergma @erohmensing  I have submitted the [PR](https://github.com/RasaHQ/rasa/pull/6183)

Commit message: update component name in tests

Patch to review:
diff --git a/changelog/5957.feature.rst b/changelog/5957.feature.rst
new file mode 100644
index 000000000000..f782ab0ddbca
--- /dev/null
+++ b/changelog/5957.feature.rst
@@ -0,0 +1,4 @@
+Add new entity extractor ``RegexEntityExtractor``. The entity extractor extracts entities using the lookup tables
+and regexes defined in the training data. You can either use the ``RegexEntityExtractor`` or the ``RegexFeaturizer``
+in your pipeline. If you use both components an error will be thrown.
+For more information see :ref:`RegexEntityExtractor`.
diff --git a/docs/nlu/components.rst b/docs/nlu/components.rst
index 8d1a19cdd650..f34cce47f0ce 100644
--- a/docs/nlu/components.rst
+++ b/docs/nlu/components.rst
@@ -469,6 +469,11 @@ RegexFeaturizer
     Regex features for entity extraction are currently only supported by the :ref:`CRFEntityExtractor` and the
     :ref:`diet-classifier` components!
 
+    .. note::
+        You cannot use the ``RegexFeaturizer`` and the :ref:`RegexEntityExtractor` in the same pipeline. You need to
+        decide whether you want to use the lookup tables and regexes to create features or to directly extract entities.
+        It is not possible to do both at the same time.
+
 :Configuration:
     Make the featurizer case insensitive by adding the ``case_sensitive: False`` option, the default being
     ``case_sensitive: True``.
@@ -951,6 +956,36 @@ EntitySynonymMapper
         When using the ``EntitySynonymMapper`` as part of an NLU pipeline, it will need to be placed
         below any entity extractors in the configuration file.
 
+
+.. _RegexEntityExtractor:
+
+RegexEntityExtractor
+~~~~~~~~~~~~~~~~~~~~~
+
+:Short: Extracts entities using the lookup tables and regexes defined in the training data
+:Outputs: ``entities``
+:Requires: Nothing
+:Description:
+    This component extract entities using the lookup tables and regexes defined in the training data.
+    The component checks if the user message contains an entry of one of the lookup tables or matches one of the
+    regexes. If a match is found, the value is extracted as entity.
+
+    .. note::
+        You cannot use the :ref:`RegexFeaturizer` and the ``RegexEntityExtractor`` in the same pipeline. You need to
+        decide whether you want to use the lookup tables and regexes to create features or to directly extract entities.
+        It is not possible to do both at the same time.
+
+:Configuration:
+    By default we convert the entries in the lookup file and the user message to lowercase before we check if there
+    is a match. You can disable this behavior by setting ``lowercase: False``.
+
+    .. code-block:: yaml
+
+        pipeline:
+        - name: LookupEntityExtractor
+          lowercase: True
+
+
 .. _CRFEntityExtractor:
 
 CRFEntityExtractor
diff --git a/docs/nlu/entity-extraction.rst b/docs/nlu/entity-extraction.rst
index 1605aff60c0f..cc0cf10838bf 100644
--- a/docs/nlu/entity-extraction.rst
+++ b/docs/nlu/entity-extraction.rst
@@ -23,6 +23,9 @@ Component                  Requires           Model           	        Notes
 ``EntitySynonymMapper``    existing entities  N/A                       maps known synonyms
 ``DIETClassifier``                            conditional random field
                                               on top of a transformer   good for training custom entities
+``RegexEntityExtractor``                      N/A                       extract entities using the lookup
+                                                                        tables and regexes defined in the
+                                                                        training data
 =========================  =================  ========================  =================================
 
 .. contents::
diff --git a/docs/nlu/training-data-format.rst b/docs/nlu/training-data-format.rst
index b6558e185442..1e36f8dfb0f0 100644
--- a/docs/nlu/training-data-format.rst
+++ b/docs/nlu/training-data-format.rst
@@ -139,33 +139,49 @@ That way you can map synonyms, or misspellings, to the same ``value``.
 
 Regular Expression Features
 ---------------------------
-Regular expressions can be used to support the intent classification and entity extraction. For example, if your entity
-has a deterministic structure (like a zipcode or an email address), you can use a regular expression to ease detection
-of that entity. For the zipcode example it might look like this:
+Regular expressions can be used in two different ways:
+
+1. They can be used to support the intent classification and entity extraction when using the :ref:`RegexFeaturizer`
+component in the pipeline.
+
+2. They can be used to directly extract entities from a user messages when using the :ref:`RegexEntityExtractor`
+component in the pipeline.
+
+For example, if your entity has a deterministic structure (like a zipcode or an email address), you can use a regular
+expression to ease detection of that entity (using the :ref:`RegexFeaturizer`) or to directly extract the entities from
+the user message (using the :ref:`RegexEntityExtractor`). For the zipcode example it might look like this:
 
 .. code-block:: md
 
     ## regex:zipcode
     - [0-9]{5}
 
+In case you are using the regular expressions for the :ref:`RegexFeaturizer` the name of the regular expression does
+not matter. If does not define the entity nor the intent, it is just a human readable description for you to remember
+what this regex is used for and is the title of the corresponding pattern feature.
+If you want to use the :ref:`RegexFeaturizer` you can also use the regex features to improve the intent
+classification performance, for example, by defining a greet clause:
+
+.. code-block:: md
+
     ## regex:greet
     - hey[^\\s]*
 
-The name doesn't define the entity nor the intent, it is just a human readable description for you to remember what
-this regex is used for and is the title of the corresponding pattern feature. As you can see in the above example, you
-can also use the regex features to improve the intent
-classification performance.
+If you are using the regular expressions to directly extract entities using the :ref:`RegexEntityExtractor`, the name
+of the regular expression should match the name of the entity you want to extract.
 
-Try to create your regular expressions in a way that they match as few words as possible. E.g. using ``hey[^\s]*``
+Try to create your regular expressions in a way that they match as few words as possible. E.g. using ``hey[^\\s]*``
 instead of ``hey.*``, as the later one might match the whole message whereas the first one only matches a single word.
 
-Regex features for entity extraction are currently only supported by the ``CRFEntityExtractor`` component! Hence, other
-entity extractors, like ``MitieEntityExtractor`` or ``SpacyEntityExtractor`` won't use the generated features and their
+When using the :ref:`RegexFeaturizer`, the regex features for entity extraction are currently only supported by the
+``CRFEntityExtractor`` and the ``DIETClassifier`` component! Hence, other entity extractors, like
+``MitieEntityExtractor`` or ``SpacyEntityExtractor`` won't use the generated features and their
 presence will not improve entity recognition for these extractors. Currently, all intent classifiers make use of
 available regex features.
 
 .. note::
-    Regex features don't define entities nor intents! They simply provide patterns to help the classifier
+    Regex features only define entities when used in combination with the :ref:`RegexEntityExtractor`. Otherwise they
+    don't define entities nor intents! They simply provide patterns to help the classifier
     recognize entities and related intents. Hence, you still need to provide intent & entity examples as part of your
     training data!
 
@@ -191,16 +207,15 @@ And can be loaded and used as shown here:
     - How about some [mapo tofu](plates)
 
 When lookup tables are supplied in training data, the contents are combined
-into a large, case-insensitive regex pattern that looks for exact matches in
+into a large regex pattern that looks for exact matches in
 the training examples. These regexes match over multiple tokens, so
 ``lettuce wrap`` would match ``get me a lettuce wrap ASAP`` as ``[0 0 0 1 1 0]``.
 These regexes are processed identically to the regular regex patterns
 directly specified in the training data.
 
 .. note::
-    For lookup tables to be effective, there must be a few examples of matches in your training data. Otherwise the
-    model will not learn to use the lookup table match features.
-
+    If you are using lookup tables in combination with :ref:`RegexFeaturizer`, there must be a few examples of matches
+    in your training data. Otherwise the model will not learn to use the lookup table match features.
 
 .. warning::
     You have to be careful when you add data to the lookup table.
diff --git a/rasa/nlu/components.py b/rasa/nlu/components.py
index f2071b44ece6..1bb402e03df5 100644
--- a/rasa/nlu/components.py
+++ b/rasa/nlu/components.py
@@ -102,6 +102,32 @@ def validate_only_one_tokenizer_is_used(pipeline: List["Component"]) -> None:
         )
 
 
+def validate_only_one_regex_component_is_used(pipeline: List["Component"]) -> None:
+    """Validates that RegexEntityExtractor and RegexFeaturizer are not used at the
+    same time.
+
+    Args:
+        pipeline: the list of the :class:`rasa.nlu.components.Component`.
+    """
+
+    from rasa.nlu.featurizers.sparse_featurizer.regex_featurizer import RegexFeaturizer
+    from rasa.nlu.extractors.regex_entity_extractor import RegexEntityExtractor
+
+    regex_featurizer_used = False
+    regex_entity_extractor_used = False
+    for component in pipeline:
+        if isinstance(component, RegexFeaturizer):
+            regex_featurizer_used = True
+        elif isinstance(component, RegexEntityExtractor):
+            regex_entity_extractor_used = True
+
+    if regex_featurizer_used and regex_entity_extractor_used:
+        raise InvalidConfigError(
+            "Your config contains a 'RegexFeaturizer' and a 'RegexEntityExtractor'. "
+            "You should just use one of those components."
+        )
+
+
 def _required_component_in_pipeline(
     required_component: Type["Component"], pipeline: List["Component"]
 ) -> bool:
@@ -179,6 +205,7 @@ def validate_pipeline(pipeline: List["Component"]) -> None:
     validate_empty_pipeline(pipeline)
     validate_only_one_tokenizer_is_used(pipeline)
     validate_required_components(pipeline)
+    validate_only_one_regex_component_is_used(pipeline)
 
 
 def any_components_in_pipeline(components: Iterable[Text], pipeline: List["Component"]):
@@ -238,21 +265,23 @@ def validate_required_components_from_data(
             )
 
     if data.regex_features and not any_components_in_pipeline(
-        ["RegexFeaturizer"], pipeline
+        ["RegexFeaturizer", "RegexEntityExtractor"], pipeline
     ):
         raise_warning(
             "You have defined training data with regexes, but "
-            "your NLU pipeline does not include a 'RegexFeaturizer'. "
-            "To featurize regexes, include a 'RegexFeaturizer' in your pipeline."
+            "your NLU pipeline does not include a 'RegexFeaturizer' or a "
+            "'RegexEntityExtractor'. To use regexes, include either a "
+            "'RegexFeaturizer' or a 'RegexEntityExtractor' in your pipeline."
         )
 
     if data.lookup_tables and not any_components_in_pipeline(
-        ["RegexFeaturizer"], pipeline
+        ["RegexFeaturizer", "RegexEntityExtractor"], pipeline
     ):
         raise_warning(
             "You have defined training data consisting of lookup tables, but "
-            "your NLU pipeline does not include a 'RegexFeaturizer'. "
-            "To featurize lookup tables, add a 'RegexFeaturizer' to your pipeline."
+            "your NLU pipeline does not include a 'RegexFeaturizer' or a "
+            "'RegexEntityExtractor'. To use lookup tables, include either a "
+            "'RegexFeaturizer' or a 'RegexEntityExtractor' in your pipeline."
         )
 
     if data.lookup_tables:
diff --git a/rasa/nlu/extractors/regex_entity_extractor.py b/rasa/nlu/extractors/regex_entity_extractor.py
new file mode 100644
index 000000000000..cd3ecf7867db
--- /dev/null
+++ b/rasa/nlu/extractors/regex_entity_extractor.py
@@ -0,0 +1,112 @@
+import logging
+import os
+import re
+from typing import Any, Dict, List, Optional, Text
+
+import rasa.utils.io as io_utils
+import rasa.nlu.utils.pattern_utils as pattern_utils
+from rasa.nlu.model import Metadata
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.training_data import TrainingData
+from rasa.nlu.constants import (
+    ENTITIES,
+    ENTITY_ATTRIBUTE_TYPE,
+    ENTITY_ATTRIBUTE_START,
+    ENTITY_ATTRIBUTE_VALUE,
+    ENTITY_ATTRIBUTE_END,
+)
+from rasa.nlu.training_data import Message
+from rasa.nlu.extractors.extractor import EntityExtractor
+
+logger = logging.getLogger(__name__)
+
+
+class RegexEntityExtractor(EntityExtractor):
+    """Searches for entities in the user's message using the lookup tables and regexes
+    defined in the training data."""
+
+    defaults = {
+        # lower case the entity value from the lookup file and
+        # user message while comparing them
+        "lowercase": True
+    }
+
+    def __init__(
+        self,
+        component_config: Optional[Dict[Text, Any]] = None,
+        patterns: Optional[List[Dict[Text, Text]]] = None,
+    ):
+        super(RegexEntityExtractor, self).__init__(component_config)
+
+        self.lowercase = self.component_config["lowercase"]
+        self.patterns = patterns or []
+
+    def train(
+        self,
+        training_data: TrainingData,
+        config: Optional[RasaNLUModelConfig] = None,
+        **kwargs: Any,
+    ) -> None:
+        self.patterns = pattern_utils.extract_patterns(training_data)
+
+    def process(self, message: Message, **kwargs: Any) -> None:
+        extracted_entities = self._extract_entities(message)
+        extracted_entities = self.add_extractor_name(extracted_entities)
+
+        message.set(
+            ENTITIES, message.get(ENTITIES, []) + extracted_entities, add_to_output=True
+        )
+
+    def _extract_entities(self, message: Message) -> List[Dict[Text, Any]]:
+        """Extract entities of the given type from the given user message."""
+        entities = []
+
+        flags = 0  # default flag
+        if self.lowercase:
+            flags = re.IGNORECASE
+
+        for pattern in self.patterns:
+            matches = re.finditer(pattern["pattern"], message.text, flags=flags)
+            matches = list(matches)
+
+            for match in matches:
+                start_index = match.start()
+                end_index = match.end()
+                entities.append(
+                    {
+                        ENTITY_ATTRIBUTE_TYPE: pattern["name"],
+                        ENTITY_ATTRIBUTE_START: start_index,
+                        ENTITY_ATTRIBUTE_END: end_index,
+                        ENTITY_ATTRIBUTE_VALUE: message.text[start_index:end_index],
+                    }
+                )
+
+        return entities
+
+    @classmethod
+    def load(
+        cls,
+        meta: Dict[Text, Any],
+        model_dir: Optional[Text] = None,
+        model_metadata: Optional[Metadata] = None,
+        cached_component: Optional["RegexEntityExtractor"] = None,
+        **kwargs: Any,
+    ) -> "RegexEntityExtractor":
+
+        file_name = meta.get("file")
+        regex_file = os.path.join(model_dir, file_name)
+
+        if os.path.exists(regex_file):
+            patterns = io_utils.read_json_file(regex_file)
+            return RegexEntityExtractor(meta, patterns=patterns)
+
+        return RegexEntityExtractor(meta)
+
+    def persist(self, file_name: Text, model_dir: Text) -> Optional[Dict[Text, Any]]:
+        """Persist this model into the passed directory.
+        Return the metadata necessary to load the model again."""
+        file_name = f"{file_name}.json"
+        regex_file = os.path.join(model_dir, file_name)
+        io_utils.dump_obj_as_json_to_file(regex_file, self.patterns)
+
+        return {"file": file_name}
diff --git a/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py b/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
index 4d61ac4ba3cb..0830607b5985 100644
--- a/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
+++ b/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
@@ -7,7 +7,7 @@
 import scipy.sparse
 
 import rasa.utils.io
-import rasa.utils.io
+import rasa.nlu.utils.pattern_utils as pattern_utils
 from rasa.nlu import utils
 from rasa.nlu.components import Component
 from rasa.nlu.config import RasaNLUModelConfig
@@ -48,9 +48,6 @@ def __init__(
         self.known_patterns = known_patterns if known_patterns else []
         self.case_sensitive = self.component_config["case_sensitive"]
 
-    def add_lookup_tables(self, lookup_tables: List[Dict[Text, Union[Text, List]]]):
-        self.known_patterns.extend(self._lookup_table_regexes(lookup_tables))
-
     def train(
         self,
         training_data: TrainingData,
@@ -58,9 +55,7 @@ def train(
         **kwargs: Any,
     ) -> None:
 
-        self.known_patterns = training_data.regex_features
-
-        self.add_lookup_tables(training_data.lookup_tables)
+        self.known_patterns = pattern_utils.extract_patterns(training_data)
 
         for example in training_data.training_examples:
             for attribute in [TEXT, RESPONSE]:
@@ -93,18 +88,6 @@ def _text_features_with_regex(self, message: Message, attribute: Text) -> None:
                 )
                 message.add_features(final_sentence_features)
 
-    def _lookup_table_regexes(
-        self, lookup_tables: List[Dict[Text, Any]]
-    ) -> List[Dict[Text, Text]]:
-        """appends the regex features from the lookup tables to self.known_patterns"""
-
-        patterns = []
-        for table in lookup_tables:
-            regex_pattern = self._generate_lookup_regex(table)
-            lookup_regex = {"name": table["name"], "pattern": regex_pattern}
-            patterns.append(lookup_regex)
-        return patterns
-
     def _features_for_patterns(
         self, message: Message, attribute: Text
     ) -> Tuple[Optional[scipy.sparse.coo_matrix], Optional[scipy.sparse.coo_matrix]]:
@@ -156,40 +139,6 @@ def _features_for_patterns(
             scipy.sparse.coo_matrix(sentence_features),
         )
 
-    def _generate_lookup_regex(
-        self, lookup_table: Dict[Text, Union[Text, List[Text]]]
-    ) -> Text:
-        """creates a regex out of the contents of a lookup table file"""
-        lookup_elements = lookup_table["elements"]
-        elements_to_regex = []
-
-        # if it's a list, it should be the elements directly
-        if isinstance(lookup_elements, list):
-            elements_to_regex = lookup_elements
-        # otherwise it's a file path.
-        else:
-
-            try:
-                f = open(lookup_elements, "r", encoding=rasa.utils.io.DEFAULT_ENCODING)
-            except OSError:
-                raise ValueError(
-                    f"Could not load lookup table {lookup_elements}. "
-                    f"Please make sure you've provided the correct path."
-                )
-
-            with f:
-                for line in f:
-                    new_element = line.strip()
-                    if new_element:
-                        elements_to_regex.append(new_element)
-
-        # sanitize the regex, escape special characters
-        elements_sanitized = [re.escape(e) for e in elements_to_regex]
-
-        # regex matching elements with word boundaries on either side
-        regex_string = "(?i)(\\b" + "\\b|\\b".join(elements_sanitized) + "\\b)"
-        return regex_string
-
     @classmethod
     def load(
         cls,
diff --git a/rasa/nlu/registry.py b/rasa/nlu/registry.py
index f4330dc87412..ea24511df7b5 100644
--- a/rasa/nlu/registry.py
+++ b/rasa/nlu/registry.py
@@ -18,6 +18,7 @@
 from rasa.nlu.extractors.entity_synonyms import EntitySynonymMapper
 from rasa.nlu.extractors.mitie_entity_extractor import MitieEntityExtractor
 from rasa.nlu.extractors.spacy_entity_extractor import SpacyEntityExtractor
+from rasa.nlu.extractors.regex_entity_extractor import RegexEntityExtractor
 from rasa.nlu.featurizers.sparse_featurizer.lexical_syntactic_featurizer import (
     LexicalSyntacticFeaturizer,
 )
@@ -69,6 +70,7 @@
     CRFEntityExtractor,
     DucklingHTTPExtractor,
     EntitySynonymMapper,
+    RegexEntityExtractor,
     # featurizers
     SpacyFeaturizer,
     MitieFeaturizer,
diff --git a/rasa/nlu/utils/pattern_utils.py b/rasa/nlu/utils/pattern_utils.py
new file mode 100644
index 000000000000..b0ae527adfc6
--- /dev/null
+++ b/rasa/nlu/utils/pattern_utils.py
@@ -0,0 +1,100 @@
+import re
+from typing import Any, Dict, List, Text, Union
+
+import rasa.utils.io as io_utils
+import rasa.utils.common as common_utils
+from rasa.nlu.training_data import TrainingData
+
+
+def _convert_lookup_tables_to_regex(
+    lookup_tables: List[Dict[Text, Any]]
+) -> List[Dict[Text, Text]]:
+    """Convert the lookup tables to regex patterns.
+    Args:
+        lookup_tables: the lookup tables
+
+    Returns:
+        A list of regex patterns.
+    """
+    patterns = []
+    for table in lookup_tables:
+        regex_pattern = _generate_lookup_regex(table)
+        lookup_regex = {"name": table["name"], "pattern": regex_pattern}
+        patterns.append(lookup_regex)
+    return patterns
+
+
+def _generate_lookup_regex(lookup_table: Dict[Text, Union[Text, List[Text]]]) -> Text:
+    """Creates a regex pattern from the given lookup table.
+
+    The lookup table is either a file or a list of entries.
+
+    Args:
+        lookup_table: The lookup table.
+
+    Returns:
+        The regex pattern.
+    """
+    lookup_elements = lookup_table["elements"]
+
+    # if it's a list, it should be the elements directly
+    if isinstance(lookup_elements, list):
+        elements_to_regex = lookup_elements
+    # otherwise it's a file path.
+    else:
+        elements_to_regex = _read_lookup_table_file(lookup_elements)
+
+    # sanitize the regex, escape special characters
+    elements_sanitized = [re.escape(e) for e in elements_to_regex]
+
+    # regex matching elements with word boundaries on either side
+    return "(\\b" + "\\b|\\b".join(elements_sanitized) + "\\b)"
+
+
+def _read_lookup_table_file(lookup_table_file: Text) -> List[Text]:
+    """Read the lookup table file.
+
+    Args:
+        lookup_table_file: the file path to the lookup table
+
+    Returns:
+        Elements listed in the lookup table file.
+    """
+    try:
+        f = open(lookup_table_file, "r", encoding=io_utils.DEFAULT_ENCODING)
+    except OSError:
+        raise ValueError(
+            f"Could not load lookup table {lookup_table_file}. "
+            f"Please make sure you've provided the correct path."
+        )
+
+    elements_to_regex = []
+    with f:
+        for line in f:
+            new_element = line.strip()
+            if new_element:
+                elements_to_regex.append(new_element)
+    return elements_to_regex
+
+
+def extract_patterns(training_data: TrainingData) -> List[Dict[Text, Text]]:
+    """Extract a list of patterns from the training data.
+
+    The patterns are constructed using the regex features and lookup tables defined
+    in the training data.
+
+    Args:
+        training_data: The training data.
+
+    Returns:
+        The list of regex patterns.
+    """
+    if not training_data.lookup_tables and not training_data.regex_features:
+        common_utils.raise_warning(
+            "No lookup tables or regexes defined in the training data."
+        )
+        return []
+
+    patterns = training_data.regex_features
+    patterns.extend(_convert_lookup_tables_to_regex(training_data.lookup_tables))
+    return patterns
diff --git a/tests/nlu/extractors/test_regex_entity_extractor.py b/tests/nlu/extractors/test_regex_entity_extractor.py
new file mode 100644
index 000000000000..59a0fdf4ffcd
--- /dev/null
+++ b/tests/nlu/extractors/test_regex_entity_extractor.py
@@ -0,0 +1,187 @@
+from typing import Any, Text, Dict, List
+
+import pytest
+
+from rasa.nlu.training_data import TrainingData
+from rasa.nlu.constants import ENTITIES
+from rasa.nlu.training_data import Message
+from rasa.nlu.extractors.regex_entity_extractor import RegexEntityExtractor
+
+
+@pytest.mark.parametrize(
+    "text, lookup, expected_entities",
+    [
+        (
+            "Berlin and London are cities.",
+            [
+                {
+                    "name": "city",
+                    "elements": ["Berlin", "Amsterdam", "New York", "London"],
+                }
+            ],
+            [
+                {
+                    "entity": "city",
+                    "value": "Berlin",
+                    "start": 0,
+                    "end": 6,
+                    "extractor": "RegexEntityExtractor",
+                },
+                {
+                    "entity": "city",
+                    "value": "London",
+                    "start": 11,
+                    "end": 17,
+                    "extractor": "RegexEntityExtractor",
+                },
+            ],
+        ),
+        (
+            "Sophie is visiting Thomas in Berlin.",
+            [
+                {
+                    "name": "city",
+                    "elements": ["Berlin", "Amsterdam", "New York", "London"],
+                },
+                {"name": "person", "elements": ["Max", "John", "Sophie", "Lisa"]},
+            ],
+            [
+                {
+                    "entity": "city",
+                    "value": "Berlin",
+                    "start": 29,
+                    "end": 35,
+                    "extractor": "RegexEntityExtractor",
+                },
+                {
+                    "entity": "person",
+                    "value": "Sophie",
+                    "start": 0,
+                    "end": 6,
+                    "extractor": "RegexEntityExtractor",
+                },
+            ],
+        ),
+        (
+            "Rasa is great.",
+            [
+                {
+                    "name": "city",
+                    "elements": ["Berlin", "Amsterdam", "New York", "London"],
+                },
+                {"name": "person", "elements": ["Max", "John", "Sophie", "Lisa"]},
+            ],
+            [],
+        ),
+    ],
+)
+def test_process(
+    text: Text,
+    lookup: List[Dict[Text, List[Text]]],
+    expected_entities: List[Dict[Text, Any]],
+):
+    message = Message(text)
+
+    training_data = TrainingData()
+    training_data.lookup_tables = lookup
+
+    entity_extractor = RegexEntityExtractor()
+    entity_extractor.train(training_data)
+    entity_extractor.process(message)
+
+    entities = message.get(ENTITIES)
+    assert entities == expected_entities
+
+
+@pytest.mark.parametrize(
+    "text, lowercase, lookup, expected_entities",
+    [
+        (
+            "berlin and London are cities.",
+            False,
+            [
+                {
+                    "name": "city",
+                    "elements": ["Berlin", "Amsterdam", "New York", "London"],
+                }
+            ],
+            [
+                {
+                    "entity": "city",
+                    "value": "London",
+                    "start": 11,
+                    "end": 17,
+                    "extractor": "RegexEntityExtractor",
+                }
+            ],
+        ),
+        (
+            "berlin and London are cities.",
+            True,
+            [
+                {
+                    "name": "city",
+                    "elements": ["Berlin", "Amsterdam", "New York", "london"],
+                }
+            ],
+            [
+                {
+                    "entity": "city",
+                    "value": "berlin",
+                    "start": 0,
+                    "end": 6,
+                    "extractor": "RegexEntityExtractor",
+                },
+                {
+                    "entity": "city",
+                    "value": "London",
+                    "start": 11,
+                    "end": 17,
+                    "extractor": "RegexEntityExtractor",
+                },
+            ],
+        ),
+    ],
+)
+def test_lowercase(
+    text: Text,
+    lowercase: bool,
+    lookup: List[Dict[Text, List[Text]]],
+    expected_entities: List[Dict[Text, Any]],
+):
+    message = Message(text)
+    training_data = TrainingData()
+    training_data.lookup_tables = lookup
+
+    entity_extractor = RegexEntityExtractor({"lowercase": lowercase})
+    entity_extractor.train(training_data)
+    entity_extractor.process(message)
+
+    entities = message.get(ENTITIES)
+    assert entities == expected_entities
+
+
+def test_do_not_overwrite_any_entities():
+    message = Message("Max lives in Berlin.")
+    message.set(ENTITIES, [{"entity": "person", "value": "Max", "start": 0, "end": 3}])
+
+    training_data = TrainingData()
+    training_data.lookup_tables = [
+        {"name": "city", "elements": ["London", "Berlin", "Amsterdam"]}
+    ]
+
+    entity_extractor = RegexEntityExtractor()
+    entity_extractor.train(training_data)
+    entity_extractor.process(message)
+
+    entities = message.get(ENTITIES)
+    assert entities == [
+        {"entity": "person", "value": "Max", "start": 0, "end": 3},
+        {
+            "entity": "city",
+            "value": "Berlin",
+            "start": 13,
+            "end": 19,
+            "extractor": "RegexEntityExtractor",
+        },
+    ]
diff --git a/tests/nlu/featurizers/test_regex_featurizer.py b/tests/nlu/featurizers/test_regex_featurizer.py
index cc0117714221..b64717a11a51 100644
--- a/tests/nlu/featurizers/test_regex_featurizer.py
+++ b/tests/nlu/featurizers/test_regex_featurizer.py
@@ -132,7 +132,9 @@ def test_lookup_tables(sentence, expected, labeled_tokens, spacy_nlp):
         {"name": "plates", "elements": "data/test/lookup_tables/plates.txt"},
     ]
     ftr = RegexFeaturizer()
-    ftr.add_lookup_tables(lookups)
+    training_data = TrainingData()
+    training_data.lookup_tables = lookups
+    ftr.train(training_data)
 
     # adds tokens to the message
     component_config = {"name": "SpacyTokenizer"}
diff --git a/tests/nlu/test_train.py b/tests/nlu/test_train.py
index 6a136e26a86c..8e5351d78b45 100644
--- a/tests/nlu/test_train.py
+++ b/tests/nlu/test_train.py
@@ -63,7 +63,11 @@ def pipelines_for_tests():
         (
             "en",
             as_pipeline(
-                "MitieNLP", "MitieTokenizer", "MitieFeaturizer", "MitieIntentClassifier"
+                "MitieNLP",
+                "MitieTokenizer",
+                "MitieFeaturizer",
+                "MitieIntentClassifier",
+                "RegexEntityExtractor",
             ),
         ),
         (
diff --git a/tests/nlu/utilities.py b/tests/nlu/utilities.py
index dd55d9e636ac..6858967e5506 100644
--- a/tests/nlu/utilities.py
+++ b/tests/nlu/utilities.py
@@ -1,4 +1,6 @@
 import tempfile
+from typing import Text, List
+
 import ruamel.yaml as yaml
 
 from rasa.nlu.model import Interpreter
@@ -14,6 +16,15 @@ def write_file_config(file_config):
         return f
 
 
+def write_tmp_file(content: List[Text]) -> Text:
+    with tempfile.NamedTemporaryFile("w+", delete=False) as f:
+        for c in content:
+            f.write(c)
+            f.write("\n")
+        f.flush()
+        return f.name
+
+
 async def interpreter_for(component_builder, data, path, config):
     (trained, _, path) = await train(
         config, data, path, component_builder=component_builder
diff --git a/tests/nlu/utils/test_pattern_utils.py b/tests/nlu/utils/test_pattern_utils.py
new file mode 100644
index 000000000000..16bb0bfbc7b2
--- /dev/null
+++ b/tests/nlu/utils/test_pattern_utils.py
@@ -0,0 +1,57 @@
+from typing import Dict, List, Text
+
+import pytest
+
+import rasa.nlu.utils.pattern_utils as pattern_utils
+from rasa.nlu.training_data import TrainingData
+
+
+@pytest.mark.parametrize(
+    "lookup_tables, regex_features, expected_patterns",
+    [
+        (
+            {"name": "person", "elements": ["Max", "John"]},
+            {},
+            [{"name": "person", "pattern": "(\\bMax\\b|\\bJohn\\b)"}],
+        ),
+        ({}, {}, []),
+        (
+            {},
+            {"name": "zipcode", "pattern": "[0-9]{5}"},
+            [{"name": "zipcode", "pattern": "[0-9]{5}"}],
+        ),
+        (
+            {"name": "person", "elements": ["Max", "John"]},
+            {"name": "zipcode", "pattern": "[0-9]{5}"},
+            [
+                {"name": "zipcode", "pattern": "[0-9]{5}"},
+                {"name": "person", "pattern": "(\\bMax\\b|\\bJohn\\b)"},
+            ],
+        ),
+        (
+            {"name": "plates", "elements": "data/test/lookup_tables/plates.txt"},
+            {"name": "zipcode", "pattern": "[0-9]{5}"},
+            [
+                {"name": "zipcode", "pattern": "[0-9]{5}"},
+                {
+                    "name": "plates",
+                    "pattern": "(\\btacos\\b|\\bbeef\\b|\\bmapo\\ tofu\\b|\\bburrito\\b|\\blettuce\\ wrap\\b)",
+                },
+            ],
+        ),
+    ],
+)
+def test_extract_patterns(
+    lookup_tables: Dict[Text, List[Text]],
+    regex_features: Dict[Text, Text],
+    expected_patterns: Dict[Text, Text],
+):
+    training_data = TrainingData()
+    if lookup_tables:
+        training_data.lookup_tables = [lookup_tables]
+    if regex_features:
+        training_data.regex_features = [regex_features]
+
+    actual_patterns = pattern_utils.extract_patterns(training_data)
+
+    assert actual_patterns == expected_patterns


Code review comments:
diff_hunk: @@ -102,6 +102,32 @@ def validate_only_one_tokenizer_is_used(pipeline: List["Component"]) -> None:
         )
 
 
+def validate_only_one_regex_component_is_used(pipeline: List["Component"]) -> None:
+    """Validates that RegexEntityExtractor and RegexFeaturizer are not used at the
+    same time.
+
+    Args:
+        pipeline: the list of the :class:`rasa.nlu.components.Component`.
+    """
+
+    from rasa.nlu.featurizers.sparse_featurizer.regex_featurizer import RegexFeaturizer
+    from rasa.nlu.extractors.regex_entity_extractor import RegexEntityExtractor
+
+    regex_featurizer_used = False
+    regex_entity_extractor_used = False
+    for component in pipeline:
+        if isinstance(component, RegexFeaturizer):
+            regex_featurizer_used = True
+        elif isinstance(component, RegexEntityExtractor):
+            regex_entity_extractor_used = True
+
+    if regex_featurizer_used and regex_entity_extractor_used:
+        raise InvalidConfigError(
path: rasa/nlu/components.py
text: @user1:
I would not raise here, but rather make a debug statement. I could imagine a situation where it useful to have both

@author:
Just had a brief discussion with @user2 about it. Don't you think it might be confusing for users as the lookup tables/regexes are used differently/have a different meaning depending on the component? Also in the docs we point our that you can also use regexes for intent classification to indicate an utterance follows a certain schema and should most likely be classified as intent xyz. Those regexes should not be there if you use the `RegexEntityExtractor`.

@user1:
can we reuse sentence features system for these regexes? then only sequence regexes will be used for entities

@author:
Not sure what you mean. Currently, `RegexFeaturizer` produces sentence and sequence features because of this. Do you suggest to remove the sentence features and update the docs to just mention that those features are good for entities?

@user1:
No, I meant in this case we only sequence regexes for entities

@user1:
then intent regexes will not influence `RegexEntityExtractor`

@author:
Currently we cannot distinguish between intent and entity regexes. But maybe I guess we could filter out only those regexes in the `RegexEntityExtractor` which have a name that is in the list of entities. Will think about something.

diff_hunk: @@ -0,0 +1,4 @@
+Add new entity extractor ``RegexEntityExtractor``. The entity extractor extracts entities using the lookup tables
+and regexes defined in the training data. You can either use the ``RegexEntityExtractor`` or the ``RegexFeaturizer``
+in your pipeline. If you use both components an error will be thrown.
path: changelog/5957.feature.rst
text: @user1:
let's remove the error. Why not use both components?

@user1:
e.g. regexentityextractor with only lookup (see comment below) and actual regexes as features

diff_hunk: @@ -951,6 +956,36 @@ EntitySynonymMapper
         When using the ``EntitySynonymMapper`` as part of an NLU pipeline, it will need to be placed
         below any entity extractors in the configuration file.
 
+
+.. _RegexEntityExtractor:
+
+RegexEntityExtractor
+~~~~~~~~~~~~~~~~~~~~~
+
+:Short: Extracts entities using the lookup tables and regexes defined in the training data
+:Outputs: ``entities``
+:Requires: Nothing
+:Description:
+    This component extract entities using the lookup tables and regexes defined in the training data.
+    The component checks if the user message contains an entry of one of the lookup tables or matches one of the
+    regexes. If a match is found, the value is extracted as entity.
+
+    .. note::
+        You cannot use the :ref:`RegexFeaturizer` and the ``RegexEntityExtractor`` in the same pipeline. You need to
+        decide whether you want to use the lookup tables and regexes to create features or to directly extract entities.
+        It is not possible to do both at the same time.
+
+:Configuration:
path: docs/nlu/components.rst
text: @user1:
I'd add a config to use only lookup or only regex or both. E.g. I can imagine when users want to use only lookup tables

@author:
Ah ok, so you would do something like this?
```
- name: RegexEntityExtractor
  use_lookup_tables: True
  use_regex_features: False
  lowercase: True
```
Would you then also add the same to the `RegexFeaturizer`? 

@user1:
I think it's fine for featurizer to use all, but maybe for consistency we could do that

@author:
Ok. Vincent mentioned a similar point but we weren't sure if the config would become even more complicated with this addition. But I guess we can add this.

@user1:
since we have defaults, this config becomes entirely optional

diff_hunk: @@ -951,6 +956,36 @@ EntitySynonymMapper
         When using the ``EntitySynonymMapper`` as part of an NLU pipeline, it will need to be placed
         below any entity extractors in the configuration file.
 
+
+.. _RegexEntityExtractor:
+
+RegexEntityExtractor
+~~~~~~~~~~~~~~~~~~~~~
+
+:Short: Extracts entities using the lookup tables and regexes defined in the training data
+:Outputs: ``entities``
+:Requires: Nothing
+:Description:
+    This component extract entities using the lookup tables and regexes defined in the training data.
+    The component checks if the user message contains an entry of one of the lookup tables or matches one of the
+    regexes. If a match is found, the value is extracted as entity.
+
+    .. note::
+        You cannot use the :ref:`RegexFeaturizer` and the ``RegexEntityExtractor`` in the same pipeline. You need to
+        decide whether you want to use the lookup tables and regexes to create features or to directly extract entities.
+        It is not possible to do both at the same time.
+
+:Configuration:
+    By default we convert the entries in the lookup file and the user message to lowercase before we check if there
+    is a match. You can disable this behavior by setting ``lowercase: False``.
+
+    .. code-block:: yaml
+
+        pipeline:
+        - name: LookupEntityExtractor
path: docs/nlu/components.rst
text: @user1:
```suggestion
        - name: RegexEntityExtractor
```