Repository: RasaHQ/rasa
Pull request: Add validation for duplicated data example in multiple intents
Closes #4566 
**Proposed changes**:
- The command `rasa data validate` checks there is no duplicated example data at nlu file.

**Expected Behavior**:
- nlu data file has some intents as shown:
```
## intent:greet
- hello

## intent:goodbye
- hello
```
- The following warning should be displayed when running `rasa data validate`:
```
WARNING  rasa.core.validator  - The example 'hello' was found in these multiples intents: goodbye, greet
```

**Status (please check what you already did)**:
- [x] made PR ready for code review
- [x] added some tests for the functionality
- [x] updated the documentation
- [x] updated the changelog
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa_nlu#code-style) for instructions)


Problem statement:
Utterance Replication Checker
**Description of Problem**:
I just wonder, if Rasa would have an feature which checks if an utterances added before to the training set before. If so, it may warn the developer. This enables utterance duplication/replication in the training set. Otherwise same utterance may be added to different intents accidentally, particularly if the intents are similar to each other and if you have a large training set. So this might hurt the model performance. 

**Overview of the Solution**:
<!-- What would a possible solution look like?
Describe, without going too low into technical details,
what changes need to happen during implementation of this feature. -->

**Examples** 
For example: 

**## intent: payBill**
- pay bill


**## intent: payment_Verification**
- pay bill

Developer my put the same utterance under the different intents.

Hints from issue discussion:
this would be a nice addition to the `rasa data validate` command https://rasa.com/docs/rasa/user-guide/validate-files/#validate-files - would you be up for contributing this ?

Commit message: Added docs for validator that checks duplication in distinct intents

Signed-off-by: Vinicius Lima <vfbdelima@gmail.com>

Patch to review:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 0e406f2f6313..5022563066f6 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -27,6 +27,7 @@ Added
   (``rasa.core.agent.handle_channels()``). The number of workers can be set using the
   environment variable ``SANIC_WORKERS`` (default: 1). A value of >1 is allowed only in
   combination with ``RedisLockStore`` as the lock store.
+- Added data validator that checks there is no duplicated example data across multiples intents
 
 Changed
 -------
diff --git a/docs/user-guide/validate-files.rst b/docs/user-guide/validate-files.rst
index 2a9d2017abae..8c6497259f67 100644
--- a/docs/user-guide/validate-files.rst
+++ b/docs/user-guide/validate-files.rst
@@ -30,6 +30,8 @@ which has the following methods:
 
 **verify_intents():** Checks if intents listed in domain file are consistent with the NLU data.
 
+**verify_example_repetition_in_intents():** Checks if there is no duplicated data among distinct intents at NLU data.
+
 **verify_intents_in_stories():** Verification for intents in the stories, to check if they are valid.
 
 **verify_utterances():** Checks domain file for consistency between utterance templates and utterances listed under
diff --git a/rasa/core/validator.py b/rasa/core/validator.py
index 1feb3834d85e..9585ee5c8475 100644
--- a/rasa/core/validator.py
+++ b/rasa/core/validator.py
@@ -1,5 +1,6 @@
 import logging
 import asyncio
+from collections import defaultdict
 from typing import List, Set, Text
 from rasa.core.domain import Domain
 from rasa.importers.importer import TrainingDataImporter
@@ -57,13 +58,43 @@ def verify_intents(self, ignore_warnings: bool = True) -> bool:
 
         return everything_is_alright
 
+    def verify_example_repetition_in_intents(
+        self, ignore_warnings: bool = True
+    ) -> bool:
+        """Checks if there is no duplicated example in different intents."""
+
+        everything_is_alright = True
+
+        duplication_hash = defaultdict(set)
+        duplicated_examples = []
+        for msg in self.intents.intent_examples:
+            msg_dict = msg.as_dict_nlu()
+            text = msg_dict.get("text")
+            duplication_hash[text].add(msg_dict.get("intent"))
+
+            if len(duplication_hash[text]) > 1:
+                duplicated_examples.append((text, duplication_hash[text]))
+                everything_is_alright = ignore_warnings and everything_is_alright
+
+        for text, intents in duplicated_examples:
+            logger.warning(
+                "The example '{}' was found in these multiples intents: {}".format(
+                    text, ", ".join(intents)
+                )
+            )
+        return everything_is_alright
+
     def verify_intents_in_stories(self, ignore_warnings: bool = True) -> bool:
         """Checks intents used in stories.
 
         Verifies if the intents used in the stories are valid, and whether
         all valid intents are used in the stories."""
 
-        everything_is_alright = self.verify_intents()
+        everything_is_alright = self.verify_intents(ignore_warnings)
+        everything_is_alright = (
+            self.verify_example_repetition_in_intents(ignore_warnings)
+            and everything_is_alright
+        )
 
         stories_intents = {
             event.intent["name"]
diff --git a/tests/core/test_validator.py b/tests/core/test_validator.py
index adbe2a8e6b29..1c9b932e9450 100644
--- a/tests/core/test_validator.py
+++ b/tests/core/test_validator.py
@@ -55,3 +55,23 @@ async def test_fail_on_invalid_utterances(tmpdir):
     importer = RasaFileImporter(domain_path=invalid_domain)
     validator = await Validator.from_importer(importer)
     assert not validator.verify_utterances()
+
+
+async def test_verify_there_is_example_repetition_in_intents():
+    # moodbot nlu data already has duplicated example 'good afternoon'
+    # for intents greet and goodbye
+    importer = RasaFileImporter(
+        domain_path="examples/moodbot/domain.yml",
+        training_data_paths=["examples/moodbot/data/nlu.md"],
+    )
+    validator = await Validator.from_importer(importer)
+    assert not validator.verify_example_repetition_in_intents(False)
+
+
+async def test_verify_there_is_not_example_repetition_in_intents():
+    importer = RasaFileImporter(
+        domain_path="examples/moodbot/domain.yml",
+        training_data_paths=["examples/knowledgebasebot/data/nlu.md"],
+    )
+    validator = await Validator.from_importer(importer)
+    assert validator.verify_example_repetition_in_intents(False)


Code review comments:
diff_hunk: @@ -57,13 +58,43 @@ def verify_intents(self, ignore_warnings: bool = True) -> bool:
 
         return everything_is_alright
 
+    def verify_example_repetition_in_intents(
+        self, ignore_warnings: bool = True
+    ) -> bool:
+        """Checks if there is no duplicated example in different intents."""
+
+        everything_is_alright = True
+
+        duplication_hash = defaultdict(set)
+        duplicated_examples = []
+        for msg in self.intents.intent_examples:
+            msg_dict = msg.as_dict_nlu()
+            text = msg_dict.get("text")
+            duplication_hash[text].add(msg_dict.get("intent"))
path: rasa/core/validator.py
text: @user1:
```suggestion
            duplication_hash[text].add(example.get("intent"))
```

diff_hunk: @@ -57,13 +58,43 @@ def verify_intents(self, ignore_warnings: bool = True) -> bool:
 
         return everything_is_alright
 
+    def verify_example_repetition_in_intents(
+        self, ignore_warnings: bool = True
+    ) -> bool:
+        """Checks if there is no duplicated example in different intents."""
+
+        everything_is_alright = True
+
+        duplication_hash = defaultdict(set)
+        duplicated_examples = []
+        for msg in self.intents.intent_examples:
+            msg_dict = msg.as_dict_nlu()
path: rasa/core/validator.py
text: @user1:
```suggestion
```

diff_hunk: @@ -57,13 +58,43 @@ def verify_intents(self, ignore_warnings: bool = True) -> bool:
 
         return everything_is_alright
 
+    def verify_example_repetition_in_intents(
+        self, ignore_warnings: bool = True
+    ) -> bool:
+        """Checks if there is no duplicated example in different intents."""
+
+        everything_is_alright = True
+
+        duplication_hash = defaultdict(set)
+        duplicated_examples = []
+        for msg in self.intents.intent_examples:
path: rasa/core/validator.py
text: @user1:
```suggestion
        for example in self.intents.intent_examples:
```

diff_hunk: @@ -57,13 +58,43 @@ def verify_intents(self, ignore_warnings: bool = True) -> bool:
 
         return everything_is_alright
 
+    def verify_example_repetition_in_intents(
+        self, ignore_warnings: bool = True
+    ) -> bool:
+        """Checks if there is no duplicated example in different intents."""
+
+        everything_is_alright = True
+
+        duplication_hash = defaultdict(set)
+        duplicated_examples = []
+        for msg in self.intents.intent_examples:
+            msg_dict = msg.as_dict_nlu()
+            text = msg_dict.get("text")
path: rasa/core/validator.py
text: @user1:
```suggestion
            text = example.text
```