Repository: RasaHQ/rasa
Pull request: Exit early if domain file is invalid
**Proposed changes**:

Closes #4619
* The command `rasa data validate` must exit early if the domain can't be loaded, avoiding a bunch of errors to be shown.

**Expected behavior**:

* If the domain file is configurated as shown below:
```
intents:
  - greet

actions:
- utter_greet
- utter_greet

templates:
  utter_greet:
  - text: "Hey! How are you?"
```
the log should be similar to this:

```
2019-10-23 12:11:26 WARNING  rasa.importers.rasa  - Loading domain from 'domain.yml' failed. Using empty domain. Error: 'Duplicate actions in domain. These actions occur more than once in the domain: 'utter_greet''
2019-10-23 12:11:26 WARNING  rasa.importers.rasa  - Loading domain from 'domain.yml' failed. Using empty domain. Error: 'Duplicate actions in domain. These actions occur more than once in the domain: 'utter_greet''
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'greet' on line 2. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'mood_great' on line 4. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'greet' on line 8. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'mood_unhappy' on line 10. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'affirm' on line 13. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'greet' on line 17. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'mood_unhappy' on line 19. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'deny' on line 22. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'goodbye' on line 26. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:11:26 WARNING  rasa.core.training.dsl  - Found unknown intent 'bot_challenge' on line 30. Please, make sure that all intents are listed in your domain yaml.
```
 and not this:

```
2019-10-23 12:12:04 WARNING  rasa.importers.rasa  - Loading domain from 'domain.yml' failed. Using empty domain. Error: 'Duplicate actions in domain. These actions occur more than once in the domain: 'utter_greet''
2019-10-23 12:12:04 WARNING  rasa.importers.rasa  - Loading domain from 'domain.yml' failed. Using empty domain. Error: 'Duplicate actions in domain. These actions occur more than once in the domain: 'utter_greet''
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'greet' on line 2. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'mood_great' on line 4. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'greet' on line 8. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'mood_unhappy' on line 10. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'affirm' on line 13. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'greet' on line 17. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'mood_unhappy' on line 19. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'deny' on line 22. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'goodbye' on line 26. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 WARNING  rasa.core.training.dsl  - Found unknown intent 'bot_challenge' on line 30. Please, make sure that all intents are listed in your domain yaml.
2019-10-23 12:12:04 INFO     rasa.core.validator  - Validating intents...
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'goodbye' is in the NLU training data, but is not listed in the domain.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'affirm' is in the NLU training data, but is not listed in the domain.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'mood_great' is in the NLU training data, but is not listed in the domain.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'bot_challenge' is in the NLU training data, but is not listed in the domain.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'deny' is in the NLU training data, but is not listed in the domain.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'greet' is in the NLU training data, but is not listed in the domain.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'mood_unhappy' is in the NLU training data, but is not listed in the domain.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'goodbye' is used in stories, but is not listed in the domain file.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'affirm' is used in stories, but is not listed in the domain file.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'mood_great' is used in stories, but is not listed in the domain file.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'bot_challenge' is used in stories, but is not listed in the domain file.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'deny' is used in stories, but is not listed in the domain file.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'greet' is used in stories, but is not listed in the domain file.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The intent 'mood_unhappy' is used in stories, but is not listed in the domain file.
2019-10-23 12:12:04 INFO     rasa.core.validator  - Validating there is no duplications...
2019-10-23 12:12:04 INFO     rasa.core.validator  - Validating utterances...
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The utterance 'utter_greet' is used in stories, but is not a valid utterance.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The utterance 'utter_happy' is used in stories, but is not a valid utterance.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The utterance 'utter_cheer_up' is used in stories, but is not a valid utterance.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The utterance 'utter_did_that_help' is used in stories, but is not a valid utterance.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The utterance 'utter_goodbye' is used in stories, but is not a valid utterance.
2019-10-23 12:12:04 ERROR    rasa.core.validator  - The utterance 'utter_iamabot' is used in stories, but is not a valid utterance.
```



**Status (please check what you already did)**:
- [x] made PR ready for code review
- [x] added some tests for the functionality
- [x] updated the documentation
- [x] updated the changelog
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa_nlu#code-style) for instructions)


Problem statement:
rasa data validate should fail early if domain cannot be read
**Description of Problem**:
Currently, if there is a syntax error in your domain, `rasa data validate` will load an empty domain and return a whole bunch of errors. Not super user friendly because you have to scroll through a bunch of errors to get to the top and see that yaml parsing failed. 

**Overview of the Solution**:
It would be nice if this command exited early if the domain can't be loaded (invalid yaml, duplicate actions, etc)

**Definition of Done**:
- [ ] Tests are added
- [ ] Feature mentioned in the changlog

Hints from issue discussion:
I would like to work on this. Are you already assigned to this, @amn41?
certainly not! would be awesome if you could work on it :)

Commit message: Merge branch 'master' into exit_early_if_domain_empty

Patch to review:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 99e93a1546a4..a2014fd940ab 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -13,6 +13,7 @@ This project adheres to `Semantic Versioning`_ starting with version 1.0.
 
 Added
 -----
+- Added data validator that checks if domain object returned is empty. If so, exit early from the command ``rasa data validate``
 - Added the KeywordIntentClassifier
 - Fall back to ``InMemoryTrackerStore`` in case there is any problem with the current
   tracker store
diff --git a/docs/user-guide/validate-files.rst b/docs/user-guide/validate-files.rst
index 8c6497259f67..5e4eff39f1bf 100644
--- a/docs/user-guide/validate-files.rst
+++ b/docs/user-guide/validate-files.rst
@@ -41,6 +41,10 @@ actions.
 
 **verify_all():** Runs all verifications above.
 
+**verify_domain_is_empty():** Check if domain returned by the importer is empty.
+
+**verify_domain():** Check if domain is valid.
+
 To use these functions it is necessary to create a `Validator` object and initialize the logger. See the following code:
 
 .. code-block:: python
diff --git a/rasa/cli/data.py b/rasa/cli/data.py
index 0f8009e035fe..8d4c9e000394 100644
--- a/rasa/cli/data.py
+++ b/rasa/cli/data.py
@@ -103,5 +103,9 @@ def validate_files(args):
     )
 
     validator = loop.run_until_complete(Validator.from_importer(file_importer))
+    domain_is_valid = validator.verify_domain_validity()
+    if not domain_is_valid:
+        sys.exit(1)
+
     everything_is_alright = validator.verify_all(not args.fail_on_warnings)
     sys.exit(0) if everything_is_alright else sys.exit(1)
diff --git a/rasa/core/validator.py b/rasa/core/validator.py
index 01e929e618cb..6abaa92876de 100644
--- a/rasa/core/validator.py
+++ b/rasa/core/validator.py
@@ -199,3 +199,8 @@ def verify_all(self, ignore_warnings: bool = True) -> bool:
         logger.info("Validating utterances...")
         stories_are_valid = self.verify_utterances_in_stories(ignore_warnings)
         return intents_are_valid and stories_are_valid and there_is_no_duplication
+
+    def verify_domain_validity(self) -> bool:
+        """ Runs the validations on domain."""
+
+        return not self.domain.is_empty()
diff --git a/tests/cli/test_rasa_data.py b/tests/cli/test_rasa_data.py
index 9ebb2a5d3eac..58089c7fd8ab 100644
--- a/tests/cli/test_rasa_data.py
+++ b/tests/cli/test_rasa_data.py
@@ -1,6 +1,9 @@
 import os
+import pytest
+from collections import namedtuple
 from typing import Callable
 from _pytest.pytester import RunResult
+from rasa.cli.data import validate_files
 
 
 def test_data_split_nlu(run_in_default_project: Callable[..., RunResult]):
@@ -63,3 +66,12 @@ def test_data_validate_help(run: Callable[..., RunResult]):
 
     for i, line in enumerate(lines):
         assert output.outlines[i] == line
+
+
+def test_validate_files_exit_early():
+    with pytest.raises(SystemExit) as pytest_e:
+        args = {"domain": "data/test_domains/duplicate_intents.yml", "data": None}
+        validate_files(namedtuple("Args", args.keys())(*args.values()))
+
+    assert pytest_e.type == SystemExit
+    assert pytest_e.value.code == 1
diff --git a/tests/core/test_validator.py b/tests/core/test_validator.py
index 07cb951f073b..43fb0192c642 100644
--- a/tests/core/test_validator.py
+++ b/tests/core/test_validator.py
@@ -1,4 +1,5 @@
 import pytest
+import os
 from rasa.core.validator import Validator
 from rasa.importers.rasa import RasaFileImporter
 from tests.core.conftest import (
@@ -88,6 +89,29 @@ async def test_verify_logging_message_for_repetition_in_intents(caplog):
     )
 
 
+async def test_verify_domain_validity(caplog):
+    domain_path = "data/test_domains/duplicate_intents.yml"
+    full_path = os.path.abspath(domain_path)
+
+    importer = RasaFileImporter(domain_path=domain_path)
+    validator = await Validator.from_importer(importer)
+    validator.verify_domain_validity()
+
+    log_object = caplog.records[-1]
+    message = log_object.message
+    level = log_object.levelname
+
+    assert "WARNING" == level
+    assert (
+        "Loading domain from '{}' failed. Using empty domain. "
+        "Error: 'Intents are not unique! Found two intents with name "
+        "'default'. Either rename or remove one of them.'".format(
+            domain_path, full_path
+        )
+        == message
+    )
+
+
 async def test_verify_there_is_not_example_repetition_in_intents():
     importer = RasaFileImporter(
         domain_path="examples/moodbot/domain.yml",


Code review comments:
diff_hunk: @@ -63,3 +66,12 @@ def test_data_validate_help(run: Callable[..., RunResult]):
 
     for i, line in enumerate(lines):
         assert output.outlines[i] == line
+
+
+def test_validate_files_exit_early():
+    with pytest.raises(SystemExit) as pytest_e:
+        args = {"domain": "data/test_domains/duplicate_intents.yml", "data": None}
+        validate_files(namedtuple("Args", args.keys())(*args.values()))
path: tests/cli/test_rasa_data.py
text: @user1:
```suggestion
        data.validate_files(namedtuple("Args", args.keys())(*args.values()))
```

diff_hunk: @@ -1,6 +1,9 @@
 import os
+import pytest
+from collections import namedtuple
 from typing import Callable
 from _pytest.pytester import RunResult
+from rasa.cli.data import validate_files
path: tests/cli/test_rasa_data.py
text: @user1:
```suggestion
from rasa.cli import data
```
We try to import modules rather than methods as much as possible, to make it clear where methods are coming from

diff_hunk: @@ -88,6 +89,29 @@
     )
 
 
+async def test_verify_domain_validity(caplog):
+    domain_path = "data/test_domains/duplicate_intents.yml"
+    full_path = os.path.abspath(domain_path)
+
+    importer = RasaFileImporter(domain_path=domain_path)
+    validator = await Validator.from_importer(importer)
+    validator.verify_domain_validity()
+
+    log_object = caplog.records[-1]
+    message = log_object.message
+    level = log_object.levelname
+
+    assert "WARNING" == level
+    assert (
+        "Loading domain from '{}' failed. Using empty domain. "
+        "Error: 'Intents are not unique! Found two intents with name "
+        "'default'. Either rename or remove one of them.'".format(
+            domain_path, full_path
path: tests/core/test_validator.py
text: @user1:
Looks like there is only one argument being passed to this `.format()`. Also we're moving to f-strings, that just hasn't been merged to master yet, would you use one here as well?

@author:
> Functionality looks great  Just some style comments and a revision to the docs after what we changed last time and we're good.
> 
> P.S. sorry to have you waiting and re-merging master, I wasn't aware you were ready (didn't get a notification), in the future feel free to just drop a comment when all the mentioned changes have been integrated and it's ready for another look!

Ok, no problem : )

diff_hunk: @@ -41,6 +41,10 @@ actions.
 
 **verify_all():** Runs all verifications above.
 
+**verify_domain_is_empty():** Check if domain returned by the importer is empty.
+
path: docs/user-guide/validate-files.rst
text: @user1:
Remove 44 and 45 after your we updated this