Repository: RasaHQ/rasa
Pull request: Allow tests to be located anywhere, not just in `tests` directory
**Proposed changes**:
- fix #7763 

**Status (please check what you already did)**:
- [x] added some tests for the functionality
- [ ] updated the documentation
- [x] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/main/changelog) for instructions)
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
`rasa test` should search for test files recursively in all sub-directories, not just under `tests/`
**Rasa version**:
2.2.4
**Python version**:
3.7.2
**Operating system** (windows, osx, ...):
OSX

**Issue**:
`rasa test` looks recursively for files prefixed with `test_`, but only in subdirectories of `tests/`. It should look recursively in all subdirectories.
`rasa test core` seems to look for these files in any subdirectory.

**Command or request that led to error**:
```
rasa test
```

Hints from issue discussion:
@kalkbrennerei finally got around to making this. Could you add some comments if you have any? Thank you!

from our Slack discussion:
Lu:"I think rasa test and rasa test core are both using test.py::run_core_test internally but with different args"
thanks @koernerfelicia! I think I have nothing to add  😃
but maybe I can mention [this PR](https://github.com/RasaHQ/rasa/pull/7483) here. I think we decided to only accept test story files with `test_` prefix, so with the fix of your issue the linked PR will become irrelevant if I'm not mistaken
I'll close my PR as well. @kalkbrennerei Do you know any specific reasons for the decisions or where we could find that? Seems like it addresses a code problem by causing friction for users.
@wochinge I think for 2.0 we wanted to enable the users to structure their projects the way they want to, so without restricting them to have a/only one `tests` folder. And therefore the tests must have the `tests_` prefix. But it also doesn't seem ideal to me that this is causing friction to the users... This [Thread](https://rasa-hq.slack.com/archives/C7B08Q5FX/p1597403645425900) seems to have a discussion about it, but not sure of the actual conclusion.

Commit message: Rollback `test_only_getting_e2e_conversation_tests_if_e2e_enabled` change

Patch to review:
diff --git a/changelog/7763.improvement.md b/changelog/7763.improvement.md
new file mode 100644
index 000000000000..54906a7b2bfa
--- /dev/null
+++ b/changelog/7763.improvement.md
@@ -0,0 +1 @@
+Allow tests to be located anywhere, not just in `tests` directory.
\ No newline at end of file
diff --git a/rasa/shared/constants.py b/rasa/shared/constants.py
index 9fa50c4dcb66..1f065206d206 100644
--- a/rasa/shared/constants.py
+++ b/rasa/shared/constants.py
@@ -44,7 +44,7 @@
 
 DEFAULT_NLU_FALLBACK_INTENT_NAME = "nlu_fallback"
 
-DEFAULT_E2E_TESTS_PATH = "tests"
+DEFAULT_E2E_TESTS_PATH = "."
 TEST_STORIES_FILE_PREFIX = "test_"
 
 DEFAULT_LOG_LEVEL = "INFO"
diff --git a/rasa/shared/core/training_data/story_reader/markdown_story_reader.py b/rasa/shared/core/training_data/story_reader/markdown_story_reader.py
index 9d1b35b4c804..bec83a42a4e4 100644
--- a/rasa/shared/core/training_data/story_reader/markdown_story_reader.py
+++ b/rasa/shared/core/training_data/story_reader/markdown_story_reader.py
@@ -13,9 +13,9 @@
     INTENT_MESSAGE_PREFIX,
     DOCS_URL_DOMAINS,
     LEGACY_DOCS_BASE_URL,
-    DEFAULT_E2E_TESTS_PATH,
     DOCS_URL_STORIES,
     DOCS_URL_MIGRATION_GUIDE_MD_DEPRECATION,
+    TEST_STORIES_FILE_PREFIX,
 )
 from rasa.shared.core.events import UserUttered
 from rasa.shared.nlu.interpreter import RegexInterpreter
@@ -341,6 +341,20 @@ def is_stories_file(file_path: Union[Text, Path]) -> bool:
             )
             return False
 
+    @staticmethod
+    def _is_tests_file(file_path: Text) -> bool:
+        """Check if the filename of a file at a path is a tests file.
+
+        Arguments:
+            file_path: path to the file
+
+        Returns:
+            `True` if the filename indicates that it's a test file, `False` otherwise.
+        """
+        return Path(file_path).name.startswith(TEST_STORIES_FILE_PREFIX) or Path(
+            file_path
+        ).name.endswith("_tests.md")
+
     @staticmethod
     def is_test_stories_file(file_path: Union[Text, Path]) -> bool:
         """Checks if a file contains test stories.
@@ -354,9 +368,8 @@ def is_test_stories_file(file_path: Union[Text, Path]) -> bool:
         if not rasa.shared.data.is_likely_markdown_file(file_path):
             return False
 
-        dirname = os.path.dirname(file_path)
         return (
-            DEFAULT_E2E_TESTS_PATH in dirname
+            MarkdownStoryReader._is_tests_file(file_path)
             and rasa.shared.data.is_story_file(file_path)
             and not rasa.shared.data.is_nlu_file(file_path)
         )
diff --git a/tests/shared/importers/test_multi_project.py b/tests/shared/importers/test_multi_project.py
index eaa85837d80a..4bc700972b53 100644
--- a/tests/shared/importers/test_multi_project.py
+++ b/tests/shared/importers/test_multi_project.py
@@ -6,7 +6,6 @@
 
 import rasa.shared.utils.io
 from rasa.shared.constants import (
-    DEFAULT_E2E_TESTS_PATH,
     DEFAULT_DOMAIN_PATH,
     DEFAULT_CORE_SUBDIRECTORY_NAME,
 )
@@ -258,10 +257,7 @@ async def test_only_getting_e2e_conversation_tests_if_e2e_enabled(
         story_file,
     )
 
-    story_test_file = (
-        tmp_path / "bots" / "Bot A" / DEFAULT_E2E_TESTS_PATH / test_stories_filename
-    )
-    story_test_file.parent.mkdir(parents=True)
+    story_test_file = tmp_path / "bots" / "Bot A" / test_stories_filename
     rasa.shared.utils.io.write_text_file(test_story, story_test_file)
 
     selector = MultiProjectImporter(config_path)
@@ -290,10 +286,7 @@ def test_not_importing_e2e_conversation_tests_in_project(tmp_path: Path,):
     story_file.parent.mkdir(parents=True)
     rasa.shared.utils.io.write_text_file("""## story""", story_file)
 
-    story_test_file = (
-        tmp_path / "bots" / "Bot A" / DEFAULT_E2E_TESTS_PATH / "test_stories.yml"
-    )
-    story_test_file.parent.mkdir(parents=True)
+    story_test_file = tmp_path / "bots" / "Bot A" / "test_stories.yml"
     rasa.shared.utils.io.write_text_file("""stories:""", story_test_file)
 
     selector = MultiProjectImporter(config_path)
diff --git a/tests/shared/test_data.py b/tests/shared/test_data.py
index 725a7bf18fc7..58d328fbcbb7 100644
--- a/tests/shared/test_data.py
+++ b/tests/shared/test_data.py
@@ -8,7 +8,6 @@
 import rasa.shared
 
 import rasa.shared.data
-from rasa.shared.constants import DEFAULT_E2E_TESTS_PATH
 from rasa.shared.nlu.training_data.loading import load_data
 from rasa.shared.utils.io import write_text_file, json_to_string
 from tests.conftest import DEFAULT_NLU_DATA
@@ -82,7 +81,15 @@ def test_default_story_files_are_story_files():
 
 
 def test_default_conversation_tests_are_conversation_tests_yml(tmpdir: Path):
-    parent = tmpdir / DEFAULT_E2E_TESTS_PATH
+    e2e_path = tmpdir / "test_stories.yml"
+    e2e_story = """stories:"""
+    write_text_file(e2e_story, e2e_path)
+
+    assert rasa.shared.data.is_test_stories_file(str(e2e_path))
+
+
+def test_conversation_tests_in_a_directory(tmpdir: Path):
+    parent = tmpdir / "tests"
     Path(parent).mkdir(parents=True)
 
     e2e_path = parent / "test_stories.yml"
@@ -94,10 +101,7 @@ def test_default_conversation_tests_are_conversation_tests_yml(tmpdir: Path):
 
 def test_default_conversation_tests_are_conversation_tests_md(tmpdir: Path):
     # can be removed once conversation tests MD support is removed
-    parent = tmpdir / DEFAULT_E2E_TESTS_PATH
-    Path(parent).mkdir(parents=True)
-
-    e2e_path = parent / "conversation_tests.md"
+    e2e_path = tmpdir / "conversation_tests.md"
     e2e_story = """## my story test"""
     write_text_file(e2e_story, e2e_path)
 
@@ -105,10 +109,7 @@ def test_default_conversation_tests_are_conversation_tests_md(tmpdir: Path):
 
 
 def test_nlu_data_files_are_not_conversation_tests(tmpdir: Path):
-    parent = tmpdir / DEFAULT_E2E_TESTS_PATH
-    Path(parent).mkdir(parents=True)
-
-    nlu_path = parent / "nlu.md"
+    nlu_path = tmpdir / "nlu.md"
     nlu_data = """
 ## intent: greet
 - hello
@@ -121,11 +122,7 @@ def test_nlu_data_files_are_not_conversation_tests(tmpdir: Path):
 
 
 def test_domain_files_are_not_conversation_tests(tmpdir: Path):
-    parent = tmpdir / DEFAULT_E2E_TESTS_PATH
-    Path(parent).mkdir(parents=True)
-
-    domain_path = parent / "domain.yml"
-
+    domain_path = tmpdir / "domain.yml"
     assert not rasa.shared.data.is_test_stories_file(str(domain_path))
 
 


Code review comments:
diff_hunk: @@ -341,6 +341,20 @@ def is_stories_file(file_path: Union[Text, Path]) -> bool:
             )
             return False
 
+    @staticmethod
+    def _is_tests_file(file_path: Text) -> bool:
+        """Check if the filename of a file at a path is a tests file.
+
+        Arguments:
+            file_path: path to the file
+
+        Returns:
+            `True` if the filename indicates that it's a test file, `False` otherwise.
+        """
+        return Path(file_path).name.startswith(TEST_STORIES_FILE_PREFIX) or Path(
+            file_path
+        ).name.endswith("_tests.md")
path: rasa/shared/core/training_data/story_reader/markdown_story_reader.py
text: @user1:
mhm, I'd keep the old logic here (check if `tests` is in dirname`).

diff_hunk: @@ -82,7 +81,15 @@ def test_default_story_files_are_story_files():
 
 
 def test_default_conversation_tests_are_conversation_tests_yml(tmpdir: Path):
-    parent = tmpdir / DEFAULT_E2E_TESTS_PATH
+    e2e_path = tmpdir / "test_stories.yml"
+    e2e_story = """stories:"""
+    write_text_file(e2e_story, e2e_path)
+
+    assert rasa.shared.data.is_test_stories_file(str(e2e_path))
+
+
+def test_conversation_tests_in_a_directory(tmpdir: Path):
path: tests/shared/test_data.py
text: @user1:
I'd use `tmp_path`. `tmpdir` is a different type and not of type `Path`
```suggestion
def test_conversation_tests_in_a_directory(tmp_path: Path):
```