Repository: jina-ai/serve
Pull request: test: add regression for chatbot
Given a system under test, `test_chatbot` depends on the `index` flow to be fired. While `f.block` blocked the process. I did some search and find the official plugin: [pytest-xprocess](https://github.com/pytest-dev/pytest-xprocess)

> pytest plugin for managing processes across test runs.

This plugin:

1. Bootstrap `hello_world` example in another process.
2. Monitor system logs to check if it is ready to test using `pattern matching`.
3. Run the test.
4. Kill the process.

This PR completes regression tests for three examples.

Problem statement:
Regression test for all three hello-world demos
**Describe the feature**

hello-world examples are important, if it breaks for a first-time user, then we lose this user.

The current tests on hello-world is not enough, it does not check (1) if index/search is successful (2) the sanity of retrieved results. 

**Your proposal**

Please add regression test for three hello-world.

---
<!-- Optional, but really help us locate the problem faster -->

**Environment**
<!-- Run `jina --version-full` and copy paste the output here -->

**Screenshots**
<!-- If applicable, add screenshots to help explain your problem. -->

Tasks: 
- [x] helloworld 
- [x] multimodal
- [x] chatbot
- [x] ci/cd pipeline

Hints from issue discussion:
Example for multimodal can be taken from reverted change in #2178. It was reverted for other reasons, but the test would still be valid
glad to work on it with other engineers.
Multimodal helloworld has already been added, please add the other integration tests in the same subfolders
One thing to be aware of.

Adding these tests, make the tests dependant on the `hub` submodule, which will bring to a situation where the tests in core do not pass because the hub needs an update and the hub will not be updatable because the new core version is not released
@bwanglzu can u make sure the helloworld examples tests are moved to their own folder independent of integration tests? Like this we can easily skip it when necessary much easier
@JoanFM I will

Commit message: test: remove updates on jinahub

Patch to review:
diff --git a/extra-requirements.txt b/extra-requirements.txt
index 9615adea042e1..649ae07f15812 100644
--- a/extra-requirements.txt
+++ b/extra-requirements.txt
@@ -59,6 +59,7 @@ pytest-mock:                test
 pytest-cov:                 test
 pytest-repeat:              test
 pytest-asyncio:             test
+pytest-xprocess             test
 flaky:                      test
 mock:                       test
 requests:                   http, devel, cicd, daemon
diff --git a/tests/system/chatbot/__init__.py b/tests/system/chatbot/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/tests/system/chatbot/test_chatbot.py b/tests/system/chatbot/test_chatbot.py
new file mode 100644
index 0000000000000..c242b87b40992
--- /dev/null
+++ b/tests/system/chatbot/test_chatbot.py
@@ -0,0 +1,49 @@
+from multiprocessing import Process
+
+import pytest
+import requests
+from xprocess import ProcessStarter
+
+from jina.helloworld.chatbot import hello_world
+from jina.parsers.helloworld import set_hw_chatbot_parser
+
+
+@pytest.fixture
+def chatbot_args(tmpdir):
+    return set_hw_chatbot_parser().parse_args(
+        ['--workdir', str(tmpdir), '--port-expose', '8080']
+    )
+
+
+@pytest.fixture
+def payload():
+    return {'top_k': 1, 'data': ['text:Is my dog safe from virus']}
+
+
+@pytest.fixture
+def post_uri():
+    return 'http://localhost:8080/api/search'
+
+
+@pytest.fixture
+def expected_result():
+    return '''There’s no evidence from the outbreak that eating garlic, sipping water every 15 minutes or taking vitamin C will protect people from the new coronavirus.'''
+
+
+@pytest.fixture(autouse=True)
+def start_server(xprocess, chatbot_args):
+    class Starter(ProcessStarter):
+        pattern = "You should see a demo page opened in your browser"
+        args = ["jina", "hello", "chatbot"]
+        max_read_lines = 10000
+
+    xprocess.ensure("server", Starter)
+    yield
+    xprocess.getinfo("server").terminate()
+
+
+def test_chatbot(payload, post_uri, expected_result):
+    """Regression test for chatbot example."""
+    resp = requests.post(post_uri, json=payload)
+    assert resp.status_code == 200
+    assert expected_result in resp.text


Code review comments:
diff_hunk: @@ -59,6 +59,7 @@ pytest-mock:                test
 pytest-cov:                 test
 pytest-repeat:              test
 pytest-asyncio:             test
+pytest-xprocess             test
path: extra-requirements.txt
text: @user1:
must be `pytest-xprocess:    test`. Colon is missing