Repository: momegas/megabots (language: python)

Issue:
Integrate with langchain-serve?
**Is your feature request related to a problem? Please describe.**
Great work with megabots. Looks like a nice & easy API to develop LLM-based bots. What plans do you have to deploy the bots without adding infrastructure headache?

**Describe the solution you'd like**
I'd recommend integrating with [langchain-serve](https://github.com/jina-ai/langchain-serve). 

- Exposes APIs from function definitions locally as well as on the cloud.
- Very few lines of code changes, ease of development remains the same as local.
- Supports both REST & Websocket endpoints
- Serverless/autoscaling endpoints with automatic tls certs.
- Real-time streaming, human-in-the-loop support


Discussion / hints:
This is the plan indeed. To create a super easy way to deploy an API. I was thinking FastAPI. There is already a function `create_api`. 

The module you propose seems to be Jina specific. This is why I went with FastAPI. I was thinking to also create a `fastapi_langchain` package to do exactly that. But ill need some more time 😅

I'll have a better look on the `langchain_serve` package, as well.
@momegas We (langchain-serve) use FastAPI to create the endpoints for both rest & websockets, and locally deploy them with uvicorn (like one would do) :slightly_smiling_face:.

Jina fills the gap from going from local to cloud with very few lines of code changes and with no infrastructure headache - and gets you the scalability.
Oh! That is great. Maybe then indeed it's what I need and can save some development time. Are you from the langchain-serve team? If so it would be nice to join our discord and discuss further integration 😁
Absolutely! ~~Would you be able to [DM ](https://twitter.com/_deepankarm_) the discord link?~~ I'm there :) 
+1 for megabots integration into langchain-serve :) 

Fix patch:
diff --git a/README.md b/README.md
index d9ace18..4985cfe 100644
--- a/README.md
+++ b/README.md
@@ -28,7 +28,7 @@
 - 🏋️ mega-easy LLM training
 - 🚀 mega-easy deployment
 
-🤖 Megabots is backed by some of the most famous tools for productionalising AI. It uses [LangChain](https://docs.langchain.com/docs/) for managing LLM chains, [FastAPI](https://fastapi.tiangolo.com/) to create a production ready API, [Gradio](https://gradio.app/) to create a UI. At the moment it uses [OpenAI](https://openai.com/) to generate answers, but we plan to support other LLMs in the future.
+🤖 Megabots is backed by some of the most famous tools for productionalising AI. It uses [LangChain](https://docs.langchain.com/docs/) for managing LLM chains, [langchain-serve](https://github.com/jina-ai/langchain-serve) to create a production ready API, [Gradio](https://gradio.app/) to create a UI. At the moment it uses [OpenAI](https://openai.com/) to generate answers, but we plan to support other LLMs in the future.
 
 ## Getting started
 
@@ -169,18 +169,45 @@ milvus = vectorstore("milvus", host="localhost", port=19530)
 bot = bot("qna-over-docs", index="./examples/files/", vectorstore=milvus)
 ```
 
-## Exposing an API with FastAPI
+## Exposing an API with [langchain-serve](https://github.com/jina-ai/langchain-serve)
 
-You can also create a FastAPI app that will expose the bot as an API using the create_app function.
-Assuming you file is called `main.py` run `uvicorn main:app --reload` to run the API locally.
-You should then be able to visit `http://localhost:8000/docs` to see the API documentation.
+You can also expose the bot endpoints locally using langchain-serve. A sample file `api.py` is provided in the `megabots` folder.
 
-```python
-from megabots import bot, create_api
+To expose the API locally, you can do 
+```bash
+lc-serve deploy local megabots.api
+```
 
-app = create_app(bot("qna-over-docs"))
+You should then be able to visit `http://localhost:8000/docs` to see & interact with the API documentation.
+
+To deploy your API to the cloud, you can do and connect to the API using the endpoint provided in the output.
+```bash
+lc-serve deploy jcloud megabots.api
 ```
 
+<details>
+<summary>Show command output</summary>
+
+```text
+╭──────────────┬──────────────────────────────────────────────────────────────────────────────────────╮
+│ App ID       │                                 langchain-dec14439a6                                 │
+├──────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
+│ Phase        │                                       Serving                                        │
+├──────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
+│ Endpoint     │                      https://langchain-dec14439a6.wolf.jina.ai                       │
+├──────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
+│ App logs     │                               dashboards.wolf.jina.ai                                │
+├──────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
+│ Swagger UI   │                    https://langchain-dec14439a6.wolf.jina.ai/docs                    │
+├──────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
+│ OpenAPI JSON │                https://langchain-dec14439a6.wolf.jina.ai/openapi.json                │
+╰──────────────┴──────────────────────────────────────────────────────────────────────────────────────╯
+```
+</details>
+
+
+You can read more about langchain-serve [here](https://github.com/jina-ai/langchain-server).
+
 ## Exposing a Gradio chat-like interface
 
 You can expose a gradio UI for the bot using `create_interface` function.
diff --git a/megabots/api.py b/megabots/api.py
new file mode 100644
index 0000000..5da94b0
--- /dev/null
+++ b/megabots/api.py
@@ -0,0 +1,12 @@
+import os
+from megabots import bot
+from lcserve import serving
+
+cur_dir = os.path.dirname(os.path.abspath(__file__))
+index_dir = os.path.join(cur_dir, "..", "examples", "files")
+
+
+@serving
+def ask(question: str) -> str:
+    mybot = bot("qna-over-docs", index=index_dir)
+    return mybot.ask(question)
diff --git a/requirements.txt b/requirements.txt
index a3e4cd2..3ac5fb7 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,150 +1,10 @@
-aiofiles==23.1.0
-aiohttp==3.8.4
-aiosignal==1.3.1
-altair==4.2.2
-anyio==3.6.2
-appnope==0.1.3
-argilla==1.6.0
-asttokens==2.2.1
-async-timeout==4.0.2
-attrs==22.2.0
-backcall==0.2.0
-backoff==2.2.1
-black==23.3.0
-bleach==6.0.0
-certifi==2022.12.7
-cffi==1.15.1
-charset-normalizer==3.1.0
-click==8.1.3
-comm==0.1.3
-commonmark==0.9.1
-contourpy==1.0.7
-cryptography==40.0.2
-cycler==0.11.0
-dataclasses-json==0.5.7
-debugpy==1.6.7
-decorator==5.1.1
-Deprecated==1.2.13
-docutils==0.19
-entrypoints==0.4
-et-xmlfile==1.1.0
-exceptiongroup==1.1.1
-executing==1.2.0
 faiss-cpu==1.7.3
 fastapi==0.95.1
-ffmpy==0.3.0
-filelock==3.11.0
-fonttools==4.39.3
-frozenlist==1.3.3
-fsspec==2023.4.0
 gradio==3.26.0
 gradio_client==0.1.2
-h11==0.14.0
-httpcore==0.16.3
 httpx==0.23.3
-huggingface-hub==0.13.4
-idna==3.4
-importlib-metadata==6.3.0
-iniconfig==2.0.0
-ipykernel==6.22.0
-ipython==8.12.0
-jaraco.classes==3.2.3
-jedi==0.18.2
-Jinja2==3.1.2
-joblib==1.2.0
-jsonschema==4.17.3
-jupyter_client==8.1.0
-jupyter_core==5.3.0
-keyring==23.13.1
-kiwisolver==1.4.4
 langchain==0.0.137
-linkify-it-py==2.0.0
-lxml==4.9.2
-Markdown==3.4.3
-markdown-it-py==2.2.0
-MarkupSafe==2.1.2
-marshmallow==3.19.0
-marshmallow-enum==1.5.1
-matplotlib==3.7.1
-matplotlib-inline==0.1.6
-mdit-py-plugins==0.3.3
-mdurl==0.1.2
-monotonic==1.6
-more-itertools==9.1.0
-msg-parser==1.2.0
-multidict==6.0.4
-mypy-extensions==1.0.0
-nest-asyncio==1.5.6
-nltk==3.8.1
-numpy==1.23.5
-olefile==0.46
+langchain-serve>=0.0.16
 openai==0.27.4
-openapi-schema-pydantic==1.2.4
-openpyxl==3.1.2
-orjson==3.8.10
-packaging==23.0
-pandas==1.5.3
-parso==0.8.3
-pathspec==0.11.1
-pdfminer.six==20221105
-pexpect==4.8.0
-pickleshare==0.7.5
-Pillow==9.5.0
-pkginfo==1.9.6
-platformdirs==3.2.0
-pluggy==1.0.0
-prompt-toolkit==3.0.38
-psutil==5.9.4
-ptyprocess==0.7.0
-pure-eval==0.2.2
-pycparser==2.21
-pycryptodome==3.17
-pydantic==1.10.7
-pydub==0.25.1
-Pygments==2.15.0
-pypandoc==1.11
-pyparsing==3.0.9
-pyrsistent==0.19.3
 pytest==7.3.0
-python-dateutil==2.8.2
-python-docx==0.8.11
-python-dotenv==1.0.0
-python-magic==0.4.27
-python-multipart==0.0.6
-python-pptx==0.6.21
-pytz==2023.3
-PyYAML==6.0
-pyzmq==25.0.2
-readme-renderer==37.3
-regex==2023.3.23
-requests==2.28.2
-requests-toolbelt==0.10.1
-rfc3986==1.5.0
-rich==13.0.1
-semantic-version==2.10.0
-six==1.16.0
-sniffio==1.3.0
-SQLAlchemy==1.4.47
-stack-data==0.6.2
-starlette==0.26.1
-tenacity==8.2.2
-tiktoken==0.3.3
-tomli==2.0.1
-toolz==0.12.0
-tornado==6.2
-tqdm==4.65.0
-traitlets==5.9.0
-twine==4.0.2
-typing-inspect==0.8.0
-typing_extensions==4.5.0
-uc-micro-py==1.0.1
-unstructured==0.5.11
-urllib3==1.26.15
-uvicorn==0.21.1
-wcwidth==0.2.6
-webencodings==0.5.1
-websockets==11.0.1
-wrapt==1.14.1
-XlsxWriter==3.0.9
-yarl==1.8.2
-zipp==3.15.0
+rich
diff --git a/setup.py b/setup.py
index d89fe9a..8d75e4e 100644
--- a/setup.py
+++ b/setup.py
@@ -16,6 +16,7 @@
         "gradio",
         "python-dotenv",
         "openai",
+        "langchain-serve",
     ],
     author="Megaklis Vasilakis",
     author_email="megaklis.vasilakis@gmail.com",


Test patch:
diff --git a/tests/test_api.py b/tests/test_api.py
index d9a6d36..405fe56 100644
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -1,21 +1,89 @@
-import json
-from fastapi.testclient import TestClient
-from megabots import bot, create_api
+def _ignore_warnings():
+    import logging
+    import warnings
 
-qnabot = bot("qna-over-docs", index="./examples/files")
-app = create_api(qnabot)
+    logging.captureWarnings(True)
+    warnings.filterwarnings(
+        "ignore",
+        category=DeprecationWarning,
+        message="Deprecated call to `pkg_resources.declare_namespace('google')`.",
+    )
 
-client = TestClient(app)
 
+_ignore_warnings()
 
-def test_successful_response():
-    response = client.get("/v1/ask/What is your name?")
-    assert response.status_code == 200
-    assert "answer" in response.json()
-    assert isinstance(response.json()["answer"], str)
+import os
+import signal
+import subprocess
 
+import requests
+from requests.adapters import HTTPAdapter, Retry
 
-def test_missing_question_parameter():
-    response = client.get("/v1/ask/")
-    assert response.status_code == 404
-    assert response.json() == {"detail": "Not Found"}
+megabot_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _session_with_retry() -> requests.Session:
+    s = requests.Session()
+    retries = Retry(
+        total=50, backoff_factor=1, status_forcelist=[404, 500, 502, 503, 504]
+    )
+    s.mount('http://', HTTPAdapter(max_retries=retries))
+    return s
+
+
+class LCServeLocally:
+    def __init__(self, port: int = 8000):
+        self.port = port
+        self.command = ' '.join(
+            [
+                "lc-serve",
+                "deploy",
+                "local",
+                "megabots.api",
+                "--port",
+                str(self.port),
+            ]
+        )
+
+    def __enter__(self):
+        self.p = subprocess.Popen(
+            self.command, cwd=megabot_dir, shell=True, preexec_fn=os.setsid
+        )
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.p.terminate()
+        os.killpg(os.getpgid(self.p.pid), signal.SIGTERM)
+
+
+def test_lcserve_successful():
+    port = 8000
+    lcserve_host = f'http://localhost:{port}'
+
+    with LCServeLocally(port=port):
+        resp = _session_with_retry().post(
+            url=f'{lcserve_host}/ask',
+            json={"question": "What is your name?"},
+        )
+        assert resp.status_code == 200
+        assert "result" in resp.json()
+        assert isinstance(resp.json()["result"], str)
+
+
+def test_lcserve_invalid_request():
+    port = 8000
+    lcserve_host = f'http://localhost:{port}'
+
+    with LCServeLocally(port=port):
+        resp = _session_with_retry().post(
+            url=f'{lcserve_host}/ask',
+            json={"foo": "bar"},
+        )
+        assert resp.status_code == 422
+        assert "detail" in resp.json()
+        assert resp.json()["detail"] == [
+            {
+                "loc": ["body", "question"],
+                "msg": "field required",
+                "type": "value_error.missing",
+            }
+        ]