Repository: gradio-app/gradio
Pull request: Add support for async functions and async generators to `gr.ChatInterface`
Added async support for all of the routes: the submit function, the example function, and the API functions. Test with any async function or generator, e.g.: 

```py
async def slow_echo(message, history):
    for i in range(len(message)):
        time.sleep(0.05)
        yield "You typed: " + message[: i+1]

demo = gr.ChatInterface(slow_echo, examples=["abcdef"], cache_examples=True).queue()

demo.launch()
```

Closes: #5115 

Problem statement:
Add support for async fn to `ChatInterface` 
### Describe the bug

I tried to mount the async function to ChatInterface and got the error. Please take a look at the logs included below.
If you have suggestions on how I can implement the Chat Interface with the async function, I'd like to read them.

### Have you searched existing issues?  🔎

- [X] I have searched and found no existing issues

### Reproduction

I used the FastApi interface to mount the application
```
async def gradio_app_for_bi(message: str, history: list[list[str]]) -> str:
    ....

chatbot = gr.ChatInterface(fn=gradio_app_for_bi, title="Business Demo")
app = gr.mount_gradio_app(app, chatbot, path="/gradio")
```

### Screenshot

_No response_

### Logs

```shell
ValueError: Invalid message for Chatbot component: <coroutine object gradio_app_for_bi at 0x16ed689e0>
```


### System Info

```shell
Gradio Environment Information:
Operating System:  Darwin


gradio version:  3.39.0

gradio Dependencies:
  aiofiles: 23.1.0
  aiohttp: 3.8.4
  altair: 5.0.1
  fastapi: 0.100.1
  ffmpy: 0.3.0
  gradio-client: 0.3.0
  httpx: 0.24.1
  huggingface-hub: 0.16.4
  jinja2: 3.1.2
  markdown-it-py: 2.2.0
  markupsafe: 2.1.3
  matplotlib: 3.7.2
  mdit-py-plugins: 0.3.3
  numpy: 1.25.1
  orjson: 3.9.2
  packaging: 23.1
  pandas: 2.0.3
  pillow: 10.0.0
  pydantic: 1.10.11
  pydub: 0.25.1
  python-multipart: 0.0.6
  pyyaml: 6.0
  requests: 2.31.0
  semantic-version: 2.10.0
  typing-extensions: 4.5.0
  uvicorn: 0.22.0
  websockets: 11.0.3


gradio_client version:  0.3.0

gradio_client Dependencies:
  fsspec: 2023.6.0
  httpx: 0.24.1
  huggingface-hub: 0.16.4
  packaging: 23.1
  requests: 2.31.0
  typing-extensions: 4.5.0
  websockets: 11.0.3
```


### Severity

Blocking usage of gradio

Hints from issue discussion:
Hi @ZeusFSX not at the moment, that's on my list of todos. Will take a look at this soon.

Commit message: chat interface

Patch to review:
diff --git a/.changeset/moody-buttons-tell.md b/.changeset/moody-buttons-tell.md
new file mode 100644
index 0000000000..edaafc405b
--- /dev/null
+++ b/.changeset/moody-buttons-tell.md
@@ -0,0 +1,5 @@
+---
+"gradio": minor
+---
+
+feat:Add support for async functions and async generators to `gr.ChatInterface`
diff --git a/gradio/chat_interface.py b/gradio/chat_interface.py
index aaee1c820e..fe1b826201 100644
--- a/gradio/chat_interface.py
+++ b/gradio/chat_interface.py
@@ -6,7 +6,7 @@
 from __future__ import annotations
 
 import inspect
-from typing import Callable, Generator
+from typing import AsyncGenerator, Callable
 
 from gradio_client import utils as client_utils
 from gradio_client.documentation import document, set_documentation_group
@@ -25,6 +25,7 @@
 from gradio.helpers import create_examples as Examples  # noqa: N812
 from gradio.layouts import Accordion, Column, Group, Row
 from gradio.themes import ThemeClass as Theme
+from gradio.utils import SyncToAsyncIterator, async_iteration
 
 set_documentation_group("chatinterface")
 
@@ -100,7 +101,12 @@ def __init__(
             theme=theme,
         )
         self.fn = fn
-        self.is_generator = inspect.isgeneratorfunction(self.fn)
+        self.is_async = inspect.iscoroutinefunction(
+            self.fn
+        ) or inspect.isasyncgenfunction(self.fn)
+        self.is_generator = inspect.isgeneratorfunction(
+            self.fn
+        ) or inspect.isasyncgenfunction(self.fn)
         self.examples = examples
         if self.space_id and cache_examples is None:
             self.cache_examples = True
@@ -397,7 +403,7 @@ def _display_input(
         history.append([message, None])
         return history, history
 
-    def _submit_fn(
+    async def _submit_fn(
         self,
         message: str,
         history_with_input: list[list[str | None]],
@@ -405,59 +411,78 @@ def _submit_fn(
         **kwargs,
     ) -> tuple[list[list[str | None]], list[list[str | None]]]:
         history = history_with_input[:-1]
-        response = self.fn(message, history, *args, **kwargs)
+        if self.is_async:
+            response = await self.fn(message, history, *args, **kwargs)
+        else:
+            response = self.fn(message, history, *args, **kwargs)
         history.append([message, response])
         return history, history
 
-    def _stream_fn(
+    async def _stream_fn(
         self,
         message: str,
         history_with_input: list[list[str | None]],
         *args,
         **kwargs,
-    ) -> Generator[tuple[list[list[str | None]], list[list[str | None]]], None, None]:
+    ) -> AsyncGenerator:
         history = history_with_input[:-1]
         generator = self.fn(message, history, *args, **kwargs)
+        if not self.is_async:
+            generator = SyncToAsyncIterator(generator, self.limiter)
         try:
-            first_response = next(generator)
+            first_response = await async_iteration(generator)
             update = history + [[message, first_response]]
             yield update, update
         except StopIteration:
             update = history + [[message, None]]
             yield update, update
-        for response in generator:
+        async for response in generator:
             update = history + [[message, response]]
             yield update, update
 
-    def _api_submit_fn(
+    async def _api_submit_fn(
         self, message: str, history: list[list[str | None]], *args, **kwargs
     ) -> tuple[str, list[list[str | None]]]:
-        response = self.fn(message, history)
+        if self.is_async:
+            response = await self.fn(message, history, *args, **kwargs)
+        else:
+            response = self.fn(message, history, *args, **kwargs)
         history.append([message, response])
         return response, history
 
-    def _api_stream_fn(
+    async def _api_stream_fn(
         self, message: str, history: list[list[str | None]], *args, **kwargs
-    ) -> Generator[tuple[str | None, list[list[str | None]]], None, None]:
+    ) -> AsyncGenerator:
         generator = self.fn(message, history, *args, **kwargs)
+        if not self.is_async:
+            generator = SyncToAsyncIterator(generator, self.limiter)
         try:
-            first_response = next(generator)
+            first_response = await async_iteration(generator)
             yield first_response, history + [[message, first_response]]
         except StopIteration:
             yield None, history + [[message, None]]
-        for response in generator:
+        async for response in generator:
             yield response, history + [[message, response]]
 
-    def _examples_fn(self, message: str, *args, **kwargs) -> list[list[str | None]]:
-        return [[message, self.fn(message, [], *args, **kwargs)]]
+    async def _examples_fn(
+        self, message: str, *args, **kwargs
+    ) -> list[list[str | None]]:
+        if self.is_async:
+            response = await self.fn(message, [], *args, **kwargs)
+        else:
+            response = self.fn(message, [], *args, **kwargs)
+        return [[message, response]]
 
-    def _examples_stream_fn(
+    async def _examples_stream_fn(
         self,
         message: str,
         *args,
         **kwargs,
-    ) -> Generator[list[list[str | None]], None, None]:
-        for response in self.fn(message, [], *args, **kwargs):
+    ) -> AsyncGenerator:
+        generator = self.fn(message, [], *args, **kwargs)
+        if not self.is_async:
+            generator = SyncToAsyncIterator(generator, self.limiter)
+        async for response in generator:
             yield [[message, response]]
 
     def _delete_prev_fn(
diff --git a/scripts/format_backend.sh b/scripts/format_backend.sh
index 69d5a3f9d8..26feae059e 100755
--- a/scripts/format_backend.sh
+++ b/scripts/format_backend.sh
@@ -5,5 +5,6 @@ cd "$(dirname ${0})/.."
 echo "Formatting the backend... Our style follows the Black code style."
 ruff --fix gradio test
 black gradio test
+bash scripts/type_check_backend.sh
 
 bash client/python/scripts/format.sh  # Call the client library's formatting script
diff --git a/test/test_chat_interface.py b/test/test_chat_interface.py
index c15ddc3585..fe96adfc2c 100644
--- a/test/test_chat_interface.py
+++ b/test/test_chat_interface.py
@@ -15,11 +15,20 @@ def double(message, history):
     return message + " " + message
 
 
+async def async_greet(message, history):
+    return "hi, " + message
+
+
 def stream(message, history):
     for i in range(len(message)):
         yield message[: i + 1]
 
 
+async def async_stream(message, history):
+    for i in range(len(message)):
+        yield message[: i + 1]
+
+
 def count(message, history):
     return str(len(history))
 
@@ -68,7 +77,8 @@ def test_events_attached(self):
             )
 
     @pytest.mark.asyncio
-    async def test_example_caching(self):
+    async def test_example_caching(self, monkeypatch):
+        monkeypatch.setattr(helpers, "CACHED_FOLDER", tempfile.mkdtemp())
         chatbot = gr.ChatInterface(
             double, examples=["hello", "hi"], cache_examples=True
         )
@@ -77,6 +87,17 @@ async def test_example_caching(self):
         assert prediction_hello[0][0] == ["hello", "hello hello"]
         assert prediction_hi[0][0] == ["hi", "hi hi"]
 
+    @pytest.mark.asyncio
+    async def test_example_caching_async(self, monkeypatch):
+        monkeypatch.setattr(helpers, "CACHED_FOLDER", tempfile.mkdtemp())
+        chatbot = gr.ChatInterface(
+            async_greet, examples=["abubakar", "tom"], cache_examples=True
+        )
+        prediction_hello = await chatbot.examples_handler.load_from_cache(0)
+        prediction_hi = await chatbot.examples_handler.load_from_cache(1)
+        assert prediction_hello[0][0] == ["abubakar", "hi, abubakar"]
+        assert prediction_hi[0][0] == ["tom", "hi, tom"]
+
     @pytest.mark.asyncio
     async def test_example_caching_with_streaming(self, monkeypatch):
         monkeypatch.setattr(helpers, "CACHED_FOLDER", tempfile.mkdtemp())
@@ -88,6 +109,17 @@ async def test_example_caching_with_streaming(self, monkeypatch):
         assert prediction_hello[0][0] == ["hello", "hello"]
         assert prediction_hi[0][0] == ["hi", "hi"]
 
+    @pytest.mark.asyncio
+    async def test_example_caching_with_streaming_async(self, monkeypatch):
+        monkeypatch.setattr(helpers, "CACHED_FOLDER", tempfile.mkdtemp())
+        chatbot = gr.ChatInterface(
+            async_stream, examples=["hello", "hi"], cache_examples=True
+        )
+        prediction_hello = await chatbot.examples_handler.load_from_cache(0)
+        prediction_hi = await chatbot.examples_handler.load_from_cache(1)
+        assert prediction_hello[0][0] == ["hello", "hello"]
+        assert prediction_hi[0][0] == ["hi", "hi"]
+
     @pytest.mark.asyncio
     async def test_example_caching_with_additional_inputs(self, monkeypatch):
         monkeypatch.setattr(helpers, "CACHED_FOLDER", tempfile.mkdtemp())
@@ -138,12 +170,25 @@ def test_streaming_api(self, connect):
             wait([job])
             assert job.outputs() == ["h", "he", "hel", "hell", "hello"]
 
+    def test_streaming_api_async(self, connect):
+        chatbot = gr.ChatInterface(async_stream).queue()
+        with connect(chatbot) as client:
+            job = client.submit("hello")
+            wait([job])
+            assert job.outputs() == ["h", "he", "hel", "hell", "hello"]
+
     def test_non_streaming_api(self, connect):
         chatbot = gr.ChatInterface(double)
         with connect(chatbot) as client:
             result = client.predict("hello")
             assert result == "hello hello"
 
+    def test_non_streaming_api_async(self, connect):
+        chatbot = gr.ChatInterface(async_greet)
+        with connect(chatbot) as client:
+            result = client.predict("gradio")
+            assert result == "hi, gradio"
+
     def test_streaming_api_with_additional_inputs(self, connect):
         chatbot = gr.ChatInterface(
             echo_system_prompt_plus_message,


Code review comments:
diff_hunk: @@ -397,67 +403,86 @@ def _display_input(
         history.append([message, None])
         return history, history
 
-    def _submit_fn(
+    async def _submit_fn(
         self,
         message: str,
         history_with_input: list[list[str | None]],
         *args,
         **kwargs,
     ) -> tuple[list[list[str | None]], list[list[str | None]]]:
         history = history_with_input[:-1]
-        response = self.fn(message, history, *args, **kwargs)
+        if self.is_async:
+            response = await self.fn(message, history, *args, **kwargs)
+        else:
+            response = self.fn(message, history, *args, **kwargs)
path: gradio/chat_interface.py
text: @user1:
We need to run the non-coroutine in a separate thread otherwise we'll block the event loop. This is because the _submit_fn is now async.

If you run this code on your branch, two events can't run concurrently anymore

```python
import random
import gradio as gr
import time

def random_response(message, history):
    time.sleep(5)
    return random.choice(["Yes", "No"])

demo = gr.ChatInterface(random_response)

if __name__ == "__main__":
    demo.launch()
```

Note how the textbox submit event has to wait:
![chatbot_concurrency](https://github.com/gradio-app/gradio/assets/41651716/29103b89-2321-4b01-a5ec-91ff2436040e)

compare to main:
![chatbot_concurrency_main](https://github.com/gradio-app/gradio/assets/41651716/b7afcef0-5094-4d97-9d35-cc39729166ed)





@author:
Ah nice thanks @user1, will fix