Repository: gradio-app/gradio

Issue:
API call not stack with the queue from the website.
### Describe the bug

I have a website about 5k DAU ,and api call from another service through api doc , i can see from console these api from another service jump queue and process immediately.

Is this supposed behavior or a bug?how can i fix this ?


### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

call api when there is a queue in  website : https://ai-creator.net/aiart ,

I am hosting stable diffusion webui, i am not so sure how  can you debug online.

### Screenshot

_No response_

### Logs

```shell
...
```
![image](https://user-images.githubusercontent.com/1669515/196015611-02998e7e-ee55-451d-aec2-43282d10efb3.png)

you can see there are two jobs ,but i have only set concurrency_count=1 in my code.


### System Info

```shell
>>> import gradio as gr
>>> gr.__version__
'3.4b3'
>>>
```


### Severity

serious, but I can work around it


Discussion:
Hi @xiaol can you please provide a link your website and/or code to your demo so that we can try to reproduce this issue? 

And one question -- have you enabled the queue for your demo? This should not happen if you have enabled the queue
sorry for this absurd information , website is here  https://ai-creator.net/aiart , 

and this is the launch code:


```
  def run(self):
        loop = asyncio.new_event_loop()
        asyncio.set_event_loop(loop)
        gradio_params = {
            'inbrowser': opt.inbrowser,
            'server_name': '0.0.0.0',
            'server_port': opt.port,
            'share': opt.share,
            'show_error': True,
            'debug':True
        }
        if not opt.share:
            demo.queue(concurrency_count=1, client_position_to_load_data=30)
        if opt.share and opt.share_password:
            gradio_params['auth'] = ('webui', opt.share_password)
        # gradio_params['auth'] = same_auth

        # Check to see if Port 7860 is open
        port_status = 1
        while port_status != 0:
            try:
                self.demo.launch(**gradio_params)
            except (OSError) as e:
                print (f'Error: Port: {opt.port} is not open yet. Please wait, this may take upwards of 60 seconds...')
                time.sleep(10)
            else:
                port_status = 0

    def stop(self):
        self.demo.close() # this tends to hang

def launch_server():
    server_thread = ServerLauncher(demo)
    server_thread.start()

    try:
        while server_thread.is_alive():
            time.sleep(60)
    except (KeyboardInterrupt, OSError) as e:
        crash(e, 'Shutting down...')
```

this is the txt2img button code:
```
use_queue = True

                txt2img_btn.click(
                    txt2img_func,
                    txt2img_inputs,
                    txt2img_outputs,
                    api_name='txt2img',
                    queue=use_queue
                )
```


Thanks @xiaol, one more clarification -- when you say "api call from another service through api doc", do you mean they make a POST request to the `/api/predict` endpoint? Would you like to disable this behavior or would like to preserve this behavior but ensure users join the queue?
Yes, make a POST request to /api/predict endpoint , and this would slow the online users' queue, it gives a privilege to api reuquest. For some api it would be fine, but some should queue with the website users since everyone can make api request. Of course , it makes online service vulnerable, but i haven't firgure out how to deal with it.

Because  i wanna to charge and limit the api  ,maybe gives them privilege is fine for now.

But after all, i wanna to control the behavior to give different api or users privilege to jump the queue, if i monetize this product.

And from what i observed , users always join the queue, except you use a phone browser and switch the browser to backend ,would stuck your queue, maybe i should post another issue about this problem.

Thank you for your patience for my bad writing skill.
Hi @xiaol !

Yes it’s true that the backend api is open when the queue is enabled, allowing users who use the rest api directly to skip the ones in the queue. 

One thing that I think we can implement in Gradio is to block all requests to the /api/ end point by default if the queue for that particular route is enabled. We can add an ‘open_routes’ parameter to the queue method so that ‘queue(open_routes=True)’ means the route is not blocked when the queue is enabled (the current behavior).

That should let you control whether you want to prevent api users to skip the queue or not. 

I’m not sure if the “charging for api” usage feature you talk about should live in Gradio. But I think you can implement your own rate limiting functionality as a FastApi middleware https://fastapi.tiangolo.com/tutorial/middleware/
I see, a switcher of api is fine.Middleware is a good practice ,great to know ,i'll work around it.


Resolving patch:
diff --git a/gradio/blocks.py b/gradio/blocks.py
--- a/gradio/blocks.py
+++ b/gradio/blocks.py
@@ -481,6 +481,7 @@ def __init__(
         self.share_url = None
         self.width = None
         self.height = None
+        self.api_open = True
 
         self.ip_address = None
         self.is_space = True if os.getenv("SYSTEM") == "spaces" else False
@@ -1029,6 +1030,7 @@ def queue(
         status_update_rate: float | str = "auto",
         client_position_to_load_data: int = 30,
         default_enabled: bool = True,
+        api_open: bool = True,
         max_size: Optional[int] = None,
     ):
         """
@@ -1038,13 +1040,15 @@ def queue(
             status_update_rate: If "auto", Queue will send status estimations to all clients whenever a job is finished. Otherwise Queue will send status at regular intervals set by this parameter as the number of seconds.
             client_position_to_load_data: Once a client's position in Queue is less that this value, the Queue will collect the input data from the client. You may make this smaller if clients can send large volumes of data, such as video, since the queued data is stored in memory.
             default_enabled: If True, all event listeners will use queueing by default.
-            max_size: Maximum number of jobs that can be queued at once. Jobs beyond this limit simply return an error message to the user asking them to try again. If None, there is no limit.
+            api_open: If True, the REST routes of the backend will be open, allowing requests made directly to those endpoints to skip the queue.
+            max_size: The maximum number of events the queue will store at any given moment.
         Example:
             demo = gr.Interface(gr.Textbox(), gr.Image(), image_generator)
             demo.queue(concurrency_count=3)
             demo.launch()
         """
         self.enable_queue = default_enabled
+        self.api_open = api_open
         self._queue = queue.Queue(
             live_updates=status_update_rate == "auto",
             concurrency_count=concurrency_count,
@@ -1108,7 +1112,7 @@ def launch(
             ssl_certfile: If a path to a file is provided, will use this as the signed certificate for https. Needs to be provided if ssl_keyfile is provided.
             ssl_keyfile_password: If a password is provided, will use this with the ssl certificate for https.
             quiet: If True, suppresses most print statements.
-            show_api: If True, shows the api docs in the footer of the app. Default True.
+            show_api: If True, shows the api docs in the footer of the app. Default True. If the queue is enabled, then api_open parameter of .queue() will determine if the api docs are shown, independent of the value of show_api.
         Returns:
             app: FastAPI app object that is running the demo
             local_url: Locally accessible link to the demo
@@ -1135,7 +1139,6 @@ def reverse(text):
         self.height = height
         self.width = width
         self.favicon_path = favicon_path
-        self.show_api = show_api
         if enable_queue is not None:
             self.enable_queue = enable_queue
             warnings.warn(
@@ -1149,13 +1152,11 @@ def reverse(text):
             self.enable_queue = self.enable_queue is True
         if self.enable_queue and not hasattr(self, "_queue"):
             self.queue()
+        self.show_api = self.api_open if self.enable_queue else show_api
 
         for dep in self.dependencies:
             for i in dep["cancels"]:
-                queue_status = self.dependencies[i]["queue"]
-                if queue_status is False or (
-                    queue_status is None and not self.enable_queue
-                ):
+                if not self.queue_enabled_for_fn(i):
                     raise ValueError(
                         "In order to cancel an event, the queue for that event must be enabled! "
                         "You may get this error by either 1) passing a function that uses the yield keyword "
@@ -1424,3 +1425,8 @@ def startup_events(self):
         if self.enable_queue:
             utils.run_coro_in_background(self._queue.start)
         utils.run_coro_in_background(self.create_limiter)
+
+    def queue_enabled_for_fn(self, fn_index: int):
+        if self.dependencies[fn_index]["queue"] is None:
+            return self.enable_queue
+        return self.dependencies[fn_index]["queue"]
diff --git a/gradio/queue.py b/gradio/queue.py
--- a/gradio/queue.py
+++ b/gradio/queue.py
@@ -63,6 +63,7 @@ def __init__(
         self.sleep_when_free = 0.05
         self.max_size = max_size
         self.blocks_dependencies = blocks_dependencies
+        self.access_token = ""
 
     async def start(self):
         run_coro_in_background(self.start_processing)
@@ -78,6 +79,9 @@ def resume(self):
     def set_url(self, url: str):
         self.server_path = url
 
+    def set_access_token(self, token: str):
+        self.access_token = token
+
     def get_active_worker_count(self) -> int:
         count = 0
         for worker in self.active_jobs:
@@ -258,6 +262,7 @@ async def call_prediction(self, events: List[Event], batch: bool):
             method=Request.Method.POST,
             url=f"{self.server_path}api/predict",
             json=dict(data),
+            headers={"Authorization": f"Bearer {self.access_token}"},
         )
         return response
 
diff --git a/gradio/routes.py b/gradio/routes.py
--- a/gradio/routes.py
+++ b/gradio/routes.py
@@ -72,6 +72,7 @@ def __init__(self, **kwargs):
         self.state_holder = {}
         self.iterators = defaultdict(dict)
         self.lock = asyncio.Lock()
+        self.queue_token = secrets.token_urlsafe(32)
         super().__init__(**kwargs)
 
     def configure_app(self, blocks: gradio.Blocks) -> None:
@@ -85,6 +86,8 @@ def configure_app(self, blocks: gradio.Blocks) -> None:
             self.auth = None
 
         self.blocks = blocks
+        if hasattr(self.blocks, "_queue"):
+            self.blocks._queue.set_access_token(self.queue_token)
         self.cwd = os.getcwd()
         self.favicon_path = blocks.favicon_path
         self.tokens = {}
@@ -303,7 +306,10 @@ async def run_predict(
         @app.post("/api/{api_name}", dependencies=[Depends(login_check)])
         @app.post("/api/{api_name}/", dependencies=[Depends(login_check)])
         async def predict(
-            api_name: str, body: PredictBody, username: str = Depends(get_current_user)
+            api_name: str,
+            body: PredictBody,
+            request: Request,
+            username: str = Depends(get_current_user),
         ):
             if body.fn_index is None:
                 for i, fn in enumerate(app.blocks.dependencies):
@@ -317,6 +323,15 @@ async def predict(
                         },
                         status_code=500,
                     )
+            if not app.blocks.api_open and app.blocks.queue_enabled_for_fn(
+                body.fn_index
+            ):
+                if f"Bearer {app.queue_token}" != request.headers.get("Authorization"):
+                    raise HTTPException(
+                        status_code=status.HTTP_401_UNAUTHORIZED,
+                        detail="Not authorized to skip the queue",
+                    )
+
             # If this fn_index cancels jobs, then the only input we need is the
             # current session hash
             if app.blocks.dependencies[body.fn_index]["cancels"]:


Test patch:
diff --git a/test/test_blocks.py b/test/test_blocks.py
--- a/test/test_blocks.py
+++ b/test/test_blocks.py
@@ -17,7 +17,6 @@
 
 import gradio as gr
 import gradio.events
-from gradio.blocks import Block
 from gradio.exceptions import DuplicateBlockError
 from gradio.routes import PredictBody
 from gradio.test_data.blocks_configs import XRAY_CONFIG
@@ -788,5 +787,21 @@ def iteration(a):
             demo.queue().launch(prevent_thread_lock=True)
 
 
+def test_queue_enabled_for_fn():
+    with gr.Blocks() as demo:
+        input = gr.Textbox()
+        output = gr.Textbox()
+        number = gr.Number()
+        button = gr.Button()
+        button.click(lambda x: f"Hello, {x}!", input, output)
+        button.click(lambda: 42, None, number, queue=True)
+
+    assert not demo.queue_enabled_for_fn(0)
+    assert demo.queue_enabled_for_fn(1)
+    demo.queue()
+    assert demo.queue_enabled_for_fn(0)
+    assert demo.queue_enabled_for_fn(1)
+
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/test/test_routes.py b/test/test_routes.py
--- a/test/test_routes.py
+++ b/test/test_routes.py
@@ -12,7 +12,7 @@
 from fastapi.testclient import TestClient
 
 import gradio as gr
-from gradio import Blocks, Interface, Textbox, close_all, routes
+from gradio import Blocks, Button, Interface, Number, Textbox, close_all, routes
 
 os.environ["GRADIO_ANALYTICS_ENABLED"] = "False"
 
@@ -183,6 +183,7 @@ def generator(string):
         response = client.post(
             "/api/predict/",
             json={"data": ["abc"], "fn_index": 0, "session_hash": "11"},
+            headers={"Authorization": f"Bearer {app.queue_token}"},
         )
         output = dict(response.json())
         assert output["data"] == ["a"]
@@ -190,6 +191,7 @@ def generator(string):
         response = client.post(
             "/api/predict/",
             json={"data": ["abc"], "fn_index": 0, "session_hash": "11"},
+            headers={"Authorization": f"Bearer {app.queue_token}"},
         )
         output = dict(response.json())
         assert output["data"] == ["b"]
@@ -197,6 +199,7 @@ def generator(string):
         response = client.post(
             "/api/predict/",
             json={"data": ["abc"], "fn_index": 0, "session_hash": "11"},
+            headers={"Authorization": f"Bearer {app.queue_token}"},
         )
         output = dict(response.json())
         assert output["data"] == ["c"]
@@ -204,6 +207,7 @@ def generator(string):
         response = client.post(
             "/api/predict/",
             json={"data": ["abc"], "fn_index": 0, "session_hash": "11"},
+            headers={"Authorization": f"Bearer {app.queue_token}"},
         )
         output = dict(response.json())
         assert output["data"] == [None]
@@ -211,6 +215,7 @@ def generator(string):
         response = client.post(
             "/api/predict/",
             json={"data": ["abc"], "fn_index": 0, "session_hash": "11"},
+            headers={"Authorization": f"Bearer {app.queue_token}"},
         )
         output = dict(response.json())
         assert output["data"] == ["a"]
@@ -307,5 +312,77 @@ def read_main():
         assert not gradio_fast_api.app.blocks.dev_mode
 
 
+def test_predict_route_is_blocked_if_api_open_false():
+    io = Interface(lambda x: x, "text", "text", examples=[["freddy"]]).queue(
+        api_open=False
+    )
+    app, _, _ = io.launch(prevent_thread_lock=True)
+    assert not io.show_api
+    client = TestClient(app)
+    result = client.post(
+        "/api/predict", json={"fn_index": 0, "data": [5], "session_hash": "foo"}
+    )
+    assert result.status_code == 401
+
+
+def test_predict_route_not_blocked_if_queue_disabled():
+    with Blocks() as demo:
+        input = Textbox()
+        output = Textbox()
+        number = Number()
+        button = Button()
+        button.click(
+            lambda x: f"Hello, {x}!", input, output, queue=False, api_name="not_blocked"
+        )
+        button.click(lambda: 42, None, number, queue=True, api_name="blocked")
+    app, _, _ = demo.queue(api_open=False).launch(
+        prevent_thread_lock=True, show_api=True
+    )
+    assert not demo.show_api
+    client = TestClient(app)
+
+    result = client.post("/api/blocked", json={"data": [], "session_hash": "foo"})
+    assert result.status_code == 401
+    result = client.post(
+        "/api/not_blocked", json={"data": ["freddy"], "session_hash": "foo"}
+    )
+    assert result.status_code == 200
+    assert result.json()["data"] == ["Hello, freddy!"]
+
+
+def test_predict_route_not_blocked_if_routes_open():
+    with Blocks() as demo:
+        input = Textbox()
+        output = Textbox()
+        button = Button()
+        button.click(
+            lambda x: f"Hello, {x}!", input, output, queue=True, api_name="not_blocked"
+        )
+    app, _, _ = demo.queue(api_open=True).launch(
+        prevent_thread_lock=True, show_api=False
+    )
+    assert demo.show_api
+    client = TestClient(app)
+
+    result = client.post(
+        "/api/not_blocked", json={"data": ["freddy"], "session_hash": "foo"}
+    )
+    assert result.status_code == 200
+    assert result.json()["data"] == ["Hello, freddy!"]
+
+    demo.close()
+    demo.queue(api_open=False).launch(prevent_thread_lock=True, show_api=False)
+    assert not demo.show_api
+
+
+def test_show_api_queue_not_enabled():
+    io = Interface(lambda x: x, "text", "text", examples=[["freddy"]])
+    app, _, _ = io.launch(prevent_thread_lock=True)
+    assert io.show_api
+    io.close()
+    io.launch(prevent_thread_lock=True, show_api=False)
+    assert not io.show_api
+
+
 if __name__ == "__main__":
     unittest.main()