Repository: InternLM/lmdeploy

Issue:
[Bug] Can't inference with stream=false 0.0.8
### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.

### Describe the bug

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/root/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/root/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 69, in app
    await response(scope, receive, send)
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/responses.py", line 280, in __call__
    await self.background()
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/background.py", line 43, in __call__
    await task()
  File "/root/miniconda3/lib/python3.11/site-packages/starlette/background.py", line 26, in __call__
    await self.func(*self.args, **self.kwargs)
  File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/serve/openai/api_server.py", line 130, in abort_request
    async for _ in VariableInterface.async_engine.generate_openai(
  File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/serve/async_engine.py", line 216, in generate_openai
    async for outputs in generator.async_stream_infer(
  File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 213, in async_stream_infer
    for output in self.stream_infer(*args, **kwargs):
  File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 347, in stream_infer
    if output[-1].item() == self.eos_id:
       ~~~~~~^^^^
IndexError: index -1 is out of bounds for dimension 0 with size 0

### Reproduction

 {
     "model": "llama2",
    "messages": [
        {"role": "user", "content": "Tell me about France"}
    ],
    "max_tokens": 372,
    "temperature": 0.7,
    "top_p": 1,
    "n": 1,
    "stream": false
}

### Error traceback

_No response_

Request with stream=true works


Discussion:
#378 brought side effects. 
Hi @hatrexltd

Can you try change `File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 347, in stream_infer` as below, and see if it solve your problem?

```
if len(output)>0 and output[-1].item() == self.eos_id:
```
> Hi @hatrexltd
> 
> Can you try change `File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 347, in stream_infer` as below, and see if it solve your problem?
> 
> ```
> if len(output)>0 and output[-1].item() == self.eos_id:
> ```

Works. Thanks.

Resolving patch:
diff --git a/lmdeploy/turbomind/turbomind.py b/lmdeploy/turbomind/turbomind.py
--- a/lmdeploy/turbomind/turbomind.py
+++ b/lmdeploy/turbomind/turbomind.py
@@ -344,7 +344,7 @@ def _broadcast_np(data, dtype, shape=(batch_size, )):
             outputs = []
             for output, len_ in zip(output_ids, sequence_length):
                 output, len_ = output, len_.item()
-                if output[-1].item() == self.eos_id:
+                if len(output) > 0 and output[-1].item() == self.eos_id:
                     outputs.append((output[:-1], len_ - 1))
                 else:
                     outputs.append((output, len_))


Test patch: