Repository: oobabooga/text-generation-webui

Issue:
Flexgen error with opt-13b-erebus
### Describe the bug

When I try to talk I get this error

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Reproduction

python server.py --model opt-13b-erebus --chat --flexgen --compress-weight --verbose

### Screenshot

_No response_

### Logs

```shell
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
CUDA SETUP: CUDA runtime path found: /home/avolk/miniconda3/envs/textgen/lib/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /home/avolk/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...
Loading opt-13b-erebus...
Loaded the model in 129.12 seconds.
Loading the extension "gallery"... Ok.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.


This is a conversation with your Assistant. The Assistant is very helpful and is eager to chat with you and answer your questions.
Assistant: Hello there!
You: hello
Assistant:
--------------------

Traceback (most recent call last):
  File "/home/avolk/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/routes.py", line 393, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/avolk/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/blocks.py", line 1108, in process_api
    result = await self.call_function(
  File "/home/avolk/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/blocks.py", line 929, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/avolk/miniconda3/envs/textgen/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/avolk/miniconda3/envs/textgen/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/avolk/miniconda3/envs/textgen/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/avolk/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/utils.py", line 490, in async_iteration
    return next(iterator)
  File "/home/avolk/text-generation-webui/modules/chat.py", line 218, in cai_chatbot_wrapper
    for history in chatbot_wrapper(text, state):
  File "/home/avolk/text-generation-webui/modules/chat.py", line 152, in chatbot_wrapper
    for reply in generate_reply(f"{prompt}{' ' if len(cumulative_reply) > 0 else ''}{cumulative_reply}", state, eos_token=eos_token, stopping_strings=stopping_strings):
  File "/home/avolk/text-generation-webui/modules/text_generation.py", line 190, in generate_reply
    generate_params['stop'] = state['eos_token_ids'][-1]
KeyError: 'eos_token_ids'
```


### System Info

```shell
OS: Ubuntu 22.04.2 LTS x86_64 
CPU: AMD Ryzen 7 4800H
GPU: NVIDIA GeForce RTX 3050 Mobile
```



Discussion:
I can confirm this bug.
Ok, does that mean I just have to wait for the update and then this bug will be fixed?

Resolving patch:
diff --git a/modules/text_generation.py b/modules/text_generation.py
--- a/modules/text_generation.py
+++ b/modules/text_generation.py
@@ -209,7 +209,7 @@ def generate_reply(question, state, eos_token=None, stopping_strings=[]):
     else:
         for k in ['max_new_tokens', 'do_sample', 'temperature']:
             generate_params[k] = state[k]
-        generate_params['stop'] = state['eos_token_ids'][-1]
+        generate_params['stop'] = eos_token_ids[-1]
         if not shared.args.no_stream:
             generate_params['max_new_tokens'] = 8
 


Test patch: