internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: oobabooga/text-generation-webui

Issue:
extension/openai chat/completion endpoint IndexError: string index out of range
### Describe the bug

openai extension /completion endpoint works correctly whilst /chat/completion returns IndexError: string index out of range in the logs and obviously no result in the API.

The issue seems at line 318 when checking if the string end with newline:
`if system_msg[-1] != '\n':` 


### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

```
curl http://192.168.178.35:5001/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'
```

### Screenshot

_No response_

### Logs

```shell
25/May/2023 14:03:09] "POST /v1/chat/completions HTTP/1.1" 200 -
----------------------------------------
Exception occurred during processing of request from ('192.168.196.154', 54388)
Traceback (most recent call last):
  File "D:\Apps\oobabooga-windows\installer_files\env\lib\socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "D:\Apps\oobabooga-windows\installer_files\env\lib\socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "D:\Apps\oobabooga-windows\installer_files\env\lib\socketserver.py", line 747, in init
    self.handle()
  File "D:\Apps\oobabooga-windows\installer_files\env\lib\http\server.py", line 433, in handle
    self.handle_one_request()
  File "D:\Apps\oobabooga-windows\installer_files\env\lib\http\server.py", line 421, in handle_one_request
    method()
  File "D:\Apps\oobabooga-windows\text-generation-webui\extensions\openai\script.py", line 318, in do_POST
    if system_msg[-1] != '\n':
IndexError: string index out of range
```


### System Info

```shell
Windows 11
Nvidia Geforce 3060Ti
Python 3.10.11
```



Discussion:

Resolving patch:
diff --git a/extensions/openai/script.py b/extensions/openai/script.py
--- a/extensions/openai/script.py
+++ b/extensions/openai/script.py
@@ -315,7 +315,7 @@ def do_POST(self):
 
                 # can't really truncate the system messages
                 system_msg = '\n'.join(system_msgs)
-                if system_msg[-1] != '\n':
+                if system_msg and system_msg[-1] != '\n':
                     system_msg = system_msg + '\n'
 
                 system_token_count = len(encode(system_msg)[0])


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
Small bug, when arbitrary loading character.json that doesn't exist
### Describe the bug

Things will blow if character json doesn't exist (that webui loads arbitrary after model load)
text-generation-webui\modules\chat.py", line 513, in load_character

file_contents = open(filepath, 'r', encoding='utf-8').read()
FileNotFoundError: [Errno 2] No such file or directory:

Probably wrap it in
try:
....
    except FileNotFoundError:

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

if json is missing

### Screenshot

_No response_

### Logs

```shell
File "H:\oobabooga_windows\installer_files\env\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "H:\oobabooga_windows\text-generation-webui\modules\chat.py", line 513, in load_character
    file_contents = open(filepath, 'r', encoding='utf-8').read()
```


### System Info

```shell
windows
```



Discussion:

Resolving patch:
diff --git a/modules/chat.py b/modules/chat.py
--- a/modules/chat.py
+++ b/modules/chat.py
@@ -513,11 +513,16 @@ def load_character(character, name1, name2, instruct=False):
     if character != 'None':
         folder = 'characters' if not instruct else 'characters/instruction-following'
         picture = generate_pfp_cache(character)
+        filepath = None
         for extension in ["yml", "yaml", "json"]:
             filepath = Path(f'{folder}/{character}.{extension}')
             if filepath.exists():
                 break
 
+        if filepath is None:
+            logger.error(f"Could not find character file for {character} in {folder} folder. Please check your spelling.")
+            return name1, name2, picture, greeting, context, turn_template.replace("\n", r"\n")
+
         file_contents = open(filepath, 'r', encoding='utf-8').read()
         data = json.loads(file_contents) if extension == "json" else yaml.safe_load(file_contents)
 


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
Error when downloading model from UI
### Describe the bug

I just downloaded the latest version of text-generation-webui on Ubuntu and started the UI but it is not longer allowing me to download a model from the UI. I tried to downloading 'anon8231489123/vicuna-13b-GPTQ-4bit-128g' but got the following error:

Traceback (most recent call last): File “/home/squirol/ben2/oobabooga_linux/text-generation-webui/server.py”, line 134, in download_model_wrapper downloader = downloader_module.ModelDownloader() TypeError: ModelDownloader.init() missing 1 required positional argument: ‘max_retries’

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

1. Launch web UI using ./start_linux.sh
2. Open browser to http://127.0.0.1:7860/
3. Enter 'anon8231489123/vicuna-13b-GPTQ-4bit-128g' and select Download in UI
4. View exception under Download button

### Screenshot

![image](https://github.com/oobabooga/text-generation-webui/assets/4974765/69a6d099-cce9-4b44-90b1-e05a891de8e0)


### Logs

```shell
N/A
```


### System Info

```shell
Ubuntu
NVIDIA
```



Discussion:
Can confirm, also was able to quickly fix for myself  by modifying line 134 of server.py to read `downloader = downloader_module.ModelDownloader(max_retries=1)`

Resolving patch:
diff --git a/download-model.py b/download-model.py
--- a/download-model.py
+++ b/download-model.py
@@ -23,7 +23,7 @@
 
 
 class ModelDownloader:
-    def __init__(self, max_retries):
+    def __init__(self, max_retries = 5):
         self.s = requests.Session()
         if max_retries:
             self.s.mount('https://cdn-lfs.huggingface.co', HTTPAdapter(max_retries=max_retries))


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
OpenAI-api Streaming: closing http socket does not cancel generation
### Describe the bug

Streaming: closing http socket does not cancel generation like when the old api was around

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

enable openai-api extension, use api with streaming, close socket, continues generating

### Screenshot

_No response_

### Logs

```shell
no logs
```


### System Info

```shell
win 10
msi
3070
local
frontend: SillyTavern
```



Discussion:
Afaik, SSE is uni-directional so the server doesn't know about the client disconnection. But it is possible to stop the generation by posting to `/v1/internal/stop-generation`:

```
  curl -X 'POST' \
    -H "Content-Type: application/json" \
    -d '' \
    'https://127.0.0.1:5000/v1/internal/stop-generation'
```

Resolving patch:
diff --git a/extensions/api/util.py b/extensions/api/util.py
--- a/extensions/api/util.py
+++ b/extensions/api/util.py
@@ -74,8 +74,8 @@ def build_parameters(body, chat=False):
         if str(character) == "None":
             character = "Assistant"
 
-        name1, name2, _, greeting, context, _ = load_character_memoized(character, str(body.get('your_name', shared.settings['name1'])), '', instruct=False)
-        name1_instruct, name2_instruct, _, _, context_instruct, turn_template = load_character_memoized(instruction_template, '', '', instruct=True)
+        name1, name2, _, greeting, context, _, _ = load_character_memoized(character, str(body.get('your_name', shared.settings['name1'])), '', instruct=False)
+        name1_instruct, name2_instruct, _, _, context_instruct, turn_template, _ = load_character_memoized(instruction_template, '', '', instruct=True)
         generate_params.update({
             'mode': str(body.get('mode', 'chat')),
             'name1': str(body.get('name1', name1)),
diff --git a/extensions/openai/script.py b/extensions/openai/script.py
--- a/extensions/openai/script.py
+++ b/extensions/openai/script.py
@@ -91,6 +91,10 @@ async def generator():
             async with streaming_semaphore:
                 response = OAIcompletions.stream_completions(to_dict(request_data), is_legacy=is_legacy)
                 for resp in response:
+                    disconnected = await request.is_disconnected()
+                    if disconnected:
+                        break
+
                     yield {"data": json.dumps(resp)}
 
         return EventSourceResponse(generator())  # SSE streaming
@@ -110,6 +114,10 @@ async def generator():
             async with streaming_semaphore:
                 response = OAIcompletions.stream_chat_completions(to_dict(request_data), is_legacy=is_legacy)
                 for resp in response:
+                    disconnected = await request.is_disconnected()
+                    if disconnected:
+                        break
+
                     yield {"data": json.dumps(resp)}
 
         return EventSourceResponse(generator())  # SSE streaming


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
Multimodal api is not longer available
### Describe the bug

I think this was forgotten in the transition to the openai API: [usage-through-api](https://github.com/oobabooga/text-generation-webui/tree/main/extensions/multimodal#usage-through-api)

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

-

### Screenshot

_No response_

### Logs

```shell
-
```


### System Info

```shell
-
```



Discussion:
FYI - The (beta) OpenAI equivalent looks to be: https://platform.openai.com/docs/guides/vision

Essentially adding image types and a URL/base64 image to the content passed into the completions APIs. Obviously none of this is implemented yet, but doesn't look too hard.

Resolving patch:
diff --git a/extensions/openai/completions.py b/extensions/openai/completions.py
--- a/extensions/openai/completions.py
+++ b/extensions/openai/completions.py
@@ -1,10 +1,15 @@
+import base64
 import copy
+import re
 import time
 from collections import deque
+from io import BytesIO
 
+import requests
 import tiktoken
 import torch
 import torch.nn.functional as F
+from PIL import Image
 from transformers import LogitsProcessor, LogitsProcessorList
 
 from extensions.openai.errors import InvalidRequestError
@@ -140,7 +145,25 @@ def convert_history(history):
     system_message = ""
 
     for entry in history:
-        content = entry["content"]
+        if "image_url" in entry:
+            image_url = entry['image_url']
+            if "base64" in image_url:
+                image_url = re.sub('^data:image/.+;base64,', '', image_url)
+                img = Image.open(BytesIO(base64.b64decode(image_url)))
+            else:
+                try:
+                    my_res = requests.get(image_url)
+                    img = Image.open(BytesIO(my_res.content))
+                except Exception:
+                    raise 'Image cannot be loaded from the URL!'
+
+            buffered = BytesIO()
+            img.save(buffered, format="JPEG")
+            img_str = base64.b64encode(buffered.getvalue()).decode('utf-8')
+            content = f'<img src="data:image/jpeg;base64,{img_str}">'
+        else:
+            content = entry["content"]
+
         role = entry["role"]
 
         if role == "user":
@@ -182,7 +205,8 @@ def chat_completions_common(body: dict, is_legacy: bool = False, stream=False) -
             raise InvalidRequestError(message="messages: missing role", param='messages')
         elif m['role'] == 'function':
             raise InvalidRequestError(message="role: function is not supported.", param='messages')
-        if 'content' not in m:
+
+        if 'content' not in m and "image_url" not in m:
             raise InvalidRequestError(message="messages: missing content", param='messages')
 
     # Chat Completions


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
Chat output has no spaces between words
I'm just starting up and managed to download the model **codellama_CodeLlama-7b-Instruct-hf**. Whenever I write a chat message with the AI, I'm getting something but has no spaces between the words so I'm not sure if I missed something or there is something wrong with the model.
This is an example Input/Output that I'm getting:

**Input:**
How many class types in default c#

**Output:**
Thereareseveralclassesavailablebydefault,including:Boolean,Byte,Char,DateTime,Decimal,Double,Int16,Int32,Int64,SByte,Single,String,TimeSpan,UInt16,UInt32,andUInt64.YoumayalsofindithelpfultoknowthatC#providessupportforarrays,lists,dictionaries,sets,tuples,andothercollectiontypesaswell.


Can someone provide few hints how to fix it?





Discussion:
I am also just starting to set up text-generation-webui and have been experiencing this for the last couple of days as well.
Using the snapshot-2023-12-03 branch I get spaces between words.
> Using the snapshot-2023-12-03 branch I get spaces between words.

Could you produce steps to fix or if there are any files to replace?
> > Using the snapshot-2023-12-03 branch I get spaces between words.
> 
> Could you produce steps to fix or if there are any files to replace?

I went to **snapshot-2023-12-03** and from it, I replaced the folders/files in my local installation:

- server.py
- modules
- js
- extensions

Now I'm getting spaces again.

Resolving patch:
diff --git a/modules/text_generation.py b/modules/text_generation.py
--- a/modules/text_generation.py
+++ b/modules/text_generation.py
@@ -265,9 +265,8 @@ def apply_stopping_strings(reply, all_stop_strings):
 
 def get_reply_from_output_ids(output_ids, state, starting_from=0):
     reply = decode(output_ids[starting_from:], state['skip_special_tokens'])
-    if type(shared.tokenizer) in [transformers.LlamaTokenizer, transformers.LlamaTokenizerFast] and len(output_ids) > starting_from:
-        if shared.tokenizer.convert_ids_to_tokens(int(output_ids[starting_from])).startswith('▁'):
-            reply = ' ' + reply
+    if hasattr(shared.tokenizer, 'convert_ids_to_tokens') and len(output_ids) > starting_from and shared.tokenizer.convert_ids_to_tokens(int(output_ids[starting_from])).startswith('▁'):
+        reply = ' ' + reply
 
     return reply
 


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
Refreshing browser causes selected model to reset to default
### Describe the bug

After loading a model, if I refresh the browser (just simple F5), I see that the model tab no longer has the loaded model selected. If I passed a default model with --model, that's the one that will appear, and if not it will just show "None"

Funny enough, the model loader settings remain, with ExLlamav2_HF and proper max_seq_len etc

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

Load the web ui
select any model that wasn't loaded, load it
refresh the page
go to model page and observe that your loaded model is no longer the one selected in the dropdown

### Screenshot

![image](https://github.com/oobabooga/text-generation-webui/assets/3266127/2315277d-d22b-4522-a436-921a40f61589)


### Logs

```shell
nothing relevant
```


### System Info

```shell
Ubuntu 22.04 docker
```



Discussion:

Resolving patch:
diff --git a/modules/ui_model_menu.py b/modules/ui_model_menu.py
--- a/modules/ui_model_menu.py
+++ b/modules/ui_model_menu.py
@@ -58,7 +58,7 @@ def create_ui():
                 with gr.Row():
                     with gr.Column():
                         with gr.Row():
-                            shared.gradio['model_menu'] = gr.Dropdown(choices=utils.get_available_models(), value=shared.model_name, label='Model', elem_classes='slim-dropdown', interactive=not mu)
+                            shared.gradio['model_menu'] = gr.Dropdown(choices=utils.get_available_models(), value=lambda: shared.model_name, label='Model', elem_classes='slim-dropdown', interactive=not mu)
                             ui.create_refresh_button(shared.gradio['model_menu'], lambda: None, lambda: {'choices': utils.get_available_models()}, 'refresh-button', interactive=not mu)
                             shared.gradio['load_model'] = gr.Button("Load", visible=not shared.settings['autoload_model'], elem_classes='refresh-button', interactive=not mu)
                             shared.gradio['unload_model'] = gr.Button("Unload", elem_classes='refresh-button', interactive=not mu)


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
coqui_tts fails to load as assumes interactive sessions to accept ToS
### Describe the bug

When enabled coqui_tts prevents textgen from starting as it expects an interactive session for a user to accept a ToS agreement

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

- Enable coqui_tts
- Restart textgen
- Note that textgen never starts
- Check console logs
```
2023-12-12 22:13:22 INFO:Loading the extension "coqui_tts"...
[XTTS] Loading XTTS...
 > You must agree to the terms of service to use this model.
 | > Please see the terms of service at https://coqui.ai/cpml.txt
 | > "I have read, understood and agreed to the Terms and Conditions." - [y/n]
```
- No way to accept non-interactively

### Screenshot

_No response_

### Logs

```shell
INFO:     Started server process [37]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
2023-12-12 22:13:18 DEBUG:Intercepting all calls to posthog.
2023-12-12 22:13:19 DEBUG:Creating Sentence Embedder...
2023-12-12 22:13:20 WARNING:Using embedded DuckDB without persistence: data will be transient
2023-12-12 22:13:22 DEBUG:Loading hyperparameters...
2023-12-12 22:13:22 INFO:Loading the extension "coqui_tts"...
[XTTS] Loading XTTS...
 > You must agree to the terms of service to use this model.
 | > Please see the terms of service at https://coqui.ai/cpml.txt
 | > "I have read, understood and agreed to the Terms and Conditions." - [y/n]
```


### System Info

```shell
Latest official docker image running on server.
```


Note that a workaround for this is to remove coqui_tts and install "alltalk_tts" instead which seems to work without issue.


Discussion:

Resolving patch:
diff --git a/extensions/coqui_tts/script.py b/extensions/coqui_tts/script.py
--- a/extensions/coqui_tts/script.py
+++ b/extensions/coqui_tts/script.py
@@ -1,3 +1,4 @@
+import os
 import html
 import json
 import random
@@ -26,6 +27,7 @@
 
     raise
 
+os.environ["COQUI_TOS_AGREED"] = "1"
 
 params = {
     "activate": True,


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
`user_bio` is None by default, get an error when replacing the character names
### Describe the bug
Using API `/v1/chat/completions` (None stream mode) without `user_bio`.

----

The new parameter `user_bio` in API chat mode raises an error because it's `None` as default.
https://github.com/oobabooga/text-generation-webui/blob/7cf1402bde48fd76af501d5efecb34227bf4d082/extensions/openai/typing.py#L110

----

Then, in `chat.py` can't replace the names correctly.
https://github.com/oobabooga/text-generation-webui/blob/7cf1402bde48fd76af501d5efecb34227bf4d082/modules/chat.py#L97

get this error
![image](https://github.com/oobabooga/text-generation-webui/assets/15323975/8d5f9519-33d5-48e8-809c-7dd29e43eee2)

-----

An empty string as default in webui.
https://github.com/oobabooga/text-generation-webui/blob/7cf1402bde48fd76af501d5efecb34227bf4d082/modules/shared.py#L60



### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

Any request in `/v1/chat/completions`

### Screenshot

_No response_

### Logs

```shell
    text = text.replace('{{user}}', name1).replace('{{char}}', name2)
AttributeError: 'NoneType' object has no attribute 'replace'
```


### System Info

```shell
None
```



Discussion:

Resolving patch:
diff --git a/extensions/openai/typing.py b/extensions/openai/typing.py
--- a/extensions/openai/typing.py
+++ b/extensions/openai/typing.py
@@ -107,7 +107,7 @@ class ChatCompletionRequestParams(BaseModel):
     context: str | None = Field(default=None, description="Overwrites the value set by character field.")
     greeting: str | None = Field(default=None, description="Overwrites the value set by character field.")
     user_name: str | None = Field(default=None, description="Your name (the user). By default, it's \"You\".", alias="name1")
-    user_bio: str | None = Field(default=None, description="The user description/personality.")
+    user_bio: str | None = Field(default='', description="The user description/personality.")
     chat_template_str: str | None = Field(default=None, description="Jinja2 template for chat.")
 
     chat_instruct_command: str | None = None


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
Save UI defaults to settings.yaml Not working
### Describe the bug

When I try to activate few options and try  using Save UI defaults to settings.yaml it save empty settings.yaml



### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

I check few options like in the image below:

![image](https://github.com/oobabooga/text-generation-webui/assets/4099839/1aee6ff0-abb3-4052-8831-a01fa9ef8e33)

and press  Save UI defaults to settings.yaml

close the termenal and start again and it goes back as if I did not set those options
Add to that when I check settings.yaml it is totally empty file


### Screenshot

_No response_

### Logs

```shell
it does not show any log
```


### System Info

```shell
Windows 11
I9 13900
Nvidia 4090
128GB RAM
```



Discussion:
I have the same problem.
Did you the "Save UI defaults to settings.yaml" show an "ERROR" and the second input tab show "./"?
I think is the problem source.

Resolving patch:
diff --git a/modules/ui.py b/modules/ui.py
--- a/modules/ui.py
+++ b/modules/ui.py
@@ -233,14 +233,16 @@ def save_settings(state, preset, extensions_list, show_controls, theme_state):
 
     # Save extension values in the UI
     for extension_name in extensions_list:
-        extension = getattr(extensions, extension_name).script
-        if hasattr(extension, 'params'):
-            params = getattr(extension, 'params')
-            for param in params:
-                _id = f"{extension_name}-{param}"
-                # Only save if different from default value
-                if param not in shared.default_settings or params[param] != shared.default_settings[param]:
-                    output[_id] = params[param]
+        extension = getattr(extensions, extension_name, None)
+        if extension:
+            extension = extension.script
+            if hasattr(extension, 'params'):
+                params = getattr(extension, 'params')
+                for param in params:
+                    _id = f"{extension_name}-{param}"
+                    # Only save if different from default value
+                    if param not in shared.default_settings or params[param] != shared.default_settings[param]:
+                        output[_id] = params[param]
 
     # Do not save unchanged settings
     for key in list(output.keys()):


Test patch: