Repository: oobabooga/text-generation-webui

Issue:
OpenAI API reply TypeError
### Describe the bug

ERROR encountered while replying other front ends (e.g. langflow):
TypeError: must be str, not list
Seems to be something wrong with stream-reply

Full log:
2023-06-25 03:20:37 INFO:Loaded the model in 14.14 seconds.

2023-06-25 03:20:37 INFO:Loading the extension "openai"...
2023-06-25 03:20:38 INFO:Loading the extension "gallery"...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

Loaded embedding model: all-mpnet-base-v2, max sequence length: 384
Starting OpenAI compatible api:
OPENAI_API_BASE=http://127.0.0.1:5001/v1
127.0.0.1 - - [25/Jun/2023 03:21:11] "POST /v1/chat/completions HTTP/1.1" 200 -
Host: 127.0.0.1:5001
X-OpenAI-Client-User-Agent: {"bindings_version": "0.27.8", "httplib": "requests", "lang": "python", "lang_version": "3.10.8", "platform": "Windows-10-10.0.22621-SP0", "publisher": "openai", "uname": "Windows 10 10.0.22621 AMD64"}
User-Agent: OpenAI/v1 PythonBindings/0.27.8
Authorization: Bearer sk-12345
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate
Content-Length: 413


{'messages': [{'role': 'user', 'content': 'The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n\nCurrent conversation:\n\nHuman: Hello\nAI:'}], 'model': 'gpt-3.5-turbo', 'max_tokens': None, 'stream': True, 'n': 1, 'temperature': 0.7}
Loaded instruction role format: Vicuna-v0
{'prompt': "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\n\n### Human: The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n\nCurrent conversation:\n\nHuman: Hello\nAI:\n### Assistant:", 'req_params': {'max_new_tokens': 1024, 'temperature': 0.7, 'top_p': 1.0, 'top_k': 1, 'repetition_penalty': 1.18, 'encoder_repetition_penalty': 1.0, 'suffix': None, 'stream': True, 'echo': False, 'seed': -1, 'truncation_length': 2048, 'add_bos_token': True, 'do_sample': True, 'typical_p': 1.0, 'epsilon_cutoff': 0.0, 'eta_cutoff': 0.0, 'tfs': 1.0, 'top_a': 0.0, 'min_length': 0, 'no_repeat_ngram_size': 0, 'num_beams': 1, 'penalty_alpha': 0.0, 'length_penalty': 1.0, 'early_stopping': False, 'mirostat_mode': 0, 'mirostat_tau': 5.0, 'mirostat_eta': 0.1, 'ban_eos_token': False, 'skip_special_tokens': True, 'custom_stopping_strings': ['\n###', '\n### Human:', '### Human:']}}
Exception ignored in: <generator object generate_reply_custom at 0x000001D4CFDD4F90>
Traceback (most recent call last):
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 327, in generate_reply_custom
    print(f'Output generated in {(t1-t0):.2f} seconds ({new_tokens/(t1-t0):.2f} tokens/s, {new_tokens} tokens, context {original_tokens}, seed {seed})')
ZeroDivisionError: float division by zero
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52167)
Traceback (most recent call last):
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 747, in __init__
    self.handle()
  File "D:\AnaConda\envs\NLP\lib\http\server.py", line 432, in handle
    self.handle_one_request()
  File "D:\AnaConda\envs\NLP\lib\http\server.py", line 420, in handle_one_request
    method()
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\extensions\openai\script.py", line 472, in do_POST
    for a in generator:
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 23, in generate_reply
    for result in _generate_reply(*args, **kwargs):
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 210, in _generate_reply
    reply, stop_found = apply_stopping_strings(reply, all_stop_strings)
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 148, in apply_stopping_strings
    idx = reply.find(string)
TypeError: must be str, not list


### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

flags:
--chat --gpu-memory 22 --model Vicuna-33B-GPTQ-4bit --trust-remote-code --loader exllama --extensions openai

webui was working well.

### Screenshot

_No response_

### Logs

```shell
2023-06-25 03:20:37 INFO:Loaded the model in 14.14 seconds.

2023-06-25 03:20:37 INFO:Loading the extension "openai"...
2023-06-25 03:20:38 INFO:Loading the extension "gallery"...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

Loaded embedding model: all-mpnet-base-v2, max sequence length: 384
Starting OpenAI compatible api:
OPENAI_API_BASE=http://127.0.0.1:5001/v1
Output generated in 2.19 seconds (12.33 tokens/s, 27 tokens, context 43, seed 951727781)
127.0.0.1 - - [25/Jun/2023 03:21:11] "POST /v1/chat/completions HTTP/1.1" 200 -
Host: 127.0.0.1:5001
X-OpenAI-Client-User-Agent: {"bindings_version": "0.27.8", "httplib": "requests", "lang": "python", "lang_version": "3.10.8", "platform": "Windows-10-10.0.22621-SP0", "publisher": "openai", "uname": "Windows 10 10.0.22621 AMD64"}
User-Agent: OpenAI/v1 PythonBindings/0.27.8
Authorization: Bearer sk-12345
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate
Content-Length: 413


{'messages': [{'role': 'user', 'content': 'The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n\nCurrent conversation:\n\nHuman: Hello\nAI:'}], 'model': 'gpt-3.5-turbo', 'max_tokens': None, 'stream': True, 'n': 1, 'temperature': 0.7}
Loaded instruction role format: Vicuna-v0
{'prompt': "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\n\n### Human: The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n\nCurrent conversation:\n\nHuman: Hello\nAI:\n### Assistant:", 'req_params': {'max_new_tokens': 1024, 'temperature': 0.7, 'top_p': 1.0, 'top_k': 1, 'repetition_penalty': 1.18, 'encoder_repetition_penalty': 1.0, 'suffix': None, 'stream': True, 'echo': False, 'seed': -1, 'truncation_length': 2048, 'add_bos_token': True, 'do_sample': True, 'typical_p': 1.0, 'epsilon_cutoff': 0.0, 'eta_cutoff': 0.0, 'tfs': 1.0, 'top_a': 0.0, 'min_length': 0, 'no_repeat_ngram_size': 0, 'num_beams': 1, 'penalty_alpha': 0.0, 'length_penalty': 1.0, 'early_stopping': False, 'mirostat_mode': 0, 'mirostat_tau': 5.0, 'mirostat_eta': 0.1, 'ban_eos_token': False, 'skip_special_tokens': True, 'custom_stopping_strings': ['\n###', '\n### Human:', '### Human:']}}
Exception ignored in: <generator object generate_reply_custom at 0x000001D4CFDD4F90>
Traceback (most recent call last):
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 327, in generate_reply_custom
    print(f'Output generated in {(t1-t0):.2f} seconds ({new_tokens/(t1-t0):.2f} tokens/s, {new_tokens} tokens, context {original_tokens}, seed {seed})')
ZeroDivisionError: float division by zero
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52167)
Traceback (most recent call last):
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 747, in __init__
    self.handle()
  File "D:\AnaConda\envs\NLP\lib\http\server.py", line 432, in handle
    self.handle_one_request()
  File "D:\AnaConda\envs\NLP\lib\http\server.py", line 420, in handle_one_request
    method()
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\extensions\openai\script.py", line 472, in do_POST
    for a in generator:
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 23, in generate_reply
    for result in _generate_reply(*args, **kwargs):
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 210, in _generate_reply
    reply, stop_found = apply_stopping_strings(reply, all_stop_strings)
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 148, in apply_stopping_strings
    idx = reply.find(string)
TypeError: must be str, not list
```


### System Info

```shell
OS: Windows 11
GPU: Nvidia Geforce RTX 4090 24G * 1
```

OpenAI API reply TypeError
### Describe the bug

ERROR encountered while replying other front ends (e.g. langflow):
TypeError: must be str, not list
Seems to be something wrong with stream-reply

Full log:
2023-06-25 03:20:37 INFO:Loaded the model in 14.14 seconds.

2023-06-25 03:20:37 INFO:Loading the extension "openai"...
2023-06-25 03:20:38 INFO:Loading the extension "gallery"...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

Loaded embedding model: all-mpnet-base-v2, max sequence length: 384
Starting OpenAI compatible api:
OPENAI_API_BASE=http://127.0.0.1:5001/v1
127.0.0.1 - - [25/Jun/2023 03:21:11] "POST /v1/chat/completions HTTP/1.1" 200 -
Host: 127.0.0.1:5001
X-OpenAI-Client-User-Agent: {"bindings_version": "0.27.8", "httplib": "requests", "lang": "python", "lang_version": "3.10.8", "platform": "Windows-10-10.0.22621-SP0", "publisher": "openai", "uname": "Windows 10 10.0.22621 AMD64"}
User-Agent: OpenAI/v1 PythonBindings/0.27.8
Authorization: Bearer sk-12345
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate
Content-Length: 413


{'messages': [{'role': 'user', 'content': 'The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n\nCurrent conversation:\n\nHuman: Hello\nAI:'}], 'model': 'gpt-3.5-turbo', 'max_tokens': None, 'stream': True, 'n': 1, 'temperature': 0.7}
Loaded instruction role format: Vicuna-v0
{'prompt': "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\n\n### Human: The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n\nCurrent conversation:\n\nHuman: Hello\nAI:\n### Assistant:", 'req_params': {'max_new_tokens': 1024, 'temperature': 0.7, 'top_p': 1.0, 'top_k': 1, 'repetition_penalty': 1.18, 'encoder_repetition_penalty': 1.0, 'suffix': None, 'stream': True, 'echo': False, 'seed': -1, 'truncation_length': 2048, 'add_bos_token': True, 'do_sample': True, 'typical_p': 1.0, 'epsilon_cutoff': 0.0, 'eta_cutoff': 0.0, 'tfs': 1.0, 'top_a': 0.0, 'min_length': 0, 'no_repeat_ngram_size': 0, 'num_beams': 1, 'penalty_alpha': 0.0, 'length_penalty': 1.0, 'early_stopping': False, 'mirostat_mode': 0, 'mirostat_tau': 5.0, 'mirostat_eta': 0.1, 'ban_eos_token': False, 'skip_special_tokens': True, 'custom_stopping_strings': ['\n###', '\n### Human:', '### Human:']}}
Exception ignored in: <generator object generate_reply_custom at 0x000001D4CFDD4F90>
Traceback (most recent call last):
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 327, in generate_reply_custom
    print(f'Output generated in {(t1-t0):.2f} seconds ({new_tokens/(t1-t0):.2f} tokens/s, {new_tokens} tokens, context {original_tokens}, seed {seed})')
ZeroDivisionError: float division by zero
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52167)
Traceback (most recent call last):
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 747, in __init__
    self.handle()
  File "D:\AnaConda\envs\NLP\lib\http\server.py", line 432, in handle
    self.handle_one_request()
  File "D:\AnaConda\envs\NLP\lib\http\server.py", line 420, in handle_one_request
    method()
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\extensions\openai\script.py", line 472, in do_POST
    for a in generator:
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 23, in generate_reply
    for result in _generate_reply(*args, **kwargs):
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 210, in _generate_reply
    reply, stop_found = apply_stopping_strings(reply, all_stop_strings)
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 148, in apply_stopping_strings
    idx = reply.find(string)
TypeError: must be str, not list


### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

flags:
--chat --gpu-memory 22 --model Vicuna-33B-GPTQ-4bit --trust-remote-code --loader exllama --extensions openai

webui was working well.

### Screenshot

_No response_

### Logs

```shell
2023-06-25 03:20:37 INFO:Loaded the model in 14.14 seconds.

2023-06-25 03:20:37 INFO:Loading the extension "openai"...
2023-06-25 03:20:38 INFO:Loading the extension "gallery"...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

Loaded embedding model: all-mpnet-base-v2, max sequence length: 384
Starting OpenAI compatible api:
OPENAI_API_BASE=http://127.0.0.1:5001/v1
Output generated in 2.19 seconds (12.33 tokens/s, 27 tokens, context 43, seed 951727781)
127.0.0.1 - - [25/Jun/2023 03:21:11] "POST /v1/chat/completions HTTP/1.1" 200 -
Host: 127.0.0.1:5001
X-OpenAI-Client-User-Agent: {"bindings_version": "0.27.8", "httplib": "requests", "lang": "python", "lang_version": "3.10.8", "platform": "Windows-10-10.0.22621-SP0", "publisher": "openai", "uname": "Windows 10 10.0.22621 AMD64"}
User-Agent: OpenAI/v1 PythonBindings/0.27.8
Authorization: Bearer sk-12345
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate
Content-Length: 413


{'messages': [{'role': 'user', 'content': 'The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n\nCurrent conversation:\n\nHuman: Hello\nAI:'}], 'model': 'gpt-3.5-turbo', 'max_tokens': None, 'stream': True, 'n': 1, 'temperature': 0.7}
Loaded instruction role format: Vicuna-v0
{'prompt': "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\n\n### Human: The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n\nCurrent conversation:\n\nHuman: Hello\nAI:\n### Assistant:", 'req_params': {'max_new_tokens': 1024, 'temperature': 0.7, 'top_p': 1.0, 'top_k': 1, 'repetition_penalty': 1.18, 'encoder_repetition_penalty': 1.0, 'suffix': None, 'stream': True, 'echo': False, 'seed': -1, 'truncation_length': 2048, 'add_bos_token': True, 'do_sample': True, 'typical_p': 1.0, 'epsilon_cutoff': 0.0, 'eta_cutoff': 0.0, 'tfs': 1.0, 'top_a': 0.0, 'min_length': 0, 'no_repeat_ngram_size': 0, 'num_beams': 1, 'penalty_alpha': 0.0, 'length_penalty': 1.0, 'early_stopping': False, 'mirostat_mode': 0, 'mirostat_tau': 5.0, 'mirostat_eta': 0.1, 'ban_eos_token': False, 'skip_special_tokens': True, 'custom_stopping_strings': ['\n###', '\n### Human:', '### Human:']}}
Exception ignored in: <generator object generate_reply_custom at 0x000001D4CFDD4F90>
Traceback (most recent call last):
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 327, in generate_reply_custom
    print(f'Output generated in {(t1-t0):.2f} seconds ({new_tokens/(t1-t0):.2f} tokens/s, {new_tokens} tokens, context {original_tokens}, seed {seed})')
ZeroDivisionError: float division by zero
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52167)
Traceback (most recent call last):
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "D:\AnaConda\envs\NLP\lib\socketserver.py", line 747, in __init__
    self.handle()
  File "D:\AnaConda\envs\NLP\lib\http\server.py", line 432, in handle
    self.handle_one_request()
  File "D:\AnaConda\envs\NLP\lib\http\server.py", line 420, in handle_one_request
    method()
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\extensions\openai\script.py", line 472, in do_POST
    for a in generator:
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 23, in generate_reply
    for result in _generate_reply(*args, **kwargs):
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 210, in _generate_reply
    reply, stop_found = apply_stopping_strings(reply, all_stop_strings)
  File "D:\PyCharm Community Edition 2022.2.3\Projects\text-generation-webui\modules\text_generation.py", line 148, in apply_stopping_strings
    idx = reply.find(string)
TypeError: must be str, not list
```


### System Info

```shell
OS: Windows 11
GPU: Nvidia Geforce RTX 4090 24G * 1
```



Discussion:

Resolving patch:
diff --git a/extensions/openai/script.py b/extensions/openai/script.py
--- a/extensions/openai/script.py
+++ b/extensions/openai/script.py
@@ -54,7 +54,7 @@
     'mirostat_eta': 0.1,
     'ban_eos_token': False,
     'skip_special_tokens': True,
-    'custom_stopping_strings': ['\n###'],
+    'custom_stopping_strings': '',
 }
 
 # Optional, install the module and download the model to enable
@@ -254,7 +254,7 @@ def do_POST(self):
                 return
 
             is_legacy = '/generate' in self.path
-            is_chat = 'chat' in self.path
+            is_chat_request = 'chat' in self.path
             resp_list = 'data' if is_legacy else 'choices'
 
             # XXX model is ignored for now
@@ -262,23 +262,23 @@ def do_POST(self):
             model = shared.model_name
             created_time = int(time.time())
 
-            cmpl_id = "chatcmpl-%d" % (created_time) if is_chat else "conv-%d" % (created_time)
+            cmpl_id = "chatcmpl-%d" % (created_time) if is_chat_request else "conv-%d" % (created_time)
 
             # Request Parameters
             # Try to use openai defaults or map them to something with the same intent
             req_params = default_req_params.copy()
-            req_params['custom_stopping_strings'] = default_req_params['custom_stopping_strings'].copy()
+            stopping_strings = []
 
             if 'stop' in body:
                 if isinstance(body['stop'], str):
-                    req_params['custom_stopping_strings'].extend([body['stop']])
+                    stopping_strings.extend([body['stop']])
                 elif isinstance(body['stop'], list):
-                    req_params['custom_stopping_strings'].extend(body['stop'])
+                    stopping_strings.extend(body['stop'])
 
             truncation_length = default(shared.settings, 'truncation_length', 2048)
             truncation_length = clamp(default(body, 'truncation_length', truncation_length), 1, truncation_length)
 
-            default_max_tokens = truncation_length if is_chat else 16  # completions default, chat default is 'inf' so we need to cap it.
+            default_max_tokens = truncation_length if is_chat_request else 16  # completions default, chat default is 'inf' so we need to cap it.
 
             max_tokens_str = 'length' if is_legacy else 'max_tokens'
             max_tokens = default(body, max_tokens_str, default(shared.settings, 'max_new_tokens', default_max_tokens))
@@ -295,9 +295,11 @@ def do_POST(self):
             req_params['seed'] = shared.settings.get('seed', default_req_params['seed'])
             req_params['add_bos_token'] = shared.settings.get('add_bos_token', default_req_params['add_bos_token'])
 
+            is_streaming = req_params['stream']
+
             self.send_response(200)
             self.send_access_control_headers()
-            if req_params['stream']:
+            if is_streaming:
                 self.send_header('Content-Type', 'text/event-stream')
                 self.send_header('Cache-Control', 'no-cache')
                 # self.send_header('Connection', 'keep-alive')
@@ -311,7 +313,7 @@ def do_POST(self):
             stream_object_type = ''
             object_type = ''
 
-            if is_chat:
+            if is_chat_request:
                 # Chat Completions
                 stream_object_type = 'chat.completions.chunk'
                 object_type = 'chat.completions'
@@ -347,20 +349,22 @@ def do_POST(self):
                             'prompt': bot_prompt,
                         }
 
-                        if instruct['user']: # WizardLM and some others have no user prompt.
-                            req_params['custom_stopping_strings'].extend(['\n' + instruct['user'], instruct['user']])
+                        if 'Alpaca' in shared.settings['instruction_template']:
+                            stopping_strings.extend(['\n###'])
+                        elif instruct['user']: # WizardLM and some others have no user prompt.
+                            stopping_strings.extend(['\n' + instruct['user'], instruct['user']])
 
                         if debug:
                             print(f"Loaded instruction role format: {shared.settings['instruction_template']}")
 
                     except Exception as e:
-                        req_params['custom_stopping_strings'].extend(['\nuser:'])
+                        stopping_strings.extend(['\nuser:'])
 
                         print(f"Exception: When loading characters/instruction-following/{shared.settings['instruction_template']}.yaml: {repr(e)}")
                         print("Warning: Loaded default instruction-following template for model.")
 
                 else:
-                    req_params['custom_stopping_strings'].extend(['\nuser:'])
+                    stopping_strings.extend(['\nuser:'])
                     print("Warning: Loaded default instruction-following template for model.")
 
                 system_msgs = []
@@ -391,7 +395,7 @@ def do_POST(self):
                     system_msg = system_msg + '\n'
 
                 system_token_count = len(encode(system_msg)[0])
-                remaining_tokens = req_params['truncation_length'] - system_token_count
+                remaining_tokens = truncation_length - system_token_count
                 chat_msg = ''
 
                 while chat_msgs:
@@ -424,20 +428,19 @@ def do_POST(self):
                     return
 
                 token_count = len(encode(prompt)[0])
-                if token_count >= req_params['truncation_length']:
+                if token_count >= truncation_length:
                     new_len = int(len(prompt) * shared.settings['truncation_length'] / token_count)
                     prompt = prompt[-new_len:]
                     new_token_count = len(encode(prompt)[0])
                     print(f"Warning: truncating prompt to {new_len} characters, was {token_count} tokens. Now: {new_token_count} tokens.")
                     token_count = new_token_count
 
-            if req_params['truncation_length'] - token_count < req_params['max_new_tokens']:
-                print(f"Warning: Ignoring max_new_tokens ({req_params['max_new_tokens']}), too large for the remaining context. Remaining tokens: {req_params['truncation_length'] - token_count}")
-                req_params['max_new_tokens'] = req_params['truncation_length'] - token_count
+            if truncation_length - token_count < req_params['max_new_tokens']:
+                print(f"Warning: Ignoring max_new_tokens ({req_params['max_new_tokens']}), too large for the remaining context. Remaining tokens: {truncation_length - token_count}")
+                req_params['max_new_tokens'] = truncation_length - token_count
                 print(f"Warning: Set max_new_tokens = {req_params['max_new_tokens']}")
 
-            if req_params['stream']:
-                shared.args.chat = True
+            if is_streaming:
                 # begin streaming
                 chunk = {
                     "id": cmpl_id,
@@ -463,11 +466,11 @@ def do_POST(self):
             # generate reply #######################################
             if debug:
                 print({'prompt': prompt, 'req_params': req_params})
-            generator = generate_reply(prompt, req_params, stopping_strings=req_params['custom_stopping_strings'], is_chat=False)
+            generator = generate_reply(prompt, req_params, stopping_strings=stopping_strings, is_chat=False)
 
             answer = ''
             seen_content = ''
-            longest_stop_len = max([len(x) for x in req_params['custom_stopping_strings']] + [0])
+            longest_stop_len = max([len(x) for x in stopping_strings] + [0])
 
             for a in generator:
                 answer = a
@@ -476,7 +479,7 @@ def do_POST(self):
                 len_seen = len(seen_content)
                 search_start = max(len_seen - longest_stop_len, 0)
 
-                for string in req_params['custom_stopping_strings']:
+                for string in stopping_strings:
                     idx = answer.find(string, search_start)
                     if idx != -1:
                         answer = answer[:idx]  # clip it.
@@ -489,7 +492,7 @@ def do_POST(self):
                 # is completed, buffer and generate more, don't send it
                 buffer_and_continue = False
 
-                for string in req_params['custom_stopping_strings']:
+                for string in stopping_strings:
                     for j in range(len(string) - 1, 0, -1):
                         if answer[-j:] == string[:j]:
                             buffer_and_continue = True
@@ -501,7 +504,7 @@ def do_POST(self):
                 if buffer_and_continue:
                     continue
 
-                if req_params['stream']:
+                if is_streaming:
                     # Streaming
                     new_content = answer[len_seen:]
 
@@ -534,7 +537,7 @@ def do_POST(self):
                     self.wfile.write(response.encode('utf-8'))
                     completion_token_count += len(encode(new_content)[0])
 
-            if req_params['stream']:
+            if is_streaming:
                 chunk = {
                     "id": cmpl_id,
                     "object": stream_object_type,
@@ -575,7 +578,7 @@ def do_POST(self):
 
             completion_token_count = len(encode(answer)[0])
             stop_reason = "stop"
-            if token_count + completion_token_count >= req_params['truncation_length']:
+            if token_count + completion_token_count >= truncation_length:
                 stop_reason = "length"
 
             resp = {
@@ -594,7 +597,7 @@ def do_POST(self):
                 }
             }
 
-            if is_chat:
+            if is_chat_request:
                 resp[resp_list][0]["message"] = {"role": "assistant", "content": answer}
             else:
                 resp[resp_list][0]["text"] = answer
@@ -620,7 +623,7 @@ def do_POST(self):
 
             # Request parameters
             req_params = default_req_params.copy()
-            req_params['custom_stopping_strings'] = default_req_params['custom_stopping_strings'].copy()
+            stopping_strings = []
 
             # Alpaca is verbose so a good default prompt
             default_template = (
@@ -632,26 +635,29 @@ def do_POST(self):
             instruction_template = default_template
             
             # Use the special instruction/input/response template for anything trained like Alpaca
-            if shared.settings['instruction_template'] and not (shared.settings['instruction_template'] in ['Alpaca', 'Alpaca-Input']):
-                try:
-                    instruct = yaml.safe_load(open(f"characters/instruction-following/{shared.settings['instruction_template']}.yaml", 'r'))
-
-                    template = instruct['turn_template']
-                    template = template\
-                        .replace('
', instruct.get('user', ''))\
-                        .replace('
', instruct.get('bot', ''))\
-                        .replace('<|user-message|>', '{instruction}\n{input}')
+            if shared.settings['instruction_template']:
+                if 'Alpaca' in shared.settings['instruction_template']:
+                    stopping_strings.extend(['\n###'])
+                else:
+                    try:
+                        instruct = yaml.safe_load(open(f"characters/instruction-following/{shared.settings['instruction_template']}.yaml", 'r'))
 
-                    instruction_template = instruct.get('context', '') + template[:template.find('<|bot-message|>')].rstrip(' ')
-                    if instruct['user']:
-                        req_params['custom_stopping_strings'].extend(['\n' + instruct['user'], instruct['user'] ])
+                        template = instruct['turn_template']
+                        template = template\
+                            .replace('
', instruct.get('user', ''))\
+                            .replace('
', instruct.get('bot', ''))\
+                            .replace('<|user-message|>', '{instruction}\n{input}')
 
-                except Exception as e:
-                    instruction_template = default_template
-                    print(f"Exception: When loading characters/instruction-following/{shared.settings['instruction_template']}.yaml: {repr(e)}")
-                    print("Warning: Loaded default instruction-following template (Alpaca) for model.")
+                        instruction_template = instruct.get('context', '') + template[:template.find('<|bot-message|>')].rstrip(' ')
+                        if instruct['user']:
+                            stopping_strings.extend(['\n' + instruct['user'], instruct['user'] ])
 
+                    except Exception as e:
+                        instruction_template = default_template
+                        print(f"Exception: When loading characters/instruction-following/{shared.settings['instruction_template']}.yaml: {repr(e)}")
+                        print("Warning: Loaded default instruction-following template (Alpaca) for model.")
             else:
+                stopping_strings.extend(['\n###'])
                 print("Warning: Loaded default instruction-following template (Alpaca) for model.")
                 
 
@@ -671,9 +677,9 @@ def do_POST(self):
             if debug:
                 print({'edit_template': edit_task, 'req_params': req_params, 'token_count': token_count})
             
-            generator = generate_reply(edit_task, req_params, stopping_strings=req_params['custom_stopping_strings'], is_chat=False)
+            generator = generate_reply(edit_task, req_params, stopping_strings=stopping_strings, is_chat=False)
 
-            longest_stop_len = max([len(x) for x in req_params['custom_stopping_strings']] + [0])
+            longest_stop_len = max([len(x) for x in stopping_strings] + [0])
             answer = ''
             seen_content = ''
             for a in generator:
@@ -683,7 +689,7 @@ def do_POST(self):
                 len_seen = len(seen_content)
                 search_start = max(len_seen - longest_stop_len, 0)
 
-                for string in req_params['custom_stopping_strings']:
+                for string in stopping_strings:
                     idx = answer.find(string, search_start)
                     if idx != -1:
                         answer = answer[:idx]  # clip it.


Test patch: