Repository: oobabooga/text-generation-webui

Issue:
Failed to load embedding model: all-mpnet-base-v2 While Running Textgen in Colab Notebook
### Describe the bug

I have used this command instead of using old Cuda in my ipynb 

`!git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa`

Now, I ran the server using following code - 

`!python server.py --extensions openai --model guanaco-7B-GPTQ --model_type LLaMa --api --public-api --share --wbits 4 --groupsize 128`

I am getting below error  - 

```
WARNING:The gradio "share link" feature uses a proprietary executable to create a reverse tunnel. Use it with care.
2023-05-30 11:21:05.243240: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so
INFO:Loading guanaco-7B-GPTQ...
INFO:Found the following quantized model: models/guanaco-7B-GPTQ/Guanaco-7B-GPTQ-4bit-128g.no-act-order.safetensors
INFO:Loaded the model in 14.96 seconds.

INFO:Loading the extension "openai"...

Failed to load embedding model: all-mpnet-base-v2
```

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

Run Colab.

Use this notebook. [Colab](https://colab.research.google.com/drive/1wURKtZgM_SWhjy-NlHNVjHl-SKT5AwtF?usp=sharing)

Openai Extension not working as intended

### Screenshot

_No response_

### Logs

```shell
WARNING:The gradio "share link" feature uses a proprietary executable to create a reverse tunnel. Use it with care.
2023-05-30 11:21:05.243240: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so
INFO:Loading guanaco-7B-GPTQ...
INFO:Found the following quantized model: models/guanaco-7B-GPTQ/Guanaco-7B-GPTQ-4bit-128g.no-act-order.safetensors
INFO:Loaded the model in 14.96 seconds.

INFO:Loading the extension "openai"...

Failed to load embedding model: all-mpnet-base-v2
```


### System Info

```shell
Google COlab Notebook with T4 GPU
```



Discussion:
I believe I know what went wrong. I installed the sentence-transformers package. That Error is resolved now. But still can not hit the openai api

Edit : added this command now - `!pip install git+https://github.com/mnt4/flask-cloudflared`
from 

[https://github.com/oobabooga/text-generation-webui/issues/1524]

This is the error stack Trace now - 

```
WARNING:The gradio "share link" feature uses a proprietary executable to create a reverse tunnel. Use it with care.
2023-05-30 12:20:26.338655: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so
INFO:Loading guanaco-7B-GPTQ...
INFO:Found the following quantized model: models/guanaco-7B-GPTQ/Guanaco-7B-GPTQ-4bit-128g.no-act-order.safetensors
INFO:Loaded the model in 18.88 seconds.

INFO:Loading the extension "openai"...
Running on local URL:  http://127.0.0.1:7860/

Loaded embedding model: all-mpnet-base-v2, max sequence length: 384
Running on public URL: https://6d95be3cd607be0555.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
Starting OpenAI compatible api at
OPENAI_API_BASE=https://relative-flex-nose-useful.trycloudflare.com/v1
127.0.0.1 - - [30/May/2023 12:20:55] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:20:55] "GET /metrics HTTP/1.1" 404 -
Starting streaming server at public url wss://current-supported-lonely-walt.trycloudflare.com/api/v1/stream
127.0.0.1 - - [30/May/2023 12:20:58] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:20:58] "GET /metrics HTTP/1.1" 404 -
127.0.0.1 - - [30/May/2023 12:21:01] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:21:01] "GET /metrics HTTP/1.1" 404 -
127.0.0.1 - - [30/May/2023 12:21:04] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:21:04] "GET /metrics HTTP/1.1" 404 -
127.0.0.1 - - [30/May/2023 12:21:07] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:21:07] "GET /metrics HTTP/1.1" 404 -
127.0.0.1 - - [30/May/2023 12:21:10] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:21:10] "GET /metrics HTTP/1.1" 404 -
127.0.0.1 - - [30/May/2023 12:21:13] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:21:13] "GET /metrics HTTP/1.1" 404 -
127.0.0.1 - - [30/May/2023 12:21:16] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:21:16] "GET /metrics HTTP/1.1" 404 -
127.0.0.1 - - [30/May/2023 12:21:19] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:21:19] "GET /metrics HTTP/1.1" 404 -
Closing server running on port: 7860
INFO:Loading the extension "openai"...
Running on local URL:  http://127.0.0.1:7860/
Running on public URL: https://0f095eaedadd0d8f1e.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
127.0.0.1 - - [30/May/2023 12:21:55] code 404, message Not Found
127.0.0.1 - - [30/May/2023 12:21:55] "GET /v1 HTTP/1.1" 404 -
127.0.0.1 - - [30/May/2023 12:22:07] "POST /v1/completions HTTP/1.1" 200 -
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 60196)
Traceback (most recent call last):
  File "/usr/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
  File "/usr/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
  File "/content/text-generation-webui/extensions/openai/script.py", line 404, in do_POST
    for a in generator:
  File "/content/text-generation-webui/modules/text_generation.py", line 24, in generate_reply
    for result in _generate_reply(*args, **kwargs):
  File "/content/text-generation-webui/modules/text_generation.py", line 191, in _generate_reply
    for reply in generate_func(question, original_question, seed, state, eos_token, stopping_strings, is_chat=is_chat):
  File "/content/text-generation-webui/modules/text_generation.py", line 198, in generate_reply_HF
    generate_params[k] = state[k]
KeyError: 'tfs'


```


I am using below code to hit the api - 

```
OPENAI_API_KEY = "dummy"
os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
OPENAI_API_BASE =  'https://relative-flex-nose-useful.trycloudflare.com/v1'
os.environ['OPENAI_API_BASE'] = OPENAI_API_BASE
import os
import openai

openai.api_key = os.getenv("OPENAI_API_KEY")
openai.api_base = os.getenv("OPENAI_API_BASE")
response = openai.Completion.create(
  model="dummy",
  prompt="I am a highly intelligent question answering bot. If you ask me a question that is rooted in truth, I will give you the answer. If you ask me a question that is nonsense, trickery, or has no clear answer, I will respond with \"Unknown\".\n\nQ: What is human life expectancy in the United States?\nA: Human life expectancy in the United States is 78 years.\n\nQ: Who was president of the United States in 1955?\nA: Dwight D. Eisenhower was president of the United States in 1955.\n\nQ: Which party did he belong to?\nA: He belonged to the Republican Party.\n\nQ: What is the square root of banana?\nA: Unknown\n\nQ: How does a telescope work?\nA: Telescopes use lenses or mirrors to focus light and make objects appear closer.\n\nQ: Where were the 1992 Olympics held?\nA: The 1992 Olympics were held in Barcelona, Spain.\n\nQ: How many squigs are in a bonk?\nA: Unknown\n\nQ: Tell me something about vcovid\nA:",
  temperature=0,
  max_tokens=100,
  top_p=1,
  frequency_penalty=0.0,
  presence_penalty=0.0,
  stop=["\n"]
)
print(response)
```
I'm fixing the tfs error now, it's a new required parameter.


Resolving patch:
diff --git a/extensions/openai/script.py b/extensions/openai/script.py
--- a/extensions/openai/script.py
+++ b/extensions/openai/script.py
@@ -18,6 +18,41 @@
 
 debug = True if 'OPENEDAI_DEBUG' in os.environ else False
 
+# Slightly different defaults for OpenAI's API
+default_req_params = {
+    'max_new_tokens': 200,
+    'temperature': 1.0,
+    'top_p': 1.0,
+    'top_k': 1,
+    'repetition_penalty': 1.18,
+    'encoder_repetition_penalty': 1.0,
+    'suffix': None,
+    'stream': False,
+    'echo': False,
+    'seed': -1,
+    # 'n' : default(body, 'n', 1),  # 'n' doesn't have a direct map
+    'truncation_length': 2048,
+    'add_bos_token': True,
+    'do_sample': True,
+    'typical_p': 1.0,
+    'epsilon_cutoff': 0,  # In units of 1e-4
+    'eta_cutoff': 0,  # In units of 1e-4
+    'tfs': 1.0,
+    'top_a': 0.0,
+    'min_length': 0,
+    'no_repeat_ngram_size': 0,
+    'num_beams': 1,
+    'penalty_alpha': 0.0,
+    'length_penalty': 1,
+    'early_stopping': False,
+    'mirostat_mode': 0,
+    'mirostat_tau': 5,
+    'mirostat_eta': 0.1,
+    'ban_eos_token': False,
+    'skip_special_tokens': True,
+    'custom_stopping_strings': [],
+}
+
 # Optional, install the module and download the model to enable
 # v1/embeddings
 try:
@@ -194,46 +229,18 @@ def do_POST(self):
             max_tokens = default(body, max_tokens_str, default(shared.settings, 'max_new_tokens', default_max_tokens))
             # if the user assumes OpenAI, the max_tokens is way too large - try to ignore it unless it's small enough
 
-            req_params = {
-                'max_new_tokens': max_tokens,
-                'temperature': default(body, 'temperature', 1.0),
-                'top_p': default(body, 'top_p', 1.0),
-                'top_k': default(body, 'best_of', 1),
-                # XXX not sure about this one, seems to be the right mapping, but the range is different (-2..2.0) vs 0..2
-                # 0 is default in openai, but 1.0 is default in other places. Maybe it's scaled? scale it.
-                'repetition_penalty': 1.18,  # (default(body, 'presence_penalty', 0) + 2.0 ) / 2.0, # 0 the real default, 1.2 is the model default, but 1.18 works better.
-                # XXX not sure about this one either, same questions. (-2..2.0), 0 is default not 1.0, scale it.
-                'encoder_repetition_penalty': 1.0,  # (default(body, 'frequency_penalty', 0) + 2.0) / 2.0,
-                'suffix': body.get('suffix', None),
-                'stream': default(body, 'stream', False),
-                'echo': default(body, 'echo', False),
-                #####################################################
-                'seed': shared.settings.get('seed', -1),
-                # int(body.get('n', 1)) # perhaps this should be num_beams or chat_generation_attempts? 'n' doesn't have a direct map
-                # unofficial, but it needs to get set anyways.
-                'truncation_length': truncation_length,
-                # no more args.
-                'add_bos_token': shared.settings.get('add_bos_token', True),
-                'do_sample': True,
-                'typical_p': 1.0,
-                'epsilon_cutoff': 0,  # In units of 1e-4
-                'eta_cutoff': 0,  # In units of 1e-4
-                'min_length': 0,
-                'no_repeat_ngram_size': 0,
-                'num_beams': 1,
-                'penalty_alpha': 0.0,
-                'length_penalty': 1,
-                'early_stopping': False,
-                'mirostat_mode': 0,
-                'mirostat_tau': 5,
-                'mirostat_eta': 0.1,
-                'ban_eos_token': False,
-                'skip_special_tokens': True,
-            }
+            req_params = default_req_params.copy()
 
-            # fixup absolute 0.0's
-            for par in ['temperature', 'repetition_penalty', 'encoder_repetition_penalty']:
-                req_params[par] = clamp(req_params[par], 0.001, 1.999)
+            req_params['max_new_tokens'] = max_tokens
+            req_params['truncation_length'] = truncation_length
+            req_params['temperature'] = clamp(default(body, 'temperature', default_req_params['temperature']), 0.001, 1.999) # fixup absolute 0.0
+            req_params['top_p'] = clamp(default(body, 'top_p', default_req_params['top_p']), 0.001, 1.0)
+            req_params['top_k'] = default(body, 'best_of', default_req_params['top_k'])
+            req_params['suffix'] = default(body, 'suffix', default_req_params['suffix'])
+            req_params['stream'] = default(body, 'stream', default_req_params['stream'])
+            req_params['echo'] = default(body, 'echo', default_req_params['echo'])
+            req_params['seed'] = shared.settings.get('seed', default_req_params['seed'])
+            req_params['add_bos_token'] = shared.settings.get('add_bos_token', default_req_params['add_bos_token'])
 
             self.send_response(200)
             if req_params['stream']:
@@ -550,37 +557,14 @@ def do_POST(self):
             token_count = len(encode(edit_task)[0])
             max_tokens = truncation_length - token_count
 
-            req_params = {
-                'max_new_tokens': max_tokens,
-                'temperature': clamp(default(body, 'temperature', 1.0), 0.001, 1.999),
-                'top_p': clamp(default(body, 'top_p', 1.0), 0.001, 1.0),
-                'top_k': 1,
-                'repetition_penalty': 1.18,
-                'encoder_repetition_penalty': 1.0,
-                'suffix': None,
-                'stream': False,
-                'echo': False,
-                'seed': shared.settings.get('seed', -1),
-                # 'n' : default(body, 'n', 1),  # 'n' doesn't have a direct map
-                'truncation_length': truncation_length,
-                'add_bos_token': shared.settings.get('add_bos_token', True),
-                'do_sample': True,
-                'typical_p': 1.0,
-                'epsilon_cutoff': 0,  # In units of 1e-4
-                'eta_cutoff': 0,  # In units of 1e-4
-                'min_length': 0,
-                'no_repeat_ngram_size': 0,
-                'num_beams': 1,
-                'penalty_alpha': 0.0,
-                'length_penalty': 1,
-                'early_stopping': False,
-                'mirostat_mode': 0,
-                'mirostat_tau': 5,
-                'mirostat_eta': 0.1,
-                'ban_eos_token': False,
-                'skip_special_tokens': True,
-                'custom_stopping_strings': [],
-            }
+            req_params = default_req_params.copy()
+
+            req_params['max_new_tokens'] = max_tokens
+            req_params['truncation_length'] = truncation_length
+            req_params['temperature'] = clamp(default(body, 'temperature', default_req_params['temperature']), 0.001, 1.999) # fixup absolute 0.0
+            req_params['top_p'] = clamp(default(body, 'top_p', default_req_params['top_p']), 0.001, 1.0)
+            req_params['seed'] = shared.settings.get('seed', default_req_params['seed'])
+            req_params['add_bos_token'] = shared.settings.get('add_bos_token', default_req_params['add_bos_token'])
 
             if debug:
                 print({'edit_template': edit_task, 'req_params': req_params, 'token_count': token_count})


Test patch: