Repository: oobabooga/text-generation-webui

Issue:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 125: character maps to <undefined>
### Describe the bug

That is probably locale-dependant thing, but some models complain on decoding error. Here's diff to fix:

```python
diff --git a/modules/models_settings.py b/modules/models_settings.py
index 8a4febd..90c7b69 100644
--- a/modules/models_settings.py
+++ b/modules/models_settings.py
@@ -102,7 +102,7 @@ def get_model_metadata(model):
     # Try to find the Jinja instruct template
     path = Path(f'{shared.args.model_dir}/{model}') / 'tokenizer_config.json'
     if path.exists():
-        metadata = json.loads(open(path, 'r').read())
+        metadata = json.loads(open(path, 'r', encoding='utf-8').read())
         if 'chat_template' in metadata:
             template = metadata['chat_template']
             for k in ['eos_token', 'bos_token']:
```

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

Eg. see https://huggingface.co/LoneStriker/deepseek-coder-33b-instruct-4.65bpw-h6-exl2/discussions/1#657daf9c9a9e347d4b9fc544

### Screenshot

_No response_

### Logs

```shell
See: https://huggingface.co/LoneStriker/deepseek-coder-33b-instruct-4.65bpw-h6-exl2/discussions/1#657daf9c9a9e347d4b9fc544
```


### System Info

```shell
windows
```



Discussion:
Thanks nice hint

A similar error occurred with the AWQ model, but it worked.
＞UnicodeDecodeError: 'cp932' codec can't decode byte 0x81 in position 451: illegal multibyte sequence

I tried made the same edits, but i still cant load any of the deep seek models unless its in ggml. Tried them all and i get the x81 byte

`Traceback (most recent call last):
  File "C:\Users\PC\Documents\text-generation-webui\modules\ui_model_menu.py", line 214, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\Documents\text-generation-webui\modules\models.py", line 79, in load_model
    metadata = get_model_metadata(model_name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\Documents\text-generation-webui\modules\models_settings.py", line 115, in get_model_metadata
    metadata = json.loads(open(path, 'r', encoding='utf-8').read())
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\PC\Documents\text-generation-webui\installer_files\env\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 125: character maps to <undefined>

`
I have the same error for GGUF and exl2 models, if someone found the solution I'll be happy to take that information
By taking inspiration from this
https://huggingface.co/LoneStriker/deepseek-coder-33b-instruct-4.65bpw-h6-exl2/discussions/1#657db96ba982e9093f675299

On the file ```text-generation-webui\modules\models_settings.py```
If you replace all the 
```
metadata = json.loads(open(path, 'r').read())
```
By
```
metadata = json.loads(open(path, 'r', encoding='utf-8').read())
```
The loading works, don't forget to set the rope_scaling (alpha_value AND compress_pos_emb) to 4, it's how this model work
![image](https://github.com/oobabooga/text-generation-webui/assets/110173477/7035be75-574f-440e-8b4d-753cf6e52299)

@oobabooga can you take a look at your models_settings.py script so that we shouldn't need to do this manually to make the deepseek coder models work? 

Resolving patch:
diff --git a/modules/models_settings.py b/modules/models_settings.py
--- a/modules/models_settings.py
+++ b/modules/models_settings.py
@@ -35,7 +35,7 @@ def get_model_metadata(model):
 
     path = Path(f'{shared.args.model_dir}/{model}/config.json')
     if path.exists():
-        hf_metadata = json.loads(open(path, 'r').read())
+        hf_metadata = json.loads(open(path, 'r', encoding='utf-8').read())
     else:
         hf_metadata = None
 
@@ -78,7 +78,7 @@ def get_model_metadata(model):
     else:
         # Transformers metadata
         if hf_metadata is not None:
-            metadata = json.loads(open(path, 'r').read())
+            metadata = json.loads(open(path, 'r', encoding='utf-8').read())
             if 'max_position_embeddings' in metadata:
                 model_settings['truncation_length'] = metadata['max_position_embeddings']
                 model_settings['max_seq_len'] = metadata['max_position_embeddings']
@@ -101,7 +101,7 @@ def get_model_metadata(model):
         # Read AutoGPTQ metadata
         path = Path(f'{shared.args.model_dir}/{model}/quantize_config.json')
         if path.exists():
-            metadata = json.loads(open(path, 'r').read())
+            metadata = json.loads(open(path, 'r', encoding='utf-8').read())
             if 'bits' in metadata:
                 model_settings['wbits'] = metadata['bits']
             if 'group_size' in metadata:
@@ -112,7 +112,7 @@ def get_model_metadata(model):
     # Try to find the Jinja instruct template
     path = Path(f'{shared.args.model_dir}/{model}') / 'tokenizer_config.json'
     if path.exists():
-        metadata = json.loads(open(path, 'r').read())
+        metadata = json.loads(open(path, 'r', encoding='utf-8').read())
         if 'chat_template' in metadata:
             template = metadata['chat_template']
             for k in ['eos_token', 'bos_token']:


Test patch: