Update README.md
Browse files
README.md
CHANGED
@@ -18,7 +18,7 @@ This is MPT-7B patched so that it can be used with a LoRA. Note that while I tes
|
|
18 |
|
19 |
Note that when using LoRA, there is a strange quirk that prevents me from causing generation with an empty prompt.
|
20 |
|
21 |
-
I also included a model-agnostic export_hf_checkpoint.py script, which you can use to merge your lora back into a new full model. Once you do this, you do not need to use the patched version of the model code anymore. That being said, if you want to be able to load the model in 8bit you will still need it. The usage is `python export_hf_checkpoint.py <source> <lora> <dest>`.
|
22 |
|
23 |
If you would like to use this with text-generation-webui, apply the following patch:
|
24 |
```
|
@@ -45,7 +45,7 @@ If you would like to use this with text-generation-webui, apply the following pa
|
|
45 |
```
|
46 |
python server.py --model mosaicml_mpt-7b-instruct --trust-remote-code --load-in-8bit
|
47 |
```
|
48 |
-
You may also need to patch bitsandbytes/nn/modules.py to prevent running out of VRAM when saving the LoRA:
|
49 |
```
|
50 |
--- a/modules.py
|
51 |
+++ b/modules.py
|
@@ -69,3 +69,4 @@ You may also need to patch bitsandbytes/nn/modules.py to prevent running out of
|
|
69 |
|
70 |
The alterations are based on the source code for the llama model from HF Transformers.
|
71 |
|
|
|
|
18 |
|
19 |
Note that when using LoRA, there is a strange quirk that prevents me from causing generation with an empty prompt.
|
20 |
|
21 |
+
I also included a model-agnostic `export_hf_checkpoint.py` script, which you can use to merge your lora back into a new full model (check Github link at the end). Once you do this, you do not need to use the patched version of the model code anymore. That being said, if you want to be able to load the model in 8bit you will still need it. The usage is `python export_hf_checkpoint.py <source> <lora> <dest>`.
|
22 |
|
23 |
If you would like to use this with text-generation-webui, apply the following patch:
|
24 |
```
|
|
|
45 |
```
|
46 |
python server.py --model mosaicml_mpt-7b-instruct --trust-remote-code --load-in-8bit
|
47 |
```
|
48 |
+
You may also need to patch `bitsandbytes/nn/modules.py` to prevent running out of VRAM when saving the LoRA:
|
49 |
```
|
50 |
--- a/modules.py
|
51 |
+++ b/modules.py
|
|
|
69 |
|
70 |
The alterations are based on the source code for the llama model from HF Transformers.
|
71 |
|
72 |
+
Big thanks to "iwalton3" for making this possible. You can find the `export_hf_checkpoint.py` here: https://github.com/iwalton3/mpt-lora-patch/blob/master/export_hf_checkpoint.py
|