Update README.md
Browse files
README.md
CHANGED
|
@@ -9,9 +9,18 @@ tags:
|
|
| 9 |
- moss
|
| 10 |
- llm
|
| 11 |
---
|
| 12 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
## Table of Contents
|
| 14 |
|
|
|
|
| 15 |
- [Open-source list](#spiral_notepad-open-source-list)
|
| 16 |
- [Models](#models)
|
| 17 |
- [Data](#data)
|
|
|
|
| 9 |
- moss
|
| 10 |
- llm
|
| 11 |
---
|
| 12 |
+
|
| 13 |
+
# MOSS Fix Autotune
|
| 14 |
+
|
| 15 |
+
This model repository serves as a temporary drop-in fix for an error that occurs during inference on int4 family of MOSS models, specifically, TypeError: '<' not supported between instances of 'tuple' and 'float' in [#159](https://github.com/OpenLMLab/MOSS/issues/159).
|
| 16 |
+
|
| 17 |
+
When loading the model with model = AutoModelForCausalLM.from_pretrained("fnlp/moss-moon-003-sft-int4", trust_remote_code=True).half().cuda(), the custom_autotune.py from the huggingface model files will be used. As such, it is necessary to update the file in the huggingface model repo.
|
| 18 |
+
|
| 19 |
+
Before the official maintainers update and fix the issue, I have applied the patch and re-uploaded the model files in this repo for convenience. The patched model files can be used with model = AutoModelForCausalLM.from_pretrained("DotIN13/moss-moon-003-sft-int4-fix-autotune", trust_remote_code=True).half().cuda().
|
| 20 |
+
|
| 21 |
## Table of Contents
|
| 22 |
|
| 23 |
+
- [Fix Autotune](#fix-autotune)
|
| 24 |
- [Open-source list](#spiral_notepad-open-source-list)
|
| 25 |
- [Models](#models)
|
| 26 |
- [Data](#data)
|