SebastianSchramm commited on
Commit
a0e9bba
1 Parent(s): c7eaa78

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +48 -0
README.md ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ pipeline_tag: text-generation
5
+ library_name: transformers
6
+ tags:
7
+ - LLM
8
+ - Universal-NER
9
+ - NER
10
+ - 4bit
11
+ inference: false
12
+ ---
13
+ ![image](qunatized_lama_color_letters_4bit_512px.png)
14
+
15
+ # Quantized version of Universal-NER/UniNER-7B-type
16
+
17
+ [Universal-NER/UniNER-7B-type](https://huggingface.co/Universal-NER/UniNER-7B-type) quantized to 4bit with GPTQ and stored with 1GB shard size.
18
+
19
+ ## Model Description
20
+
21
+ The model [Universal-NER/UniNER-7B-type](https://huggingface.co/Universal-NER/UniNER-7B-type) was quantized to 4bit, group_size 128, and act-order=True with auto-gptq integration in transformers (https://huggingface.co/blog/gptq-integration).
22
+
23
+ ## Evaluation
24
+ TODO
25
+
26
+ ## Prompt template
27
+
28
+ Prompt template is the same as for the full precision model:
29
+
30
+ ```python
31
+ prompt_template = """A virtual assistant answers questions from a user based on the provided text.
32
+ USER: Text: {input_text}
33
+ ASSISTANT: I’ve read this text.
34
+ USER: What describes {entity_name} in the text?
35
+ ASSISTANT:
36
+ """
37
+ ```
38
+
39
+ ## Usage
40
+
41
+ It is recommended to format input according to the prompt template mentioned above during inference for best results.
42
+
43
+ ```python
44
+ prompt = prompt_template.format_map({"input_text": "Cologne is a great city in Germany - maybe even the greatest ;)", "entity_name": "city"})
45
+ ```
46
+
47
+ ## License
48
+ The original full precision model and its associated data are released under the CC BY-NC 4.0 license. Hence, the same license applies for the 4bit version.