Qubitium commited on
Commit
e7a9661
1 Parent(s): 83c9f59

Note to use dbrx-base-tokenizer

Browse files
Files changed (1) hide show
  1. README.md +3 -2
README.md CHANGED
@@ -16,9 +16,10 @@ Training Notes/Observations:
16
  # start with this as reference point and move up or down based on eval/train loss
17
  learning_rate = 1.5e-5
18
  ```
19
- 2. Due to nature of BPE (tiktoken), tokenizer expansion/resize is not very friendly to training. Use text based special tokens if you need/use extra tokens to avoid bad train/eval losses
20
 
21
- Quants:
 
22
 
23
  1. 4bit gptq/marlin: https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-marlin
24
  2. 4bit gptq/gptq: https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-gptq
 
16
  # start with this as reference point and move up or down based on eval/train loss
17
  learning_rate = 1.5e-5
18
  ```
19
+ 2. Highly recommend to train this model with `dbrx-base-tokenizer` tokenizer (fully-compatible): https://huggingface.co/LnL-AI/dbrx-base-tokenizer
20
 
21
+
22
+ # Quants:
23
 
24
  1. 4bit gptq/marlin: https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-marlin
25
  2. 4bit gptq/gptq: https://huggingface.co/LnL-AI/dbrx-base-converted-v2-4bit-gptq-gptq