How? :)
How were you able to do this? :) I get an error when I try to convert
Traceback (most recent call last):
File "/home/text-generation/convert/exllamav2/convert.py", line 1, in <module>
import exllamav2.conversion.convert_exl2
File "/home/text-generation/convert/exllamav2/exllamav2/conversion/convert_exl2.py", line 197, in <module>
tokenizer = ExLlamaV2Tokenizer(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/text-generation/convert/exllamav2/exllamav2/tokenizer/tokenizer.py", line 130, in __init__
self.tokenizer_model = ExLlamaV2TokenizerHF(path_hf)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/text-generation/convert/exllamav2/exllamav2/tokenizer/hf.py", line 22, in __init__
self.hf_tokenizer = Tokenizer.from_file(tokenizer_json)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: data did not match any variant of untagged enum ModelWrapper at line 757450 column 3
@Apel-sin
For me quantizing went without any issues. This error didn't appear when I made the quants.
I used a prebuilt exllamav2 0.2.3 package that was installed with Text-Generation-WebUI / TabbyAPI with 0.2.3 source package from Github.
My usual workflow is something like this:
Activating python environment with prebuilt exllamav2 0.2.3.
cd exllamav2-0.2.3 (source package dir)
python convert.py -i source_dir -o output_dir -om measurements.json -hsol 100000
python convert.py -i source_dir -o output_dir -m measurements.json -hsol 100000 -cf quant_dir/model-4bpwh6-exl2 -nr -b 4 -hb 6
That's about it, I didn't do anything else to quantize this model.
So I have no idea what could've caused the error and why it didn't happen on my end.