OpenLLM-France/Luciole-23B-Instruct-1.1-FP8

Luciole-23B-Instruct-1.1-FP8 is a FP8-quantized version of Luciole-23B-Instruct-1.1 in the transformers / compressed-tensors (https://github.com/neuralmagic/compressed-tensors) format, quantized with LLM Compressor (https://github.com/vllm-project/llm-compressor).

It is intended to be served with vLLM (https://github.com/vllm-project/vllm):

vllm serve OpenLLM-France/Luciole-23B-Instruct-1.1-FP8

FP8 acceleration requires a GPU with native FP8 support (NVIDIA Hopper, Ada Lovelace, or Blackwell).