Transformers
GGUF
English
mixtral
text-generation-inference

Is Metal support available?

#2
by LeaveNhA - opened

Is Metal support available for your models or do you plan to support it?

Thank you for your beautiful work!

@LeaveNhA i believe it is,
to use metal, just install llama.cpp or whatever you are using with metal.
It depends on what you are using so what are you using?

Sorry for poor question and your time because of that. What I meant is real metal support, with the latest framework named MLX?

It simply requires me to convert it to npz format. But I couldn't manage that. Forgive me if it's trivial.

@LeaveNhA llama.cpp is infact faster then mlx for metal even though mlx is strictly for metal.
but if you still want mlx models and the way to convert it to mlx here
https://huggingface.co/mlx-community

I couldnt find this specific one but either you could convert it by yourself or use another alternative like the orignal mixtral instruct.

Sign up or log in to comment